OnLinux

shell script for website deployment through git or svn

This summary is not available. Please click here to view the post.

amazing regex generator

http://txt2re.com

extract data from html table using HTML::TableExtract

in this tutorial you will get to learn :

1) How to extract data from a html table

2 ) regex

3) appending a hash to a hash

4) for loops

----------------------------------------------------------------------------------------------

HTML file

----------------------------------------------------------------------------------------------

<table class="table1" border="1" cellspacing="1" cellpadding="1" style="width: 1144px; height: 883px;">

<tr>

<td> Dom0 

</td><td> DomU 

</td><td> ipaddress 

</td><td> server name 

</td><td> application 

</td><td> website 

</td><td> assigned to 

</td></tr>

<tr>

<td> S1 (1.51) 

</td><td> yourwebsitenews-db 10GB ram 100 GB HDD   

</td><td> 1.0.0.101 

</td><td> yourwebsitenews-db

</td><td> win2k8 ,sql

</td><td> MSSQL Server

</td><td> yourwebsitenews

</td></tr>

<tr>

<td> 

</td><td> yourwebsitetvweb3  4GB ram 48 GB HDD 

</td><td> 9.9.02.132 

</td><td> yourwebsitetvweb3 

</td><td> win2k8,IIS 

</td><td> yourwebsite TV 

</td><td> yourwebsitetv 

</td></tr>

<tr>

<td> 

</td><td> yourwebsitetvweb4  4GB ram 30 GB HDD 

</td><td> 9.9.202.146 

</td><td> yourwebsitetvweb4 

</td><td> win2k8,IIS 

</td><td> yourwebsite TV 

</td><td> yourwebsitetv 

</td></tr>

<tr>

<td> 

</td><td> 

</td></tr>

<tr>

<td> 

</td><td> 

</td></tr>

<tr>

<td> S2 (1.52) 

</td><td> web1 5.5GB ram 50GB hdd 

</td><td> 1.0.0.106 

</td><td> web1 

</td><td> apache,php,memcache 

</td><td> someweb,cndas,lms.someweb.com 

</td><td> IWPL

</td></tr>

<tr>

<td> 

</td><td> web4 5.5GB ram 50GB hdd 

</td><td> 1.0.0.109 

</td><td> web4 

</td><td> apache,php,memcache 

</td><td> b.in,gs.in.com

</td><td> IWPL

</td></tr>

<tr>

<td> 

</td><td> 

</td></tr>

<tr>

<td> S3 (1.53) 

</td><td> DB1 11GB ram 100GB hdd 

</td><td> 1.0.0.111 

</td><td> db1 

</td><td> mysql 

</td><td> all commons (auth ),someweb 

</td><td> IL

</td></tr>

<tr>

<td> 

<tr>

<td> 

</td><td> 

</td></tr>

<tr>

<td> S4 (1.54) 

</td><td> DB3 10GB ram 200GB hdd 

</td><td> 1.0.0.116 

</td><td> db3 

</td><td> mysql 

</td><td> ors_db,ls_ting

</td><td> IWPL

</td></tr>

<tr>

<td> 

</td><td> yourwebsitetvmemcach-1 10 Gb ram 24 Gb hdd 

</td><td> 9.9.202.134 

</td><td> yourwebsitetvmem-1 

</td><td> yourwebsite Tv Memcached Server-1 

</td><td> yourwebsitetv

</td></tr>

<tr>

</td><td> 

</td></tr>

<tr>

<td> S5 (1.55) 

</td><td> 1 systems ops 8GB 100GB 

</td><td> 1.0.0.121 

</td><td> ops 

</td><td> mysql,php,apache 

</td><td> odw opsview reports runtime 

</td><td> IWPL

</td></tr>

<tr>

<td> 

</td><td> cache 4GB ram 129GB hdd 

</td><td> 9.9.202.209 

</td><td> linux-yourwebsite 

</td><td> php,apache 

</td><td> yourwebsite 

</td><td> yourwebsitetv

</td></tr>

<tr>

<td> 

</td><td> cache 6GB ram 20GB hdd 

</td><td> 1.0.0.125 

</td><td> cache-9 

</td><td> backup server

</td><td> IL

</td></tr>

<tr>

<td> 

</td><td> web16- 5 GB RAM 46GB HDD

</td><td> 1.0.0.103

</td><td> web16

</td><td> php,apache

</td><td> voiofa.com,solr instance

</td><td> Il

</td></tr>

<tr>

<td> 

</td><td> 

</td></tr>

<tr>

<td> S6 (1.56) 

</td><td> web9 5GBram 

</td><td> 1.0.0.126 

</td><td> web9 

</td><td> apache,php,memcache 

</td><td> www.ia.com,dia.com,dia.com,bon.ia.com 

</td><td> Il

</td></tr>

<tr>

<td> 

</td><td> web10 

</td><td> 1.0.0.127 

</td><td> web10 

</td><td> apache,php,memcache 

</td><td> hea.ia.com,esults.ia.com

</td><td> Il

</td></tr>

<tr>

<td> 

</td><td> r1s6v1web13 5-GB ram - 50-GB hdd 

</td><td> 1.0.0.129 

</td><td> web13 

</td><td> apache,php,memcache

</td><td> critry,m.try,admin.ccry.com

</td><td> IL

</td></tr></table>

==================================================================

PERL script

==================================================================

#!/usr/bin/perl

use strict;
use warnings;
use Data::Dumper;
use LWP::Simple;
use HTML::TableExtract;

#host your table on a local site
my $url = 'http://site2.int.com/table_tmp1.html';

my @data = get_data($url) ;

shift(@data);
my %serverdata;
my %serverdata_sub;

my $DOM0;

my @aeey;

my $re1='((?:[a-z][a-z]*[0-9]+[a-z0-9]*))'; # Alphanum 1
my $re2='(\\s+)'; # White Space 1
my $re3='\$(.*)\$'; # Round Braces 1

foreach my $y (@data) {
next unless ($y->[2]);
$DOM0 = ($y->[0])? $y->[0] : $DOM0;

my $re=$re1.$re2.$re3;
($DOM0 = $DOM0) =~ s/$re/172.16.$3/isg; # S1 (1.51)

# Dom0 DomU ipaddress server name application website assigned to
# "$DOM0 -----> $y->[2] ========== > $y->[1], $y->[3], $y->[4] $y->[5] $y->[6] ";

push (@{$serverdata{$DOM0}{$y->[2]}}, { 'serverdetails' => $y->[1] , 'servername' => $y->[3], 'application' => $y->[4] , 'websites' => $y->[5] } ) ;

if ( $y->[0] ) {
$DOM0= "$y->[0]";
}else {
$DOM0 = $DOM0;
}
}

foreach my $keysi ( keys %serverdata) {

print "================== $keysi ===============\n";
print Dumper ($serverdata{$keysi}) ;
print "================== ===============\n";

}

sub get_data {
my $sub_url = $_[0];
my $html = get "$sub_url";

my @arry;
my $te = new HTML::TableExtract( depth => 0, count => 0, attribs => { 'class' => "table1" } );

$te->parse($html);
foreach my $ts ($te->tables) {
foreach my $row ($ts->rows) {
# next unless $row->[0] =~ /\w/; # skip garbage rows
@{$row} = grep(s/\n\n|(^\s+|\s+$)//g, @{$row});

push( @arry , [@{$row}] ) ;
# print "$row->[0] , $row->[1],$row->[2],$row->[3]\n";
}
}
return @arry ;
}

AWS -- EC2 ELB -- shell script to change ip in backend varnish incase the ELB ips keeep changing

in AWS the ELP ( load balancer ) ip is dynamic .. so we need to change this in the varnish config file . this script does the trick .

------------------------------------------------------------

#!/bin/bash

#check if varnish is reunning

#if running reload if not restart

varnishstatus=$(/usr/bin/pgrep varnishd |wc -l )

if [ $varnishstatus -gt 0 ]

then

vstatus='reload'

else

vstatus='restart'

#check IP in hosts set in vairrable

#get array of all ips of cname --- internal-EC2-ELB-xxxx-xxxxx3xxxx275.ap-southeast-1.elb.amazonaws.com

temdigcount=/tmp/temdigcount

tmpbackendconfig=/tmp/tmpbackendconfig

backendconfig='/etc/varnish/health.backend.vcl'

LBS_aws='/etc/varnish/LBS_aws'

touch $LBS_aws

defaultvcl=''

LB2='internal-EC2-ELB-xxxx-xxxxx3xxxx275.ap-southeast-1.elb.amazonaws.com'

OIFS=$IFS

IFS=$'\n'

array=($(dig +short $LB2|sort))

IFS=$OIFS

echo ${array[@]} > $temdigcount

DIFF1=`diff $temdigcount $LBS_aws|wc -l `

echo $DIFF1

if [ $DIFF1 -eq 0 ]

then

echo -e "no diff in ip \n"

exit

set_count=0

cat /dev/null > $tmpbackendconfig

for i in ${array[@]}

let set_count++

SERVERIP=$i

cat <<EOF >> $tmpbackendconfig

backend health$set_count {

.host = "$SERVERIP";

.port = "http";

.first_byte_timeout = 300s;

.probe = {

.request = "GET / HTTP/1.1" "Host: yoursite.com" "Connection: close";

.interval = 10s;

.timeout = 10s;

.window = 5;

.threshold = 3;

}

EOF

done

echo -e "\n Bakcend changed, updating VCL and Reloading VARNISH\n\n"

/bin/cp -f $tmpbackendconfig $backendconfig

/etc/init.d/varnish reload

[ -f $FILE ] && \

cp ${FILE} $FILE_${NOW} && \

varnishadm -T localhost:1234 vcl.load vcl_${NOW} $FILE && \

varnishadm -T localhost:1234 vcl.use vcl_${NOW} && \

varnishadm -T localhost:1234 vcl.list |tail -n 3

script to clear php-fpm / restart php-fpm on continuous 503 http error

This script does the following

1) check whether the site is available

2) if not available then clear apc cache

3) if still not available for more thne 5 counts then restart phpfpm and send mail

4) logging happens at every level

================

#!/usr/bin/perl -w

use strict;

use warnings;

use Data::Dumper;

use Getopt::Long;

use WWW::Mechanize ;

use JSON -support_by_pp;

use POSIX qw/ strftime /;

use Net::SMTP;

my $host ;

my $string ;

my $mech = WWW::Mechanize->new(timeout => 5);

my $time = strftime("%d-%m-%Y", localtime());

my $log = '/var/log/php-fpm/check_php-fpm.log';

my $url = 'http://www.yourwebsite.com/';

my $file_counter = '/var/log/php-fpm/filecounter';

my $count ;

if ( eval{ $mech->get($url) } ) {

logit("OK");

$count = get_count();

if ($count > 0 ) {

print "setting count to 0\n";

$count = '0';

add_count($count);

}

}else{

$count = get_count();

add_count($count+1);

logit("apc FAILED -- removing opcode and users cache -- increasing counter to $count");

system("curl http://127.0.0.1/clear-apc.php > /dev/null 2>&1 ");

if ($count > 4 ) {

logit ("restarting php-fpm ");

system("/etc/init.d/php-fpm reload");

sendemail(" Reloading APC $url [ $time ] "," Not able to get content for $url , reloading php-fpm [ $time ]");

print "setting counter back to 0 \n";

add_count('0');

}

sub get_count {

open my $fh, '<', "$file_counter";

read $fh, my $string, -s $fh;

close $fh;

chomp($string);

return $string || '0';

}

sub add_count {

my $count = shift;

open my $fh, '>', "$file_counter";

print $fh $count;

close $fh;

}

sub sendemail {

#usage sendemail("subject","mssg");

my $from = 'from-email-id' ;

my $sub = $_[0] ;

my $content = $_[1] ;

my $to = 'to-emailid' ;

my $relayhost = 'mailserver' || 'localhost';

my $smtp = Net::SMTP->new("$relayhost",

Debug => 1,

);

$smtp->mail($from);

$smtp->to($to);

$smtp->data();

$smtp->datasend("Subject: $sub");

$smtp->datasend("\n");

$smtp->datasend("$content");

$smtp->dataend();

$smtp->quit();

}

sub logit {

my $s = shift;

my $logtimestamp = strftime("%d-%m-%Y -- %H:%M:%S", localtime());

print "$s\n";

my $fh;

open($fh, '>>', "$log") or die "$log: $!";

print $fh "[$logtimestamp] : $s\n";

close($fh);

}

grab pattern from start and end block -- input record separator

usage of local $/ or $INPUT_RECORD_SEPARATOR

INPUT script

start

name:agnello

dob:2 april

address:123 street

end

start

name:babit

dob:13 april

address:3 street

end

start

name:ganesh

dob:1 april

address:23 street

end

i need to get the OUTPUT data in the following format

name:agnello, dob:23 april ,address:123 street

name:babit,dob:13 april,address:3 street

name:ganesh,dob:1 april,address:23 street

i came up with this , is there a better way to do this :

===============================

#!/usr/bin/perl

use strict;

use warnings;

open my $FH , 'data.txt' or die "cannot open file $!";

read $FH, my $string, -s $FH;

close($FH);

my @string = split ( /start/ , $string ) ;

my %data;

foreach ( @string ) {

chomp;

next if /^$/ ;

s/^ $//g;

s/end//;

my @data = split(/\n/, "$_");

foreach my $i (@data) {

print "$i,";

}

print "\n";

}

alternate script is :

========================

#!/usr/bin/env perl

use strict;

use warnings;

# --------------------------------------

use charnames qw( :full :short );

use English qw( -no_match_vars ); # Avoids regex performance penalty

# --------------------------------------

sub read_record {

my $record_fh = shift @_;

# all records are from "start\n" to "end\n"

local $INPUT_RECORD_SEPARATOR = "end\n";

my $record = <$record_fh>;

# $record undef at end of file

if( defined $record ){

# removes "end\n"

chomp $record;

print "$record";

exit ;

# removes "start\n"

$record =~ s{ \A start \s* }{}msx;

}

return $record;

}

# --------------------------------------

while( my $record = read_record( *DATA )){

# remove trailing "\n"

chomp $record;

# replace field separators with commas

$record =~ s{ \n+ }{,}gmsx;

# display the record

print "$record\n";

}

__DATA__

start

name:agnello

dob:2 april

address:123 street

end

start

name:babit

dob:13 april

address:3 street

end

start

name:ganesh

dob:1 april

address:23 street

end

Agnello George sent you a video: "AWS Webcast - AWS OpsWorks Continuous Integration Demo"

Agnello George has shared a video with you on YouTube

AWS Webcast - AWS OpsWorks Continuous Integration Demo

by AmazonWebServices

In this session we'll discuss and demonstrate key concepts and design patterns for continuous deployment and integration using technologies like AWS OpsWorks and Chef to enable better control of applications and infrastructures.