What Makes a DevOps
Solid Linux/Unix administration skills.
Strong Puppet or Chef experience (either for Configuration Management
and/or Continuous Delivery)
and
how on repository systems such as Git and SVN are a must.
Can understand the developer needs and propose a Continuous Integrationplan and implement
CI tools (Jenkins, TeamCity, Bamboo).
Can understand the infrastructure needs and design a Continuous Deliveryplan.
Can understand the management needs and create a Release Management plan.
Can easily setup application stacks (LAMP, LEMP).
Can easily setup application server stacks (Tomcat, Webshpere, JBoss).
Can easily integrate applications with services (ActiveMQ, Redis, Memcache).
Know how and when to scale horizontally & vertically.
Can work on different caching levels through modern applications (Varnish, Terracotta EHCache,
Memcache).
Understand the need of DRPs, backup strategies and security audits.
DevOps
Devops
- Infrastructure: Amazon Web Services, VMware, RackSpace, Microsoft Azure, KVM, Heroku, Amazon OpsWorks, Amazon ElasticBeanstalk, Docker, Vagrant
- Web / Application Architecture / Site Performance / Load Balancing: Apache, NGINX, JBoss, Apache Tomcat, Rails, Node.js, DJango, PHP, Amazon CloudFront, Akamai, Amazon AutoScaling, Varnish Software, Memcached, Redis, Amazon ElastiCache, Red Hat Cluster Suite, F5 LTM (Local Traffic Manager) & GTM (Global Traffic Manager), Amazon Elastic Load Balancer, HAProxy
- Authentication, Authorization, and Access Control OpenLDAP:RSA Authentication, Radius Authentication, MFA (Multi-Factor Authentication)/Google Authenticator, Centrify/Powerbroker, SAML, OAuth
- Environment Build and Deployment: Packer, PXE, Foreman, Red Hat Kickstart, Spacewalk, RedHat Satellite Service, Mrepo, Pulp
- Orchestration / Service Discovery: Consul, Registrator, Consul Template, Shipyard, CloudFormation, Terraform
- Configuration Management / Automation: Puppet, Chef, SaltStack, Ansible, Beaker, TestKitchen, ServerSpec, Berkshelf, r10k
- Big Data Operations: Apache Hadoop, Cloudera, MapR, Hortonworks, Kafka, Storm, Oozie, Sqoop, Pig
- Operational Visibility (Service Health Monitoring, Capacity Planning, and Reporting): Elasticsearch, Logstash, Kibana, Splunk, Graphite, Grafana, Cacti, SNMP, Diamond, CollectD, NRPE, Nagios, Sensu, CloudWatch
- Release Management: GIT, SVN, Stash, GitLab, GitHub, BitBucket, Docker Registry Hub / Private Registry, Jenkins, Bamboo, TravisCI
- Message Queuing: RabbitMQ, ZeroMQ, ActiveMQ, Amazon SQS,
- Storage: NFS File Server, iSCSI, Fiber Channel, NAS / SAN, Netapp, 3par, EMC, Multipath / MPIO, device-mapper, PowerPath
- Security: Host based Firewalls (Iptables), SSL VPN (OpenVPN), IPSEC VPN (OpenSwan, LibraSwan), Amazon VPC, IAM, Security Groups & ACLs, SELinux, Auditing and hardening Linux servers
- ITIL Service Operations (Incident, Problem, and Event Management): Design and implement change control solutions, Design and implement disaster recovery solutions, Design and implement business continuity solutions, Root-cause analysis investigation, Linux Kernel performance troubleshooting and tuning, Application performance troubleshooting and tuning
- Backup and Recovery: Tivoli – TSM, HP DataProtector, Bacula, NetBackup, BackupExec, Rsync-based solutions
what is Devops ?
Jobs and Careers
As a software engineer, how do I shift my career to devops?
I work as a Software Engineer ( mainly with java , python , xml ,.. etc ) but I've been engaging lately in the operations part like working with Oracle weblogic , write shell scripting and so . so I've been thinking about instead of entirely leave the development ( this my current case ) instead of leaving my entire experience with the development ( 4 years for that matter ) I can get the best of both world and join the DevOps and maybe use my experience at development and newly added experience with the operations . I need to know how exactly I could do that in the best/software-enginner-mindset-friendly way possible ,
Re-AskFollow33
3 ANSWERS
Jonathan Fenocchi
Jonathan Fenocchi, DevOps Engineer at Bazaarvoice
108 upvotes by Vishal Biyani, William Emmanuel Yu, Quora User, Chris Krough, (more)
I'm going to answer this question as wholly and honestly as I possibly can. Nevertheless, I have filled this answer with an occasional shameless plug. You've been warned.
For starters, I work at www.bazaarvoice.com | Bazaarvoice. We're big fans of cloud. Real big. We're also fans of DevOps. There's a lot of discussion these days about "What is DevOps?" Usually, this term is used to describe Systems Programmers (sometimes called Infrastructure Engineers, Systems Engineers, Operations Engineers or, in the most unfortunate case, Systems Administrators). This is not what DevOps means, but in the context of career development, it carries the connotation of a "modern" Systems Programmer.
So, you're a developer and you want to get in on the ops action. Boy, are you in for a surprise. This isn't about installing Arch Linux and learning to write Perl. There's a place for that kind of thing (a very small, dark place in a very distant corner of the universe), but let's first consider a few things about what DevOps is and what DevOps is not.
What a DevOps role is:
Writing code / software.
Building tools.
Doing the painful things, as often as you can.
Being on the on-call rotation for 2 a.m. production outages.
Infrastructure design.
*NIX. Because Windows freaking sucks. (That said, I use OS X and prefer it to any *NIX distribution, but under the hood, it's *NIX, too.)
Scaling stuff. Scale matters.
Maintenance. Like rebooting that frail vhost with a memory leak that no one's bothered to fix or take ownership of.
Monitoring. Lots of it.
Virtualization.
Agile development methodology.
Software release cycles and management.
Automation. Automation. Automation.
Designing a branch/release strategy for the provided SCM (git, Mercurial, svn, etc).
Metrics / reporting. Goes hand-in-hand with monitoring.
Optimization / tuning.
Load / performance testing and benchmarking, including performance testing of highly complex systems.
Cloud. Really, you don't have to have cloud experience, but it can fundamentally change the way you think about complex systems.
Configuration management. (You've surely heard of Puppet, Chef, Ansible, etc. Yes?)
Security.
Load balancing / proxying. (Of services, systems, components and processes.)
Authentication services.
Command-line fu (like awk.
Package management. Freaking package management.
CI/CIT/CD -- continuous integration, continuous integration testing, and continuous deployment. This is the closest thing to the real meaning of "DevOps" that a Systems Programmer will do.
Databases. All of them. SQL, NoSQL, whatever.
Solid systems expertise. We're talking about the networking stack, how hard disks work, how filesystems work, how system memory works, how CPU's work, and how all these things come together. This is the traditional "operations" expertise you've heard about.
What a DevOps role is not:
Easier than being a software engineer.
Never writing code. I write tons of code.
Installing Linux and never touching your favorite OS again.
Working the third shift. (At least, it shouldn't be; if it is, quit your job and come work with me: Keep Austin Bazaar.)
More "fun" than being a software engineer.
Greenfield. You'll deal with old stuff in addition to new stuff.
A career shift
Here are a few things you should do to begin positioning yourself as a DevOps engineer.
Interview at a company that's hiring DevOps. If you get hired, you'll learn the operations side of things fast. Real fast. Or get fired. If you don't get hired, you'll learn what is still missing from your resume / experience that's preventing you from becoming a full-time DevOps engineer. Incidentally, we're hiring: Keep Austin Bazaar
Tell your boss you want to become a DevOps engineer at your company. Your boss should help you to this end. If he/she does not, quit. Then come work at Bazaarvoice with me and a bunch of other really awesome, super talented engineers working on some really awesome and challenging problems.
Obtain practical experience by using your skills as a software engineer to build tools rather than software. Look at any of the open source projects Netflix has written for examples / ideas.
Learn OpenStack. You can do this on your own time and budget. It's not important whether OpenStack sucks compared to Rackspace Cloud. What's important is that you understand all of the various components and why they are important. Have a wad of cash lying around? Learn Amazon Web Services instead.
Participate in anything your team does involving operations -- deployment, scale, etc. (See list above: "What DevOps is.") If your team doesn't do any of that (i.e., they send artifacts over to Operations and the Operations team does deployment), go over to the Operations team and sit in on a few deployments.
puppet test
Write a puppet master and slave application that does the following.
1: Ensures that all nodes in the system have the following config
a)Nginx is configured, running and has the following config in its home.conf file.
server_name home.somedomin.in 192.168.10.1;
# uWSGI serving Django.
upstream django-somedomin-home {
# Distribute requests to servers based on client IP. This keeps load
# balancing fair but consistent per-client. In this instance we're
# only using one uWGSI worker anyway.
ip_hash;
#server unix:///tmp/home-somedomin-uwsgi.sock;
server localhost:8005;
}
b) DNSMasq is running, it exists and has the following config
address=/home.somedomin.in/192.168.10.1
c) Interfaces file exists, network manager is running and the following config is applied to all Client nodes
# /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
# Example to keep MAC address between reboots
#hwaddress ether DE:AD:BE:EF:CA:FE
# WiFi Example
auto wlan0
iface wlan0 inet static
address 192.168.10.1
netmask 255.255.255.0
network 192.168.1.0
gateway 192.168.1.1
Once you are done with the application. Please push your code to github and share the github account with me.
You will be rated on
1: Code quality
2: Commenting and clarity of thought
3: using libraries to achieve your task.
Questions to answer -
1: If you had to solve one problem in the world, what would that be.
2: If we had to deploy a 1 million nodes network server infrastructure, what do you for see as possible challenges, issues. How would you overcome each? What measures would you put in place to detect such issues before the customer? How would you ensure that somedomin product is running 100% of the time, both online on AWS and offline through the network servers.
Draw out the possible architecture for such a solution and list down the technologies you will chose.
Script to add new contacts automatically( inLinkedin )
This script will do the following things :
1) will get all user url from each page in the linkedinGroup
2) using that user url with will get the form
3) once the form is got , it will fill the form and send sumbit
4) have add multiple user agents in the request headers
==========================================================
#!/usr/bin/perl
use strict;
use warnings;
use WWW::Mechanize;
use LWP;
use JSON;
use Data::Dumper;
use Getopt::Long;
use POSIX qw/ strftime /;
use LWP::Simple;
use HTML::TreeBuilder;
use HTML::TreeBuilder::XPath;
use List::Util qw/shuffle/;
my $username = 'login id ';
my $password = ' password ';
my $maiurl= 'https://www.linkedin.com';
my $loginurl= $maiurl.'/uas/login?goback=&trk=hb_signin';
#my $grp="https://www.linkedin.com/grp/members?gid=124838&page=";# canada group
my $grp="https://www.linkedin.com/grp/members?gid=4504649&page=";# devops group
$|++;
my $mech = login_linkedin();
my $tree = HTML::TreeBuilder::XPath->new_from_content();
my @all_content = get_data($mech,$grp,"1","10");
my @all_new_linkein_users = get_linkedin_user_url($tree,@all_content) ;
send_invitation($mech,@all_new_linkein_users);
sub set_dynamic_user_agent {
my ($mech) = @_;
my @user_agents = (
'Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10; rv:33.0) Gecko/20100101 Firefox/33.0',
'Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0',
'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20130401 Firefox/31.0',
'Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0',
'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0',
'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0',
'Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:24.0) Gecko/20100101 Firefox/24.0'
);
my @shuff_user_agents = shuffle( @user_agents );
# dynamically set custom HTTP User-agents
$mech->add_header( 'User-agent' => $shuff_user_agents[0]);
print sprintf("set User-Agent : %s \n", $shuff_user_agents[0] ) ;
return $mech;
}
sub send_invitation {
my ( $mech,@all_user_url) = @_;
for my $user_url (@all_user_url){
sleep(2);
$mech= set_dynamic_user_agent($mech);
$mech -> get("${maiurl}$user_url");
# print sprintf( "User-Agent : %s\n\n", $mech->uri() );
#<input type="radio" name="reason" value="IF" id="IF-reason-iweReconnect" class="radio-btn">
#<input type="submit" name="iweReconnectSubmit" value="Send Invitation" id="send-invite-button" class="btn-primary"/
#$mech ->click_button( name => "iweReconnectSubmit" );
if ( $mech->form_name("iweReconnect") ) {
print "sending invitation : $user_url \n";
$mech ->set_fields( 'reason' => 'IF' );
$mech->submit();
$mech->success or die "form POST fail";
} elsif ( $mech->form_name("invitation")) {
print " invitation was already sent to $user_url sent \n" ;
next ;
}else {
print "No From matched $user_url\n ";
next;
}
}
}
sub get_linkedin_user_url {
my ($tree,@all_content) = @_;
my $all_new_linkein_users;
for my $content (@all_content) {
$tree->parse($content);
my @newconn = $tree->findnodes('//li[@class="member"');
print "getting url that which is not in my connection \n";
for my $custm (@newconn){
my @all_a_tag = $custm->look_down('_tag' => 'a' ) ;
my $count = @all_a_tag;
my $invte_link ;
if ( $all_a_tag[$count - 1]->attr('href') =~ m/people/ig ) {
$invte_link = $all_a_tag[$count - 1]->attr('href') ;
} else {
next;
}
print $invte_link ." ---- \n";
push ( @all_new_linkein_users, "$invte_link") ;
exit unless(@all_new_linkein_users);
}
}
return (@all_new_linkein_users);
}
sub get_data {
my ( $mech,$url,$spage,$epage ) = @_;
my @all_content;
print "Getting the content form page $spage to page $epage\n";
for my $page ($spage .. $epage ) {
print " ----------- ${url}$page -----------------"."\n";
sleep(1) ;
$mech -> get ("${url}$page");
push ( @all_content, $mech->content()) ;
}
return (@all_content);
}
sub login_linkedin {
my $mech = WWW::Mechanize->new();
$mech -> cookie_jar(HTTP::Cookies->new());
$mech -> get("$loginurl");
$mech -> form_id('login');
$mech -> field ('session_key' => $username);
$mech -> field ('session_password' => $password);
$mech -> click_button (value => 'Sign In');
return $mech ;
}
============================================================
python script to fetch gmail ids and script to send out mail with attachment
-----------------------------------------------------------------------------------------------------------------
script to fetch gmail ids
------------------------------------------------------------------------------------------------------------------
import imaplib, sys, email
mail = imaplib.IMAP4_SSL('imap.gmail.com')
mail.login('gmail@gmail.com', 'password')
mail.list()
# Out: list of "folders" aka labels in gmail.
mail.select("inbox") # connect to inbox.
result, data = mail.search(None, '(HEADER From "info@monsterindia.com")')
for num in reversed(data[0].split()):
rv, data = mail.fetch(num, '(RFC822)')
raw_email = data[0][1] # here's the body, which is raw text of the whole email
# including headers and alternate payloads
email_message = email.message_from_string(raw_email)
if email_message['Reply-To'] in ('naukrialerts@naukri.com', None):
continue
print email_message['Reply-To']
-----------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------
script to send out mail with attachment
----------------------------------------------------------------------------------------------------------------#!/usr/bin/python
import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEBase import MIMEBase
from email.MIMEText import MIMEText
from email import Encoders
import os,re
gmail_user = "some@gmail.com"
gmail_pwd = "passowrd"
def mail(to, subject, attach, recruitername ):
msg = MIMEMultipart()
msg['From'] = gmail_user
msg['To'] = to
msg['Subject'] = subject
text = """
Hi %s
balah
balah
balah
balah
Regards
balah
balah
""" % recruitername
msg.attach(MIMEText(text))
part = MIMEBase('application', 'octet-stream')
part.set_payload(open(attach, 'rb').read())
Encoders.encode_base64(part)
part.add_header('Content-Disposition',
'attachment; filename="%s"' % os.path.basename(attach))
msg.attach(part)
mailServer = smtplib.SMTP("smtp.gmail.com", 587)
mailServer.ehlo()
mailServer.starttls()
mailServer.ehlo()
mailServer.login(gmail_user, gmail_pwd)
mailServer.sendmail(gmail_user, to, msg.as_string())
# Should be mailServer.quit(), but that crashes...
mailServer.close()
myemailfile = "/scripts/cyber_send"
emaillist = open( myemailfile, 'r')
for i in emaillist.readlines():
match = re.search(r'[\w\.-]+@[\w\.-]+', i)
if not match:
continue
print "%s " % match.group(0)
hrname = str(re.split("\.|_|-", match.group(0).split('@')[0] )[0])
#hrname = ''
mail( match.group(0) , "Sr Engineer ( information technology )",
"/scripts/resume_.pdf" , hrname )
-------------------------------------------------------------------------------------------------------------------
whats app plugin for Nagios
If you want to send alerts to your phone on watsapp this is one way .
apt-get install python python-dateutil python-argparse python-dev python-setuptools
git clone https://github.com/tgalal/yowsup
python install setup.py
cd yowsup
./yowsup-cli registration --requestcode sms --phone 91yournumber --cc 91 --mcc 404 --mnc 20
./yowsup-cli registration --register 671587 --phone 91yournumber --cc 91
cat yowsup-cli.config
cc=91
phone=yournumber
id=yourimeanumber
password=yourpassword
./yowsup-cli demos -c yowsup-cli.config -s tonumber " saleeeeee"
Nagios Plugin config :
cat /etc/nagios3/commands.cfg
define command{
command_name notify-service-by-watsapp
command_line /usr/lib/nagios/plugins/whatsapp.sh --numbers $_CONTACTDEVOPSPHONE$ --msg "***** Nagios : $NOTIFICATIONTYPE$ ***** \n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$ "
}
cat /etc/nagios3/contacts.cfg
define contact{
contact_name admin
alias root
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,u,c,r
host_notification_options d,r
service_notification_commands notify-service-by-email,notify-service-by-watsapp
host_notification_commands notify-host-by-email
_devopsphone 9819031945
email youremailid
}
cat /usr/lib/nagios/plugins/whatsapp.sh
#!/bin/bash
while [ "$1" != "" ]
do
case "$1" in
--numbers) numbers=$2; shift;;
--msg) msg=$2; shift;;
*) echo "$0: invalid option $1" >&2
echo "Usage: $0 --numbers 9819031911 --msg \" test mgs\"" >&2
exit 1
esac
shift
done
ph_nums=($(echo $numbers | sed 's/,/ /g' ))
prntmsg=$(printf "$msg")
for i in ${ph_nums[@]}
do
sudo /opt/yowsup/yowsup/yowsup-cli demos -c /opt/yowsup/yowsup/yowsup-cli.config -s 91${i} "$prntmsg"
echo "$i $msg"
done
send email using MIME::Lite::TT::HTML
#!/usr/bin/perl
use strict;
use warnings;
use MIME::Lite::TT::HTML;
email ('agnello','agnello.dsouza@gmail.com');
sub email {
my ( $recruiter_name, $to ) = @_;
my %options;
$options{INCLUDE_PATH} = '/scripts';
my %params;
$params{recruiter_name} = $recruiter_name ;
my $msg = MIME::Lite::TT::HTML->new(
From => 'agnello.dsouza@gmail.com' ,
To => $to,
Subject => 'Some subject ',
Template => {
html => 'resume.tt',
text => 'text.resume.tt',
},
TmplOptions => \%options,
TmplParams => \%params,
);
# Set our content type properly
$msg->attr("content-type" => "multipart/mixed");
# Attach a PDF to the message
$msg->attach( Type => 'application/pdf',
Path => '/scripts',
Filename => 'Agnello_dsouza.pdf',
Disposition => 'attachment'
);
# $msg->send('smtp', "smtp.gmail.com", SSL=>1, AuthUser=>'agnello.dsouza.linux@gmail.com', AuthPass=>'xxxxxxxxxx', Debug=>1);
$msg->send();
}
HACK : script to get all linkedin Contact email IDs -- part 1
#!/usr/bin/perl
use strict;
use warnings;
use WWW::Mechanize;
use LWP;
use JSON;
use Data::Dumper;
use Getopt::Long;
use POSIX qw/ strftime /;
my $username = 'agnello.dsouza';
my $password = 'xxxxxxxxxxxxxx';
my $maiurl= 'https://www.linkedin.com';
my $loginurl= $maiurl.'/uas/login?goback=&trk=hb_signin';
my $all_data = 'https://www.linkedin.com/contacts/api/contacts/?fields=id%2Cname&sort=-last_interaction';
my $json_file = '/tmp/json.data';
my $mech = login_linkedin();
#my $jdata = get_jdata($mech,$all_data);
#write_to_file($jdata,$json_file);
my $jdata = read_from_file($json_file);
my @contacts = gen_url ($jdata);
foreach (@contacts){
my $fnd_details_json = get_jdata($mech,$_);
my $email = $fnd_details_json->{'contact_data'}->{'emails_extended'}->[0]->{'email'};
next unless ( $email);
print "$email\n"
}
sub gen_url {
my $jdata= shift;
my @url ;
for my $i (@{$jdata->{'contacts'}} ){
my $uri = $maiurl.'/contacts/api/contacts/'.$i->{'id'}.'/?fields=name,emails_extended,birthday,phone_numbers,sites,addresses,company,title,location,ims,profiles,twitter,wechat,display_sources';
push (@url, $uri );
}
return @url;
}
sub get_jdata {
my ( $mech,$url ) =@_;
$mech -> get ("$url");
return decode_json($mech ->content()) ;
}
sub login_linkedin {
my $mech = WWW::Mechanize->new();
$mech -> cookie_jar(HTTP::Cookies->new());
$mech -> get("$loginurl");
$mech -> form_id('login');
$mech -> field ('session_key' => $username);
$mech -> field ('session_password' => $password);
$mech -> click_button (value => 'Sign In');
return $mech ;
}
sub read_from_file {
my $file = shift;
my $json;
{
local $/; #Enable 'slurp' mode
open my $fh, "<", "$file";
$json = <$fh>;
close $fh;
}
return decode_json($json);
}
sub write_to_file {
my $data = shift;
my $file = shift;
open my $fh, ">", "$file";
print $fh encode_json($data);
close $fh;
}
perl script to send download file from website - integration with Jenkins- using Machanize
#!/usr/bin/perl
#use strict;
#use warnings;
use WWW::Mechanize;
use LWP;
use JSON;
use Data::Dumper;
use Getopt::Long;
use POSIX qw/ strftime /;
$| = 1;
my $username = 'something@something.com';
my $password = 'xxxxxx';
my $loginurl= 'https://loginurl';
my $insighturl= 'https://loginurl';
my %region_Squeryurl= ( 'apac' => '/cloud/paths/download/prod/slow-query/9538?s=1767176',
'na' => '/cloud/paths/download/prod/slow-query/8972?s=1767151',
'emea' => '/cloud/paths/download/prod/slow-query/3664?s=670131');
my $region;
GetOptions(
'region|r=s' => \$region );
die 'usage: $0 -r [apac|na|emea]' unless( $region_Squeryurl{$region} );
my $date = my $time = strftime("%Y%m%d", localtime());
#my $log_dir = "$ENV{JENKINS_HOME}/docroot/getlog/";
my $log_dir = "/mnt/getlog/";
my $fullpath= "$log_dir"."$region/"."$date/";
unless (-d $fullpath ){
print "creating $fullpath ";
system( "sudo mkdir -p $fullpath" ) ;
system( "sudo chown -R www-data:www-data $fullpath" ) ;
}
my $mech = WWW::Mechanize->new();
$mech -> cookie_jar(HTTP::Cookies->new());
download_file ($username,$password,$loginurl,$insighturl,$region_Squeryurl{$region},$date,$fullpath);
sub download_file {
my ($username,$password,$loginurl,$insighturl,$region_Squeryurl,$date,$fullpath)=@_;
$mech -> get("$loginurl");
$mech -> form_id('user-login');
$mech -> field ('name' => $username);
$mech -> field ('pass' => $password);
$mech -> click_button (value => 'Sign in');
$mech -> get ("$insighturl$region_Squeryurl");
my $jdata_1 = decode_json($mech ->content());
$jdata_1 = $jdata_1->[1]->{'url'};
$jdata_1 =~ s/'//g;
print "$insighturl$jdata\n";
while (1){
sleep(2);
$mech -> get ("$insighturl$jdata_1");
$jdata = decode_json($mech ->content());
print "get url $insighturl$jdata_1\n";
if ( $jdata->[3]->{'delay'} =~ /^0$/ ) {
print "mnt url --> ".$jdata->[3]->{'url'}."\n";
print "delay --->". $jdata->[3]->{'delay'}."\n";
last;
}
}
my $D_url = $jdata->[3]->{'url'}."\n";
$mech -> get ("$D_url");
$mech->save_content( "/tmp/mysql-slow-log-$date.$$.log" );
system("sudo mv /tmp/mysql-slow-log-$date.$$.log ${fullpath}mysql-slow-log-$date.$$.log" );
print $mech->content_type()."\n";
if ($mech ->status()){
print "created successfully ".$fullpath."mysql-slow-log-$date.$$.log\n ";
my $msql_slow_log_q=$fullpath."mysql-slow-log-$date.$$.log";
system("/bin/bash $ENV{WORKSPACE}/scripts/pt-query-digest.sh $region $msql_slow_log_q ");
}
}
#print $mech ->status();
#print $mech ->content();
#print $mech-> content();
logstash - ELK
input {
file {
path => [ "/var/log/apache2/example.com_access_log",
"/var/log/apache2/domain.com_access_log",
"/var/log/apache2/somesite.com_access_log",
"/var/log/apache2/access.log"
]
type => "apache-access-log"
}
}
filter {
if [type] == "apache-access-log" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
logstash.conf
output {
stdout {
codec => rubydebug
}
elasticsearch {
bind_host => "127.0.0.1"
cluster => "elasticsearch"
host => "127.0.0.1"
}
}
web scraping part 1
This script download all emails of all recruiters.
=============================================
using simple regex.
=============================================
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
use LWP::Simple;
my $url = 'http://www.cybercoders.com/recruiter/';
my $html = get "$url";
my @html = split(/\n/,$html);
for my $i (@html){
next unless $i =~ /<a href="\/recruiter\/(\S.*\/)">/ ;
print "${url}${1}\n";
my $indiv_recruiter = get("$url$1");
my @indiv_recruiter = split(/\n/,$indiv_recruiter);
for my $j (@indiv_recruiter ){
next unless $j =~ /<a class="recruiter-email-link" href="mailto:(\S.*?\@CyberCoders\.com)">/ ;
print "$1\n"
}
}
=========================================================
Using HTM::treebuilder::Xpath
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
use LWP::Simple;
use HTML::TreeBuilder;
use HTML::TreeBuilder::XPath;
my $url = 'http://www.cybercoders.com/recruiter/';
my $html = get "$url";
my $tree = HTML::TreeBuilder::XPath->new_from_content();
$tree->parse($html);
my @customers = $tree->findnodes('//div[@class="recruiter-item"');
for my $custm (@customers){
my @all_a_tag = $custm->look_down('_tag' => 'a' ) ;
my $recruiter_pth = $all_a_tag[0]->attr('href') ."\n" ;
$recruiter_pth =~ s/\/recruiter\///g;
get_email($recruiter_pth);
}
sub get_email {
my ($r_pth)=@_;
my $html = get ("$url$r_pth") ;
my $tree2 = HTML::TreeBuilder->new_from_content($html);
my @mailto = $tree2->look_down( _tag => 'a' , class => 'recruiter-email-link' );
print $mailto[0]->attr('href')."\n";
print "get $url$r_pth";
}
Cloud developer Chef
> The sender of this email is registered with Naukri.com as dynproindia. To respond back directly to the Employer, please click on Reply button, or send an email to savitha@dynproindia.com
> Experience required for the Job: 5 - 15 years
> Annual Salary of the Job: 0.0 - 30.0 Lacs
>
> Apply Now Reply
>
>
> Dear Candidate,
>>
>> Hi All
>>
>> About Dynpro
>> * DynPro specializes in helping companies build infrastructures that can weather the most fierce storms by focusing on strong fundamentals: appropriate and effective systems, the right people, and proven business processes. Our worldwide force of technology and business experts can identify the right solutions to meet any challenge. Our relationships with the world's best service and product providers ensures that you get the most practical, value-driven solutions available anywhere. Location: USA,UK, Kolkata,Bangalore,Noida
>>
>> Skill sets -Cloud,Chef,AWS
>>
>> Experience: 5+ Yrs
>>
>> Location: Bangalore
>>
>> NP: Who can join Immediate or within 15 to 20 days of time
>> JD
>>
>> Following is the detailed Job description:
>> 1. Hands on experience (preferably 2 years) on Chef and Ruby scripting.
>> 2. Aware of cloud technologies such as AWS, Azure.
>> 3. Experience on working on DevOps methodology.
>> 4. Good to have - shell, python scripting hands on.
>> 5. Technically strong with a min of 5 years of experience.
>> 6. Self-motivated, team player and ability to work in high pressure scenarios.
>> 7. Capable of taking instructions and action them without too much hand holding.
>> 8. Show technical innovation and strive for continuous improvement on the platform.
>>
>> Key skills required for the job are:
>>
>> Devops-Chef(Mandatory)
>> Amazon Web Services - AWS(Mandatory)
>> ServiceNow - SaaS
>>
>>
>>
>> Mode Of Employment: Contract to Hire
>>
>> If your profile is relevant to the JD then kindly revert me back with your updated resume and with below Details
>>
>> Total Exp :
>> Rel Exp :
>> Current CTC :
>> Expected CTC :
>> Notice Period :
>> Work Location :
>> Pan no:
>> Passport no:
>> Educational Background with University name and Year of Passing and with %
>>
>> Thanks & Regards,
>>
>> Savita
>> Dynpro India Pvt Ltd.
>> Bangalore
>> Email: savitha@dynproindia.com
>> URL: www.dynproindia.com ; www.dynpro.com
>
>
> Is this job relevant to you? Yes No
> Your feedback would help us in sending you the most relevant job opportunities
> Disclaimer:
> The sender of this email is registered with naukri.com as dynproindia (venkatesh@dynpro.com, #3793, 7th Main HAL 2nd Stage Indiranagar, 121, Dickenson Road,, BANGALORE, Karnataka - 560038) using Naukri.com services. The responsibility of checking the authenticity of offers/correspondence lies with you. If you consider the content of this email inappropriate or spam, you may: Forward this email to: compliance@naukri.com or you can Block this Company from searching your resume in the database.
> Advisory: Please do not pay any money to anyone who promises to find you a job. This could be in the form of a registration fee, or document processing fee or visa charges or any other pretext. The money could be asked for upfront or it could be asked after trust has been built after some correspondence has been exchanged. Also please note that in case you get a job offer or a letter of intent without having been through an interview process it is probably a scam and you should contact compliance@naukri.com for advise.
> Update Your Resume Now to be on the top of Employer's Resume Search
> Create your own job alert to receive similar jobs directly in your inbox.
> Your Resume has only 30 seconds to impress a Recruiter. Get your resume written by Experts.
> Call 1800-102-5557 now! (Toll-Free)
Devops professionals with chef automation
>>
>> Experienced DevOps Consultant working in an Agile Environment (Scrum) and specialized in design, implementation, administration, configuration, upgrade and maintenance of Chef, Clearcase, ClearQuest, TFS, JIRA, Splunk, VMware ESX, NetApp.Storage, Flexlm License Server, Perforce, Tableau, and BladeLogic. Strong hands on experience in Chef, docker, AWS, Openstack, Mysql.
>> Good experience in automation using Unix shell scripting, PowerShell, PowerCLI and Perl. Experienced in delivering virtualised environments for multiple applications like TFS, Splunk, Bladelogic.. Good experience in Build and Release activities. Excellent interpersonal skills and ability to communicate effectively with all levels of management.A quick learner with the confidence to master new applications and technologies in order to deliver results.
>>
>> * Help companies setup Enterprise Chef Environment and conducted demos/PoCs.
>> * Enable macbook provisioning for a using Chef.
>> * Write Chef cookbooks for various requirements from clients.
>> * Write Ohai plugins, chef handlers.
>> * Work on Chef cookbook testing using foodcritic, chefspec, rubocop, Test Kitchen, Vagrant.
>> * Write cookbooks to create Docker containers using Chef.
>> * Design and implement complex and efficient Splunk dashboards using Splunk DB Connect and Sideview Utils.
>> * Involve in the administration of AWS, Openstack and Cloudstack setups.
>> * Setup of MySQL database, importing data from external sources into MySQL
>>
>>.
Senior Puppet Engineer
>
> Job Description:
>
> A candidate with good DevOps exposure and expertise on managing Puppet infrastructure on various cloud platforms (preferably AWS). He/She should have an excellent working knowledge of Linux with Puppet Master/Client setup.
>
> Desired Skills & Experience
> Strong hands-on experience on various• linux platforms. Good knowledge of linux shell scripting for automation of regular activities.
> Extensive experience in• DevOps and Puppet Master/Client setup.
> Proficient in deploying• Puppet, Puppet Dashboard, and Puppet DB for configuration management to the existing infrastructure.
> Experience in configuring parameters in Puppet for• different deployments.
> Experience in enterprise software monitoring tools• like Nagios, Cacti, OpsView, ManageEngine.
> Experience working with• scripting/programming languages (shell, PHP, Ruby, Python and/or Perl)
> Knowledge in configuring Puppet Foreman.•
> Knowledge in Puppet Hiera.•
> Understanding of cloud computing e.g.• AWS, Azure, and Google etc. will be an advantage.
> • Puppet Certified Professional will be an advantage.
> Monitoring: Nagios, Cacti, ManageEngine•
devops
>> 1Very Strong Linux/Unix administration background expertise in Scripting.
>> 2. Responsible for Design manage and maintain tools to automate operational processes using Chef Puppet Ansible and SaltStack etc.
>> 3. Implement systems that are highly available scalable and self-healing system and & IT Operations.
>> 4. Understand implement and automate security controls and governance processes.
>> 5. Build and manage development and testing environments assisting developers in debugging application issues.
>> 6. Ability to Use wide verity of open source technologies and tools.
>> 7.Contribute to the creation of system support documents such as run books Mandatory Skill Sets
>> 8. Experience with networking properly architecting high availability systems and security in an in premises and Cloud context
amazon training
Overview
We are developing a large scale website and are looking to automate the environment setup and parts of the software engineering process, build, package, deploy, and test within AWS.
- For this test assignment, we are asking you to provide a development plan document for completing each of the tasks detailed below.
- We will use your document to evaluate your knowledge of AWS and other software packages for build, package, deploy, and test
- We are expecting that a great candidate will spend less than 8 hours and produce a clear and well researched dev plan.
- If we like your plan, and your interview goes well, we will then hire you for a full time role, and your first tasks will be to execute your implementations!
Task 1 - Fully Automated environment setup
Create the cloudformation template and chief scripting required to bring up a dev, test, staging, or production web site
- Must be fully automated; no manual intervention required whatsoever
- web site AWS environment consists of RDS, Cloudfront, SQS, SNS, SES, and elastic beanstalk as well as other third party non-Amazon software.
- must notify when environment is completely setup ready
- DOD (Definition of Done) - this task is complete when I can specify a few parameters (examples, name of environment, size of farm), click a button, and receive an email notification containing the details of the environment when the environment is up and running.
Task 2 - Continuous 24/7 Testing Automation
- create and administer a 24/7 test suite that runs against a selected environment
- setup fully automated testing pipeline that detects SVN commits, builds and packages the website, then deploys it into the staging environment, without disrupting the automated testing
- requires using existing monitoring tools
- DOD - this task is complete when the following 2 capabilities are enabled
- an svn commit command triggers a build, package, deploy into a configured environment, and then an email notification is received containing details of the deployment
- a web page is available at all times that shows the results of all tests run in the last 8 hours against the configured environment
Instructions
When complete and ready to submit:
- Please create and provide two (2) distinct plans. Both plans should be contained in a single Google document or folder, saved with the file name ACAE$60K1003_Lastname_Firstname (substituting your name for Lastname_Firstname) share it publicly, and proceed to this link for completing your submission.
Each plan should contain the following sections
Plan Summary
- Define in one paragraph or 5 bullet points the most important parts of your plan.
- Have you selected great services that will do the work for you without having to write much code? Is there a design pattern you will use that will make the automation maintainable and more robust? This is the section to highlight what makes your plan special.
Software List
- Identify all software you will use, including development tools
- You must reuse Amazon tools and best practices when they apply
- When Amazon tools and best practices don't apply, open source standards should be used.
- Writing code should only been done as a last resort. Configuring "off the shelf" software is preferable by far to writing code or frameworks.
Task Breakdown
- Plan should cover the task breakdown to the level where the maximum task effort is 3 man hours.
- use a Google spreadsheet and link to it from the Google doc
- make sure it shared with the same people
- provide the task breakdown in a form that you feel is as simple and easy to understand as possible
Total Effort
- Each plan should include the total effort in man-hours. Assume you are the only one working on each project in your estimates.
Calendar Duration
- Each plan should include the calendar duration-to-complete, measured in weeks and days. Again assume that you are the only one working each project
Finally, if we like your plan, you will be invited to the final phase of the application process. [NOTE: You may be asked to sign legal documents related to security and proprietary information before you move forward to the next step of the process. This is sometimes necessary if, in the next application phase, you will have access to such information for completing this phase.] The next phase for this position is an online video interview with the hiring manager. Instructions for scheduling this interview will be included in the invitation, should you successfully complete this testing phase. Once that interview is complete, only minor (and brief) administrative steps remain.
You have only 5 days from today to complete this testing phase.
steps how to invest your money
1. Follow the money. In today's economic environment you cannot save your way to millionaire status. The first step is to focus on increasing your income in increments and repeating that.
My income was $3,000 a month and nine years later it was $20,000 a month. Start following the money and it will force you to control revenue and see opportunities.
2. Don't show off. Show up! I didn't buy my first luxury watch or car until my businesses and investments were producing multiple secure flows of income. I was still driving a Toyota Camry when I had become a millionaire. Be known for your work ethic, not the trinkets that you buy.
3. Save to invest, don't save to save. The only reason to save money is to invest it. Put your saved money into secured, sacred (untouchable) accounts. Never use these accounts for anything, not even an emergency. This will force you to continue to follow step one (increase income). To this day, at least twice a year, I am broke because I always invest my surpluses into ventures I cannot access.
4. Avoid debt that doesn't pay you. Make it a rule that you never use debt that won't make you money. I borrowed money for a car only because I knew it could increase my income. Rich people use debt to leverage investments and grow cash flows. Poor people use debt to buy things that make rich people richer.
5. Treat money like a jealous lover. Millions wish for financial freedom, but only those that make it a priority have millions. To get rich and stay rich you will have to make it a priority. Money is like a jealous lover. Ignore it and it will ignore you, or worse, it will leave you for someone who makes it a priority.
6. Money doesn't sleep. Money doesn't know about clocks, schedules, or holidays, and you shouldn't either. Money loves people that have a great work ethic. When I was 26 years old, I was in retail and the store I worked at closed at 7 pm. Most times you could find me there at 11 pm making an extra sale. Never try to be the smartest or luckiest person — just make sure you outwork everyone.
7. Poor makes no sense. I have been poor, and it sucks. I have had just enough and that sucks almost as bad. Eliminate any and all ideas that being poor is somehow OK. Bill Gates has said, "If you're born poor, it's not your mistake. But if you die poor, it is your mistake."
8. Get a millionaire mentor. Most of us were brought up middle class or poor and then hold ourselves to the limits and ideas of that group. I have been studying millionaires to duplicate what they did. Get your own personal millionaire mentor and study them. Most rich people are extremely generous with their knowledge and their resources.
9. Get your money to do the heavy lifting. Investing is the Holy Grail in becoming a millionaire and you should make more money off your investments than your work. If you don't have surplus money you won't make investments. The second company I started required a $50,000 investment. That company has paid me back that $50,000 every month for the last 10 years.
My third investment was in real estate, where I started with $350,000, a large part of my net worth at the time. I still own that property today and it continues to provide me with income. Investing is the only reason to do the other steps, and your money must work for you and do your heavy lifting.
10. Shoot for $10 million, not $1 million. The single biggest financial mistake I've made was not thinking big enough. I encourage you to go for more than a million. There is no shortage of money on this planet, only a shortage of people thinking big enough.
Apply these 10 steps and they will make you rich. Steer clear of people that suggest your financial dreams are born of greed. Avoid get-rich-quick schemes, be ethical, never give up, and once you make it, be willing to help others get there too.
My income was $3,000 a month and nine years later it was $20,000 a month. Start following the money and it will force you to control revenue and see opportunities.
2. Don't show off. Show up! I didn't buy my first luxury watch or car until my businesses and investments were producing multiple secure flows of income. I was still driving a Toyota Camry when I had become a millionaire. Be known for your work ethic, not the trinkets that you buy.
3. Save to invest, don't save to save. The only reason to save money is to invest it. Put your saved money into secured, sacred (untouchable) accounts. Never use these accounts for anything, not even an emergency. This will force you to continue to follow step one (increase income). To this day, at least twice a year, I am broke because I always invest my surpluses into ventures I cannot access.
4. Avoid debt that doesn't pay you. Make it a rule that you never use debt that won't make you money. I borrowed money for a car only because I knew it could increase my income. Rich people use debt to leverage investments and grow cash flows. Poor people use debt to buy things that make rich people richer.
5. Treat money like a jealous lover. Millions wish for financial freedom, but only those that make it a priority have millions. To get rich and stay rich you will have to make it a priority. Money is like a jealous lover. Ignore it and it will ignore you, or worse, it will leave you for someone who makes it a priority.
6. Money doesn't sleep. Money doesn't know about clocks, schedules, or holidays, and you shouldn't either. Money loves people that have a great work ethic. When I was 26 years old, I was in retail and the store I worked at closed at 7 pm. Most times you could find me there at 11 pm making an extra sale. Never try to be the smartest or luckiest person — just make sure you outwork everyone.
7. Poor makes no sense. I have been poor, and it sucks. I have had just enough and that sucks almost as bad. Eliminate any and all ideas that being poor is somehow OK. Bill Gates has said, "If you're born poor, it's not your mistake. But if you die poor, it is your mistake."
8. Get a millionaire mentor. Most of us were brought up middle class or poor and then hold ourselves to the limits and ideas of that group. I have been studying millionaires to duplicate what they did. Get your own personal millionaire mentor and study them. Most rich people are extremely generous with their knowledge and their resources.
9. Get your money to do the heavy lifting. Investing is the Holy Grail in becoming a millionaire and you should make more money off your investments than your work. If you don't have surplus money you won't make investments. The second company I started required a $50,000 investment. That company has paid me back that $50,000 every month for the last 10 years.
My third investment was in real estate, where I started with $350,000, a large part of my net worth at the time. I still own that property today and it continues to provide me with income. Investing is the only reason to do the other steps, and your money must work for you and do your heavy lifting.
10. Shoot for $10 million, not $1 million. The single biggest financial mistake I've made was not thinking big enough. I encourage you to go for more than a million. There is no shortage of money on this planet, only a shortage of people thinking big enough.
Apply these 10 steps and they will make you rich. Steer clear of people that suggest your financial dreams are born of greed. Avoid get-rich-quick schemes, be ethical, never give up, and once you make it, be willing to help others get there too.
Subscribe to:
Posts (Atom)