devops tool need to look into

dashboards 

https://razorflow.com/tour/

https://github.com/capitalone/Hygieia

What Makes a DevOps

 Solid Linux/Unix administration skills.

 Strong Puppet or Chef experience (either for Configuration Management 

and/or Continuous Delivery) 

and

  how on repository systems such as Git and SVN are a must.

 Can understand the developer needs and propose a Continuous Integrationplan and implement 

CI tools (Jenkins, TeamCity, Bamboo).

 Can understand the infrastructure needs and design a Continuous Deliveryplan.

 Can understand the management needs and create a Release Management plan.

 Can easily setup application stacks (LAMP, LEMP).

 Can easily setup application server stacks (Tomcat, Webshpere, JBoss).

 Can easily integrate applications with services (ActiveMQ, Redis, Memcache).

 Know how and when to scale horizontally & vertically.

 Can work on different caching levels through modern applications (Varnish, Terracotta EHCache, 

Memcache).

 Understand the need of DRPs, backup strategies and security audits.

DevOps

Devops 

• Infrastructure: Amazon Web Services, VMware, RackSpace, Microsoft Azure, KVM, Heroku, Amazon OpsWorks, Amazon ElasticBeanstalk, Docker, Vagrant
• Web / Application Architecture / Site Performance / Load Balancing: Apache, NGINX, JBoss, Apache Tomcat, Rails, Node.js, DJango, PHP, Amazon CloudFront, Akamai, Amazon AutoScaling, Varnish Software, Memcached, Redis, Amazon ElastiCache, Red Hat Cluster Suite, F5 LTM (Local Traffic Manager) & GTM (Global Traffic Manager), Amazon Elastic Load Balancer, HAProxy
• Authentication, Authorization, and Access Control OpenLDAP:RSA Authentication, Radius Authentication, MFA (Multi-Factor Authentication)/Google Authenticator, Centrify/Powerbroker, SAML, OAuth
• Environment Build and Deployment: Packer, PXE, Foreman, Red Hat Kickstart, Spacewalk, RedHat Satellite Service, Mrepo, Pulp
• Orchestration / Service Discovery: Consul, Registrator, Consul Template, Shipyard, CloudFormation, Terraform
• Configuration Management / Automation: Puppet, Chef, SaltStack, Ansible, Beaker, TestKitchen, ServerSpec, Berkshelf, r10k
• Big Data Operations: Apache Hadoop, Cloudera, MapR, Hortonworks, Kafka, Storm, Oozie, Sqoop, Pig

• Operational Visibility (Service Health Monitoring, Capacity Planning, and Reporting): Elasticsearch, Logstash, Kibana, Splunk, Graphite, Grafana, Cacti, SNMP, Diamond, CollectD, NRPE, Nagios, Sensu, CloudWatch
• Release Management: GIT, SVN, Stash, GitLab, GitHub, BitBucket, Docker Registry Hub / Private Registry, Jenkins, Bamboo, TravisCI
• Message Queuing: RabbitMQ, ZeroMQ, ActiveMQ, Amazon SQS,
• Storage: NFS File Server, iSCSI, Fiber Channel, NAS / SAN, Netapp, 3par, EMC, Multipath / MPIO, device-mapper, PowerPath
• Security: Host based Firewalls (Iptables), SSL VPN (OpenVPN), IPSEC VPN (OpenSwan, LibraSwan), Amazon VPC, IAM, Security Groups & ACLs, SELinux, Auditing and hardening Linux servers
• ITIL Service Operations (Incident, Problem, and Event Management): Design and implement change control solutions, Design and implement disaster recovery solutions, Design and implement business continuity solutions, Root-cause analysis investigation, Linux Kernel performance troubleshooting and tuning, Application performance troubleshooting and tuning
• Backup and Recovery: Tivoli – TSM, HP DataProtector, Bacula, NetBackup, BackupExec, Rsync-based solutions

what is Devops ?

Jobs and Careers

As a software engineer, how do I shift my career to devops?

I work as a Software Engineer ( mainly with java , python , xml ,.. etc ) but I've been engaging lately in the operations part like working with Oracle weblogic , write shell scripting and so . so I've been thinking about instead of entirely leave the development ( this my current case ) instead of leaving my entire experience with the development ( 4 years for that matter ) I can get the best of both world and join the DevOps and maybe use my experience at development and newly added experience with the operations . I need to know how exactly I could do that in the best/software-enginner-mindset-friendly way possible ,

Re-AskFollow33

3 ANSWERS

Jonathan Fenocchi

Jonathan Fenocchi, DevOps Engineer at Bazaarvoice

108 upvotes by Vishal Biyani, William Emmanuel Yu, Quora User, Chris Krough, (more)

I'm going to answer this question as wholly and honestly as I possibly can. Nevertheless, I have filled this answer with an occasional shameless plug. You've been warned.

For starters, I work at www.bazaarvoice.com | Bazaarvoice. We're big fans of cloud. Real big. We're also fans of DevOps. There's a lot of discussion these days about "What is DevOps?" Usually, this term is used to describe Systems Programmers (sometimes called Infrastructure Engineers, Systems Engineers, Operations Engineers or, in the most unfortunate case, Systems Administrators). This is not what DevOps means, but in the context of career development, it carries the connotation of a "modern" Systems Programmer.

So, you're a developer and you want to get in on the ops action. Boy, are you in for a surprise. This isn't about installing Arch Linux and learning to write Perl. There's a place for that kind of thing (a very small, dark place in a very distant corner of the universe), but let's first consider a few things about what DevOps is and what DevOps is not.

What a DevOps role is:

Writing code / software.

Building tools.

Doing the painful things, as often as you can.

Being on the on-call rotation for 2 a.m. production outages.

Infrastructure design.

*NIX. Because Windows freaking sucks. (That said, I use OS X and prefer it to any *NIX distribution, but under the hood, it's *NIX, too.)

Scaling stuff. Scale matters.

Maintenance. Like rebooting that frail vhost with a memory leak that no one's bothered to fix or take ownership of.

Monitoring. Lots of it.

Virtualization.

Agile development methodology.

Software release cycles and management.

Automation. Automation. Automation.

Designing a branch/release strategy for the provided SCM (git, Mercurial, svn, etc).

Metrics / reporting. Goes hand-in-hand with monitoring.

Optimization / tuning.

Load / performance testing and benchmarking, including performance testing of highly complex systems.

Cloud. Really, you don't have to have cloud experience, but it can fundamentally change the way you think about complex systems.

Configuration management. (You've surely heard of Puppet, Chef, Ansible, etc. Yes?)

Security.

Load balancing / proxying. (Of services, systems, components and processes.)

Authentication services.

Command-line fu (like awk.

Package management. Freaking package management.

CI/CIT/CD -- continuous integration, continuous integration testing, and continuous deployment. This is the closest thing to the real meaning of "DevOps" that a Systems Programmer will do.

Databases. All of them. SQL, NoSQL, whatever.

Solid systems expertise. We're talking about the networking stack, how hard disks work, how filesystems work, how system memory works, how CPU's work, and how all these things come together. This is the traditional "operations" expertise you've heard about.

What a DevOps role is not:

Easier than being a software engineer.

Never writing code. I write tons of code.

Installing Linux and never touching your favorite OS again.

Working the third shift. (At least, it shouldn't be; if it is, quit your job and come work with me: Keep Austin Bazaar.)

More "fun" than being a software engineer.

Greenfield. You'll deal with old stuff in addition to new stuff.

A career shift

Here are a few things you should do to begin positioning yourself as a DevOps engineer.

Interview at a company that's hiring DevOps. If you get hired, you'll learn the operations side of things fast. Real fast. Or get fired. If you don't get hired, you'll learn what is still missing from your resume / experience that's preventing you from becoming a full-time DevOps engineer. Incidentally, we're hiring: Keep Austin Bazaar

Tell your boss you want to become a DevOps engineer at your company. Your boss should help you to this end. If he/she does not, quit. Then come work at Bazaarvoice with me and a bunch of other really awesome, super talented engineers working on some really awesome and challenging problems.

Obtain practical experience by using your skills as a software engineer to build tools rather than software. Look at any of the open source projects Netflix has written for examples / ideas.

Learn OpenStack. You can do this on your own time and budget. It's not important whether OpenStack sucks compared to Rackspace Cloud. What's important is that you understand all of the various components and why they are important. Have a wad of cash lying around? Learn Amazon Web Services instead.

Participate in anything your team does involving operations -- deployment, scale, etc. (See list above: "What DevOps is.") If your team doesn't do any of that (i.e., they send artifacts over to Operations and the Operations team does deployment), go over to the Operations team and sit in on a few deployments.

puppet test

Write a puppet master and slave application that does the following.

1: Ensures that all nodes in the system have the following config

a)Nginx is configured, running and has the following config in its home.conf file.

server_name home.somedomin.in 192.168.10.1;

# uWSGI serving Django.

upstream django-somedomin-home {

# Distribute requests to servers based on client IP. This keeps load

# balancing fair but consistent per-client. In this instance we're

# only using one uWGSI worker anyway.

ip_hash;

#server unix:///tmp/home-somedomin-uwsgi.sock;

server localhost:8005;

}

b) DNSMasq is running, it exists and has the following config 

address=/home.somedomin.in/192.168.10.1

c) Interfaces file exists, network manager is running and the following config is applied to all Client nodes

# /etc/network/interfaces

# This file describes the network interfaces available on your system

# and how to activate them. For more information, see interfaces(5).

# The loopback network interface

auto lo

iface lo inet loopback

# The primary network interface

auto eth0

iface eth0 inet dhcp

# Example to keep MAC address between reboots

#hwaddress ether DE:AD:BE:EF:CA:FE

# WiFi Example

auto wlan0

iface wlan0 inet static

address 192.168.10.1

netmask 255.255.255.0

network 192.168.1.0

gateway 192.168.1.1

Once you are done with the application. Please push your code to github and share the github account with me. 

You will be rated on 

1: Code quality

2: Commenting and clarity of thought

3: using libraries to achieve your task.

Questions to answer - 

1: If you had to solve one problem in the world, what would that be.

2: If we had to deploy a 1 million nodes network server infrastructure, what do you for see as possible challenges, issues. How would you overcome each? What measures would you put in place to detect such issues before the customer? How would you ensure that somedomin product is running 100% of the time, both online on AWS and offline through the network servers.  

Draw out the possible architecture for such a solution and list down the technologies you will chose. 

Script to add new contacts automatically( inLinkedin )

This script will do the following things : 

1) will get all user url from each page in the  linkedinGroup

2) using that user url with will get the form 

3) once the form is got , it will fill the form and send sumbit 

4) have add multiple user agents  in the request headers  

==========================================================

#!/usr/bin/perl

use strict;

use warnings;

use WWW::Mechanize;

use LWP;

use JSON;

use Data::Dumper;

use Getopt::Long;

use POSIX qw/ strftime /;

use LWP::Simple;

use HTML::TreeBuilder;

use HTML::TreeBuilder::XPath;

use List::Util qw/shuffle/;

my $username = 'login id ';

my $password = ' password ';

my $maiurl= 'https://www.linkedin.com';

my $loginurl=  $maiurl.'/uas/login?goback=&trk=hb_signin';

#my $grp="https://www.linkedin.com/grp/members?gid=124838&page=";# canada group

my $grp="https://www.linkedin.com/grp/members?gid=4504649&page=";# devops group

$|++;

my $mech = login_linkedin();

my $tree = HTML::TreeBuilder::XPath->new_from_content();

my @all_content = get_data($mech,$grp,"1","10");

my @all_new_linkein_users = get_linkedin_user_url($tree,@all_content) ;

send_invitation($mech,@all_new_linkein_users);

sub set_dynamic_user_agent {

my ($mech) = @_;

my @user_agents = (

   'Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0',

   'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10; rv:33.0) Gecko/20100101 Firefox/33.0',

   'Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0',

    'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20130401 Firefox/31.0',

    'Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0',

    'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0',

    'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0',

    'Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0',

    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:24.0) Gecko/20100101 Firefox/24.0'

);

my @shuff_user_agents = shuffle( @user_agents );

    # dynamically set custom HTTP User-agents

    $mech->add_header( 'User-agent' => $shuff_user_agents[0]);

    print  sprintf("set User-Agent : %s \n", $shuff_user_agents[0] ) ;

return $mech;

}

sub send_invitation {

my ( $mech,@all_user_url) = @_;

  for my $user_url (@all_user_url){

    sleep(2);

  $mech= set_dynamic_user_agent($mech);

 $mech -> get("${maiurl}$user_url");

# print sprintf( "User-Agent : %s\n\n",  $mech->uri() );

#<input type="radio" name="reason" value="IF" id="IF-reason-iweReconnect" class="radio-btn">

#<input type="submit" name="iweReconnectSubmit" value="Send Invitation" id="send-invite-button" class="btn-primary"/

 #$mech ->click_button( name => "iweReconnectSubmit" );

 if ( $mech->form_name("iweReconnect") ) {

    print "sending invitation : $user_url \n";

 $mech ->set_fields( 'reason' => 'IF' );

 $mech->submit();

 $mech->success or die "form  POST fail";

  } elsif ( $mech->form_name("invitation")) {

  print " invitation was already sent to $user_url sent \n" ;

  next ;

  }else {

  print "No From matched $user_url\n ";

    next;

  }

 }

}

sub get_linkedin_user_url {

    my ($tree,@all_content) = @_;

    my $all_new_linkein_users;

    for my $content  (@all_content) {

        $tree->parse($content);

        my @newconn = $tree->findnodes('//li[@class="member"');

        print "getting url  that which is not in my connection \n";

        for my $custm (@newconn){

            my @all_a_tag = $custm->look_down('_tag' => 'a'  ) ;

             my $count =  @all_a_tag;

            my $invte_link  ;

             if (   $all_a_tag[$count - 1]->attr('href') =~ m/people/ig   ) {

                $invte_link = $all_a_tag[$count - 1]->attr('href')  ;

             } else {

             next;

             }

             print $invte_link ." ---- \n";

        push ( @all_new_linkein_users, "$invte_link")  ;

  exit unless(@all_new_linkein_users);

        }

    }

return (@all_new_linkein_users);

}

sub get_data {

    my ( $mech,$url,$spage,$epage ) = @_;

    my @all_content;

    print "Getting the content form page $spage to page $epage\n";

    for my $page ($spage .. $epage ) {

        print " ----------- ${url}$page -----------------"."\n";

        sleep(1) ;

        $mech -> get ("${url}$page");

        push ( @all_content,    $mech->content()) ;

    }

    return (@all_content);

}

sub login_linkedin {

    my $mech = WWW::Mechanize->new();

    $mech -> cookie_jar(HTTP::Cookies->new());

    $mech -> get("$loginurl");

    $mech -> form_id('login');

    $mech -> field ('session_key' => $username);

    $mech -> field ('session_password' => $password);

    $mech -> click_button (value => 'Sign In');

    return $mech ;

}

============================================================

python script to fetch gmail ids and script to send out mail with attachment

-----------------------------------------------------------------------------------------------------------------

script to fetch gmail ids

------------------------------------------------------------------------------------------------------------------

import imaplib, sys, email

mail = imaplib.IMAP4_SSL('imap.gmail.com')

mail.login('gmail@gmail.com', 'password')

mail.list()

# Out: list of "folders" aka labels in gmail.

mail.select("inbox") # connect to inbox.

result, data = mail.search(None, '(HEADER From "info@monsterindia.com")')

for num in reversed(data[0].split()):

    rv, data = mail.fetch(num, '(RFC822)')

    raw_email = data[0][1] # here's the body, which is raw text of the whole email

# including headers and alternate payloads

    email_message = email.message_from_string(raw_email)

    if email_message['Reply-To'] in ('naukrialerts@naukri.com', None):

        continue

    print email_message['Reply-To']

-----------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------------

script to send out mail with attachment 

----------------------------------------------------------------------------------------------------------------

#!/usr/bin/python

import smtplib

from email.MIMEMultipart import MIMEMultipart

from email.MIMEBase import MIMEBase

from email.MIMEText import MIMEText

from email import Encoders

import os,re

gmail_user = "some@gmail.com"

gmail_pwd = "passowrd"

def mail(to, subject, attach, recruitername ):

   msg = MIMEMultipart()

   msg['From'] = gmail_user

   msg['To'] = to

   msg['Subject'] = subject

   text = """

Hi  %s

balah 

balah 

balah 

balah 

Regards 

balah 

balah 

""" %  recruitername

   msg.attach(MIMEText(text))

   part = MIMEBase('application', 'octet-stream')

   part.set_payload(open(attach, 'rb').read())

   Encoders.encode_base64(part)

   part.add_header('Content-Disposition',

           'attachment; filename="%s"' % os.path.basename(attach))

   msg.attach(part)

   mailServer = smtplib.SMTP("smtp.gmail.com", 587)

   mailServer.ehlo()

   mailServer.starttls()

   mailServer.ehlo()

   mailServer.login(gmail_user, gmail_pwd)

   mailServer.sendmail(gmail_user, to, msg.as_string())

   # Should be mailServer.quit(), but that crashes...

   mailServer.close()

myemailfile = "/scripts/cyber_send"

emaillist =  open( myemailfile, 'r')

for i in emaillist.readlines():

    match = re.search(r'[\w\.-]+@[\w\.-]+', i)

    if not  match:

        continue

    print "%s " % match.group(0)

    hrname =  str(re.split("\.|_|-", match.group(0).split('@')[0] )[0])

    #hrname = ''

    mail( match.group(0) , "Sr Engineer ( information technology )",

   "/scripts/resume_.pdf" , hrname )

-------------------------------------------------------------------------------------------------------------------

whats app plugin for Nagios

If you want to send alerts to your phone on watsapp this is one way . 

apt-get install python python-dateutil python-argparse python-dev python-setuptools

git clone https://github.com/tgalal/yowsup

python install setup.py

cd yowsup

./yowsup-cli registration --requestcode sms --phone 91yournumber  --cc 91  --mcc 404 --mnc 20

./yowsup-cli registration --register 671587 --phone 91yournumber --cc 91

cat yowsup-cli.config

cc=91

phone=yournumber

id=yourimeanumber

password=yourpassword

./yowsup-cli demos -c yowsup-cli.config  -s tonumber " saleeeeee"

Nagios Plugin config : 

cat /etc/nagios3/commands.cfg

 

define command{

        command_name    notify-service-by-watsapp

        command_line    /usr/lib/nagios/plugins/whatsapp.sh --numbers $_CONTACTDEVOPSPHONE$  --msg "***** Nagios : $NOTIFICATIONTYPE$ ***** \n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$ "

}

cat  /etc/nagios3/contacts.cfg

define contact{

        contact_name                    admin

        alias                           root

        service_notification_period     24x7

        host_notification_period        24x7

        service_notification_options    w,u,c,r

        host_notification_options       d,r

        service_notification_commands   notify-service-by-email,notify-service-by-watsapp

        host_notification_commands      notify-host-by-email

        _devopsphone                    9819031945

        email                           youremailid

        }

cat /usr/lib/nagios/plugins/whatsapp.sh

#!/bin/bash

while [  "$1" != "" ]

do

case "$1" in

--numbers) numbers=$2; shift;;

--msg) msg=$2; shift;;

*) echo "$0: invalid option $1" >&2

echo "Usage: $0   --numbers 9819031911 --msg \" test mgs\"" >&2

exit 1

esac

shift

done

ph_nums=($(echo $numbers | sed 's/,/ /g' ))

prntmsg=$(printf "$msg")

for i in ${ph_nums[@]}

do

sudo /opt/yowsup/yowsup/yowsup-cli demos -c /opt/yowsup/yowsup/yowsup-cli.config  -s 91${i} "$prntmsg"

echo "$i $msg"

done

send email using MIME::Lite::TT::HTML

#!/usr/bin/perl

 use strict;

 use warnings;

 use MIME::Lite::TT::HTML;

email ('agnello','agnello.dsouza@gmail.com');

sub email {

 my ( $recruiter_name, $to ) = @_;

 my %options;

 $options{INCLUDE_PATH} = '/scripts';

 my %params;

 $params{recruiter_name} = $recruiter_name ;

 my $msg = MIME::Lite::TT::HTML->new(

          From        =>  'agnello.dsouza@gmail.com'  ,

            To          =>  $to,

            Subject     =>  'Some subject  ',

            Template    =>  {

                                html     =>  'resume.tt',

                                text     =>  'text.resume.tt',

                            },

            TmplOptions =>  \%options,

            TmplParams  =>  \%params,

 );

# Set our content type properly

 $msg->attr("content-type"  => "multipart/mixed");

 # Attach a PDF to the message

 $msg->attach(  Type        =>  'application/pdf',

                Path        =>  '/scripts',

                Filename    =>  'Agnello_dsouza.pdf',

                Disposition =>  'attachment'

 );

# $msg->send('smtp', "smtp.gmail.com",   SSL=>1,  AuthUser=>'agnello.dsouza.linux@gmail.com',  AuthPass=>'xxxxxxxxxx',  Debug=>1);

 $msg->send();

}

HACK : script to get all linkedin Contact email IDs -- part 1

#!/usr/bin/perl

use strict;

use warnings;

use WWW::Mechanize;

use LWP;

use JSON;

use Data::Dumper;

use Getopt::Long;

use POSIX qw/ strftime /;

my $username = 'agnello.dsouza';

my $password = 'xxxxxxxxxxxxxx';

my $maiurl= 'https://www.linkedin.com';

my $loginurl=  $maiurl.'/uas/login?goback=&trk=hb_signin';

my $all_data = 'https://www.linkedin.com/contacts/api/contacts/?fields=id%2Cname&sort=-last_interaction';

my $json_file = '/tmp/json.data';

my $mech = login_linkedin();

#my $jdata = get_jdata($mech,$all_data);

#write_to_file($jdata,$json_file);

my $jdata = read_from_file($json_file);

my @contacts = gen_url ($jdata);

foreach (@contacts){

my $fnd_details_json = get_jdata($mech,$_);

my $email = $fnd_details_json->{'contact_data'}->{'emails_extended'}->[0]->{'email'};

next unless ( $email);

print "$email\n"

}

sub gen_url {

my $jdata= shift;

my @url ;

for my $i (@{$jdata->{'contacts'}} ){

my $uri = $maiurl.'/contacts/api/contacts/'.$i->{'id'}.'/?fields=name,emails_extended,birthday,phone_numbers,sites,addresses,company,title,location,ims,profiles,twitter,wechat,display_sources';

push (@url,  $uri );

}

return @url;

}

sub get_jdata {

my ( $mech,$url ) =@_;

$mech -> get ("$url");

return    decode_json($mech ->content()) ;

}

sub login_linkedin {

my $mech = WWW::Mechanize->new();

$mech -> cookie_jar(HTTP::Cookies->new());

$mech -> get("$loginurl");

$mech -> form_id('login');

$mech -> field ('session_key' => $username);

$mech -> field ('session_password' => $password);

$mech -> click_button (value => 'Sign In');

return $mech ;

}

sub read_from_file {

my $file = shift;

my $json;

{

  local $/; #Enable 'slurp' mode

  open my $fh, "<", "$file";

  $json = <$fh>;

  close $fh;

 }

return decode_json($json);

}

sub write_to_file {

my $data  = shift;

my $file  = shift;

open my $fh, ">", "$file";

print $fh encode_json($data);

close $fh;

}