things to learn in linux
Advance administrative tasks on OS linux
Linux Server, Installation and configuration of Apache, FTP, DNS, DHCP, SAMBA Server, networking, RAID configuration on LINUX platforms
Troubleshooting issues of linux Operating System.
Snapshot based backup technology.
Build and support complex linux systems of physical and virtual(XEN based).
experience on another Open Source Hypervisor such as VBox, KVM with experience on P2V, V2P, BMR
Backup solutions Veritas / Arc serve / NTBackup.
large server cluster
systems automation
Amazon EC2, SQS, S3
tools like Puppet, MemCached, Varnish, TokyoCabinet, HAProxy, Nagios, Cacti, Ganglia, and MySQL
C and Kernel programming, multithreading and synchronization
Linux kernel, driver development and troubleshooting
OS concepts, multiprocessing, File Systems, Virtual Memory
processor architecture, SMP systems, memory hierarchy
high availability database driven environments.
Apache HA, High Availability, SAN, DRBD, Load balancing, TCP/IP, database driven environment
Load Balancing, DRBD, Apache HA, IP tables, Web application
VMWare ESX / vSphere
RedHat Cluster
Sun Cluster
EMC and Hitachi SAN storage and zoning
Fault Tolerant networking, High Availability
VMware : Clustering, High Availablity, Dynamic Resource Scheduling
Storage distribution
Switches, physical networking
VMWare Certified Professional
FibreChannel storage
Loadbalancers, Packet Filtering, IPFilter, Security, Penetration testing, Break/Fix
install, tune, troubleshoot, apply patches and support Web servers and integrate third party applications Responsible for uptime of the Web severs and improving performance
knowledge in conducting VAPT and providing solutions
knowledge of source control management.
Knowledge in ethical hacking and information security management systems
implementation and maintenance of web analytics
Apache, Tomcat and IIS web servers administration
nmap, snmpwalk, nessus, dnswalk, libwhisker, strace
HP Openview, Nagios, ZenOSS
Automated Web Application Stress Testing
Tomcat, and Bea Weblogic java application servers
openldap, slurpd to syncrepl migration, ldapsearch
Advanced usage of sniffers for network troubleshooing and optimization
tcpdump, ethereal, and ngrep all covered indepth
Linux and Unix proactive system hardening
tcpdump, ntop, ethereal / wireshark
Apache Web Server, Bind DNS,
Vulnerability scanning and network penetration testing of customer facing applications
Customized and branded Perl open source web metrics reporting software for intranet usage statistics
Linux Server, Installation and configuration of Apache, FTP, DNS, DHCP, SAMBA Server, networking, RAID configuration on LINUX platforms
Troubleshooting issues of linux Operating System.
Snapshot based backup technology.
Build and support complex linux systems of physical and virtual(XEN based).
experience on another Open Source Hypervisor such as VBox, KVM with experience on P2V, V2P, BMR
Backup solutions Veritas / Arc serve / NTBackup.
large server cluster
systems automation
Amazon EC2, SQS, S3
tools like Puppet, MemCached, Varnish, TokyoCabinet, HAProxy, Nagios, Cacti, Ganglia, and MySQL
C and Kernel programming, multithreading and synchronization
Linux kernel, driver development and troubleshooting
OS concepts, multiprocessing, File Systems, Virtual Memory
processor architecture, SMP systems, memory hierarchy
high availability database driven environments.
Apache HA, High Availability, SAN, DRBD, Load balancing, TCP/IP, database driven environment
Load Balancing, DRBD, Apache HA, IP tables, Web application
VMWare ESX / vSphere
RedHat Cluster
Sun Cluster
EMC and Hitachi SAN storage and zoning
Fault Tolerant networking, High Availability
VMware : Clustering, High Availablity, Dynamic Resource Scheduling
Storage distribution
Switches, physical networking
VMWare Certified Professional
FibreChannel storage
Loadbalancers, Packet Filtering, IPFilter, Security, Penetration testing, Break/Fix
install, tune, troubleshoot, apply patches and support Web servers and integrate third party applications Responsible for uptime of the Web severs and improving performance
knowledge in conducting VAPT and providing solutions
knowledge of source control management.
Knowledge in ethical hacking and information security management systems
implementation and maintenance of web analytics
Apache, Tomcat and IIS web servers administration
nmap, snmpwalk, nessus, dnswalk, libwhisker, strace
HP Openview, Nagios, ZenOSS
Automated Web Application Stress Testing
Tomcat, and Bea Weblogic java application servers
openldap, slurpd to syncrepl migration, ldapsearch
Advanced usage of sniffers for network troubleshooing and optimization
tcpdump, ethereal, and ngrep all covered indepth
Linux and Unix proactive system hardening
tcpdump, ntop, ethereal / wireshark
Apache Web Server, Bind DNS,
Vulnerability scanning and network penetration testing of customer facing applications
Customized and branded Perl open source web metrics reporting software for intranet usage statistics
Useful Linux scripts - lsof, ps, fuser, netstat
Important PS command:
I use the following ps commands in order to check for performance probelms:
1) Displaying top CPU_consuming processes:
ps aux | head -1; ps aux | sort -rn +2 | head -10
2) Displaying top 10 memory-consuming processes:
ps aux | head -1; ps aux | sort -rn +3 | head
3) Displaying process in order of being penalized:
ps -eakl | head -1; ps -eakl | sort -rn +5
4) Displaying process in order of priority:
ps -eakl | sort -n +6 | head
5) Displaying process in order of nice value
ps -eakl | sort -n +7
6) Displaying the process in order of time
ps vx | head -1;ps vx | grep -v PID | sort -rn +3 | head -10
7) Displaying the process in order of real memory use
ps vx | head -1; ps vx | grep -v PID | sort -rn +6 | head -10
8) Displaying the process in order of I/O
ps vx | head -1; ps vx | grep -v PID | sort -rn +4 | head -10
9) Displaying WLM classes
ps -a -o pid, user, class, pcpu, pmem, args
10) Determinimg process ID of wait processes:
ps vg | head -1; ps vg | grep -w wait
11) Wait process bound to CPU
ps -mo THREAD -p <PID>
lsof:
To list all open files, use:
# lsof
To list all open Internet, x.25 (HP-UX), and UNIX domain files, use:
# lsof -i -U
To list all open IPv4 network files in use by the process whose PID is 1234, use:
# lsof -i 4 -a -p 1234
To list all files using any protocol on ports 513, 514, or 515 of host wonderland.cc.purdue.edu, use:
# lsof -i @wonderland.cc.purdue.edu:513-515
To list all files using any protocol on any port of mace.cc.purdue.edu (cc.purdue.edu is the default domain), use:
# lsof -i @mace
To list all open files for login name ââabeââ, or user ID 1234, or process 456, or process 123, or process 789, use:
# lsof -p 456,123,789 -u 1234,abe
To list all open files on device /dev/hd4, use:
# lsof /dev/hd4
To find the process that has /u/abe/foo open, use:
# lsof /u/abe/foo
To send a SIGHUP to the processes that have /u/abe/bar open, use:
# kill -HUP âlsof -t /u/abe/barâ
To find any open file, including an open UNIX domain socket file, with the name /dev/log, use:
# lsof /dev/log
To find processes with open files on the NFS file system named /nfs/mount/point whose server is inaccessible, and presuming your mount table supplies the device number for /nfs/mount/point, use:
# lsof -b /nfs/mount/point
To do the preceding search with warning messages suppressed, use:
# lsof -bw /nfs/mount/point
To ignore the device cache file, use:
# lsof -Di
To obtain PID and command name field output for each process, file descriptor, file device number, and file inode number for each file of each process, use:
# lsof -FpcfDi
To list the files at descriptors 1 and 3 of every process running the lsof command for login ID ââabeââ every 10 seconds, use:
# lsof -c lsof -a -d 1 -d 3 -u abe -r10
To list the current working directory of processes running a command that is exactly four characters long and has an âoâ or âOâ in character three, use this regular expression form of the -c c option:
# lsof -c /^..o.$/i -a -d cwd
To find an IP version 4 socket file by its associated numeric dot-form address, use:
# lsof -i@128.210.15.17
fuser:
# fuser -km /home
kills all processes accessing the file system /home in any way.
# if fuser -s /dev/ttyS1; then :; else something; fi
invokes something if no other process is using /dev/ttyS1.
# fuser telnet/tcp shows all processes at the (local) TELNET port.
Some Important Command to find DDOS Attack
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
netstat -ntu | grep -v TIME_WAIT | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
netstat -an | grep :80 | awk '{print $5}' | cut -f1 -d":" | sort | uniq -c | sort -n
netstat Command Example
# netstat –listen
Display open ports and established TCP connections:
# netstat -vatn
For UDP port try following command:
# netstat -vaun
If you want to see FQDN then remove -n flag:
# netstat -vat
lsof Command Examples
Display list of open ports
# lsof -i
To display all open files, use:
# lsof
To display all open IPv4 network files in use by the process whose PID is 9255, use:
# lsof -i 4 -a -p 9255
list information about TCP sessions on your server (specifically SSH in this example)
# lsof -i tcp@`hostname`:22
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd2 7585 root 5u IPv4 16105 TCP localbox:ssh->your.src.ip.here:5897 (ESTABLISHED)
sshd2 7653 root 5u IPv4 16188 TCP localbox:ssh->your.src.ip.here:2262 (ESTABLISHED)
list information about all TCP session
# lsof -i tcp@`hostname`
list information about all sockets using port 53 (will display named information on UDP/TCP)
# lsof -i @`hostname`:53
list information about all UDP sessions
# lsof -i udp@`hostname`
will list all open files with "ssh" in them
# lsof -c ssh
list everything but with UIDs insted of the UID name from /etc/passwd
# lsof -l
list all open files with "ssh" and only the UIDs
# lsof -l -c ssh
list all open files for the /tmp dir (very slow), but good for finding that nasty process that's holding a file open (although: fuser -m /tmp, will do the same thing)
# lsof +D /tmp
Sample Outputs:
Get Information About Active / Inactive Memory Pages
I use the following ps commands in order to check for performance probelms:
1) Displaying top CPU_consuming processes:
ps aux | head -1; ps aux | sort -rn +2 | head -10
2) Displaying top 10 memory-consuming processes:
ps aux | head -1; ps aux | sort -rn +3 | head
3) Displaying process in order of being penalized:
ps -eakl | head -1; ps -eakl | sort -rn +5
4) Displaying process in order of priority:
ps -eakl | sort -n +6 | head
5) Displaying process in order of nice value
ps -eakl | sort -n +7
6) Displaying the process in order of time
ps vx | head -1;ps vx | grep -v PID | sort -rn +3 | head -10
7) Displaying the process in order of real memory use
ps vx | head -1; ps vx | grep -v PID | sort -rn +6 | head -10
8) Displaying the process in order of I/O
ps vx | head -1; ps vx | grep -v PID | sort -rn +4 | head -10
9) Displaying WLM classes
ps -a -o pid, user, class, pcpu, pmem, args
10) Determinimg process ID of wait processes:
ps vg | head -1; ps vg | grep -w wait
11) Wait process bound to CPU
ps -mo THREAD -p <PID>
lsof:
To list all open files, use:
# lsof
To list all open Internet, x.25 (HP-UX), and UNIX domain files, use:
# lsof -i -U
To list all open IPv4 network files in use by the process whose PID is 1234, use:
# lsof -i 4 -a -p 1234
To list all files using any protocol on ports 513, 514, or 515 of host wonderland.cc.purdue.edu, use:
# lsof -i @wonderland.cc.purdue.edu:513-515
To list all files using any protocol on any port of mace.cc.purdue.edu (cc.purdue.edu is the default domain), use:
# lsof -i @mace
To list all open files for login name ââabeââ, or user ID 1234, or process 456, or process 123, or process 789, use:
# lsof -p 456,123,789 -u 1234,abe
To list all open files on device /dev/hd4, use:
# lsof /dev/hd4
To find the process that has /u/abe/foo open, use:
# lsof /u/abe/foo
To send a SIGHUP to the processes that have /u/abe/bar open, use:
# kill -HUP âlsof -t /u/abe/barâ
To find any open file, including an open UNIX domain socket file, with the name /dev/log, use:
# lsof /dev/log
To find processes with open files on the NFS file system named /nfs/mount/point whose server is inaccessible, and presuming your mount table supplies the device number for /nfs/mount/point, use:
# lsof -b /nfs/mount/point
To do the preceding search with warning messages suppressed, use:
# lsof -bw /nfs/mount/point
To ignore the device cache file, use:
# lsof -Di
To obtain PID and command name field output for each process, file descriptor, file device number, and file inode number for each file of each process, use:
# lsof -FpcfDi
To list the files at descriptors 1 and 3 of every process running the lsof command for login ID ââabeââ every 10 seconds, use:
# lsof -c lsof -a -d 1 -d 3 -u abe -r10
To list the current working directory of processes running a command that is exactly four characters long and has an âoâ or âOâ in character three, use this regular expression form of the -c c option:
# lsof -c /^..o.$/i -a -d cwd
To find an IP version 4 socket file by its associated numeric dot-form address, use:
# lsof -i@128.210.15.17
fuser:
# fuser -km /home
kills all processes accessing the file system /home in any way.
# if fuser -s /dev/ttyS1; then :; else something; fi
invokes something if no other process is using /dev/ttyS1.
# fuser telnet/tcp shows all processes at the (local) TELNET port.
Some Important Command to find DDOS Attack
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
netstat -ntu | grep -v TIME_WAIT | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
netstat -an | grep :80 | awk '{print $5}' | cut -f1 -d":" | sort | uniq -c | sort -n
netstat Command Example
# netstat –listen
Display open ports and established TCP connections:
# netstat -vatn
For UDP port try following command:
# netstat -vaun
If you want to see FQDN then remove -n flag:
# netstat -vat
lsof Command Examples
Display list of open ports
# lsof -i
To display all open files, use:
# lsof
To display all open IPv4 network files in use by the process whose PID is 9255, use:
# lsof -i 4 -a -p 9255
list information about TCP sessions on your server (specifically SSH in this example)
# lsof -i tcp@`hostname`:22
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd2 7585 root 5u IPv4 16105 TCP localbox:ssh->your.src.ip.here:5897 (ESTABLISHED)
sshd2 7653 root 5u IPv4 16188 TCP localbox:ssh->your.src.ip.here:2262 (ESTABLISHED)
list information about all TCP session
# lsof -i tcp@`hostname`
list information about all sockets using port 53 (will display named information on UDP/TCP)
# lsof -i @`hostname`:53
list information about all UDP sessions
# lsof -i udp@`hostname`
will list all open files with "ssh" in them
# lsof -c ssh
list everything but with UIDs insted of the UID name from /etc/passwd
# lsof -l
list all open files with "ssh" and only the UIDs
# lsof -l -c ssh
list all open files for the /tmp dir (very slow), but good for finding that nasty process that's holding a file open (although: fuser -m /tmp, will do the same thing)
# lsof +D /tmp
The command vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity.
# vmstat 3Sample Outputs:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 2540988 522188 5130400 0 0 2 32 4 2 4 1 96 0 0 1 0 0 2540988 522188 5130400 0 0 0 720 1199 665 1 0 99 0 0 0 0 0 2540956 522188 5130400 0 0 0 0 1151 1569 4 1 95 0 0 0 0 0 2540956 522188 5130500 0 0 0 6 1117 439 1 0 99 0 0 0 0 0 2540940 522188 5130512 0 0 0 536 1189 932 1 0 98 0 0 0 0 0 2538444 522188 5130588 0 0 0 0 1187 1417 4 1 96 0 0 0 0 0 2490060 522188 5130640 0 0 0 18 1253 1123 5 1 94 0 0
Display Memory Utilization Slabinfo
# vmstat -m
Get Information About Active / Inactive Memory Pages
# vmstat -aFind Out The Top 10 Memory Consuming Process
# ps -auxf | sort -nr -k 4 | head -10Find Out top 10 CPU Consuming Process
# ps -auxf | sort -nr -k 3 | head -10iostat - Average CPU Load, Disk Activity
The command iostat report Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems (NFS).
Sample Outputs:
# iostatSample Outputs:
Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in) 06/26/2009
avg-cpu: %user %nice %system %iowait %steal %idle
3.50 0.09 0.51 0.03 0.00 95.86
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 22.04 31.88 512.03 16193351 260102868
sda1 0.00 0.00 0.00 2166 180
sda2 22.04 31.87 512.03 16189010 260102688
sda3 0.00 0.00 0.00 1615 0
pmap - Process Memory Usage
The command pmap report memory map of a process. Use this command to find out causes of memory bottlenecks.
To display process memory information for pid # 47394, enter:
Sample Outputs:
# pmap -d PIDTo display process memory information for pid # 47394, enter:
# pmap -d 47394Sample Outputs:
47394: /usr/bin/php-cgi Address Kbytes Mode Offset Device Mapping 0000000000400000 2584 r-x-- 0000000000000000 008:00002 php-cgi 0000000000886000 140 rw--- 0000000000286000 008:00002 php-cgi 00000000008a9000 52 rw--- 00000000008a9000 000:00000 [ anon ] 0000000000aa8000 76 rw--- 00000000002a8000 008:00002 php-cgi 000000000f678000 1980 rw--- 000000000f678000 000:00000 [ anon ] 000000314a600000 112 r-x-- 0000000000000000 008:00002 ld-2.5.so 000000314a81b000 4 r---- 000000000001b000 008:00002 ld-2.5.so 000000314a81c000 4 rw--- 000000000001c000 008:00002 ld-2.5.so 000000314aa00000 1328 r-x-- 0000000000000000 008:00002 libc-2.5.so 000000314ab4c000 2048 ----- 000000000014c000 008:00002 libc-2.5.so ..... ...... .. 00002af8d48fd000 4 rw--- 0000000000006000 008:00002 xsl.so 00002af8d490c000 40 r-x-- 0000000000000000 008:00002 libnss_files-2.5.so 00002af8d4916000 2044 ----- 000000000000a000 008:00002 libnss_files-2.5.so 00002af8d4b15000 4 r---- 0000000000009000 008:00002 libnss_files-2.5.so 00002af8d4b16000 4 rw--- 000000000000a000 008:00002 libnss_files-2.5.so 00002af8d4b17000 768000 rw-s- 0000000000000000 000:00009 zero (deleted) 00007fffc95fe000 84 rw--- 00007ffffffea000 000:00000 [ stack ] ffffffffff600000 8192 ----- 0000000000000000 000:00000 [ anon ] mapped: 933712K writeable/private: 4304K shared: 768000K
The last line is very important:
- mapped: 933712K total amount of memory mapped to files
- writeable/private: 4304K the amount of private address space
- shared: 768000K the amount of address space this process is sharing with others
iptraf - Real-time Network Statistics
The iptraf command is interactive colorful IP LAN monitor. It is an ncurses-based IP LAN monitor that generates various network statistics including TCP info, UDP counts, ICMP and OSPF information, Ethernet load info, node stats, IP checksum errors, and others. It can provide the following info in easy to read format:
- Network traffic statistics by TCP connection
- IP traffic statistics by network interface
- Network traffic statistics by protocol
- Network traffic statistics by TCP/UDP port and by packet size
- Network traffic statistics by Layer2 address
Fig.02: General interface statistics: IP traffic statistics by network interface
tcpdump - Detailed Network Traffic Analysis
The tcpdump is simple command that dump traffic on a network. However, you need good understanding of TCP/IP protocol to utilize this tool. For.e.g to display traffic info about DNS, enter:
To display all IPv4 HTTP packets to and from port 80, i.e. print only packets that contain data, not, for example, SYN and FIN packets and ACK-only packets, enter:
To display all FTP session to 202.54.1.5, enter:
To display all HTTP session to 192.168.1.5:
Use wireshark to view detailed information about files, enter:
# tcpdump -i eth1 'udp port 53'To display all IPv4 HTTP packets to and from port 80, i.e. print only packets that contain data, not, for example, SYN and FIN packets and ACK-only packets, enter:
# tcpdump 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'To display all FTP session to 202.54.1.5, enter:
# tcpdump -i eth1 'dst 202.54.1.5 and (port 21 or 20'To display all HTTP session to 192.168.1.5:
# tcpdump -ni eth0 'dst 192.168.1.5 and tcp and port http'Use wireshark to view detailed information about files, enter:
# tcpdump -n -i eth1 -s 0 -w output.txt src or dst port 80system admin techniques implemented at slideshare.com
Do you speak native Linux and have operations experience with a large server cluster?
Are you enthusiastic about systems automation (we love Puppet, Chef or CFEngine experience will also give you the right philosophical background)?
SlideShare is looking for dev-ops engineers who will be responsible for scaling SlideShare to hundreds of servers, using automation, virtualization, and cloud computing technologies like Amazon EC2, SQS, S3 (which we use extensively).
We don't believe ops should spend all day firefighting and doing chores to keep the system healthy.
We believe ops should create valuable intellectual property the way software engineers do, using tools like Puppet, MemCached, Varnish, TokyoCabinet, HAProxy, Nagios, Cacti, Ganglia, and MySQL
Are you enthusiastic about systems automation (we love Puppet, Chef or CFEngine experience will also give you the right philosophical background)?
SlideShare is looking for dev-ops engineers who will be responsible for scaling SlideShare to hundreds of servers, using automation, virtualization, and cloud computing technologies like Amazon EC2, SQS, S3 (which we use extensively).
We don't believe ops should spend all day firefighting and doing chores to keep the system healthy.
We believe ops should create valuable intellectual property the way software engineers do, using tools like Puppet, MemCached, Varnish, TokyoCabinet, HAProxy, Nagios, Cacti, Ganglia, and MySQL
Subscribe to:
Posts (Atom)