script to convert mdir to mbox format

#! /bin/sh
#
# Get a directory name as input and convert all mail files inside
# to mbox format
#
# NOTE: processing of subdirectories not yet implemented correctly:
# all mails in subfolders are put into the same mbox
# (it would be better if an mbox file will be generated for
# each subfolder)
# NOTE: calculation of message date in case of 'From:' doesn't recognise
# time zones
#
# History:
# Feb 06 2001 Joerg Reinhardt
# - first edition
# Feb 07 2001 Joerg Reinhardt
# - added usage output
# Feb 12 2001 Joerg Reinhardt
# - mails not containing a 'From:' field but an 'X-From-Line:' or a
# 'Reply-To:' field are now recognised and also processed (e.g. put into
# the mbox file); this works fine for all my mails
# - added progress information
# - warning about corrupt files is now written to stderr

# check for argument or help argument respectively
if [[ ($1 == "") ||
  ($1 == "-h") ||
  ($1 == "--help") ||
  ($1 == "-help") ]]; then
  echo "Usage: "$0" <Xfmail-mail-directory>";
fi;

# check if parameter is a directory
if [[ -d $1 ]]; then
# set target filename
  dirname=`echo $1 | awk '{while(substr($0,length($0),1)=="/"){$0=substr($0,1,length($0)-1);}print $0;}'`;
  mboxfile=$dirname'.mbox';

# check if directory is empty
  if [[ `find $dirname -type f` == "" ]]; then
 echo $dirname": directory empty."
 exit 1;
  fi;

# prevent automatic overwriting of target
  if [[ -e $mboxfile ]]; then \
 dialogtext="Write file "$mboxfile"?";
 if dialog --yesno "$dialogtext" 10 60; then
  clear;
  rm -vf $mboxfile;
 else
  clear; exit 1;
 fi;
  fi;

  echo "writing xfmail mail directory '$1' to '$mboxfile'.";


# collect files inside Xfmail mail-directory and produce MBOX format
# target file
  for i in `find $1/* -type f`; do
# output progress information
 echo -n -e \\r" "
 echo -n -e \\rprocessing $i
# look for senders email address in the order
# 'From:'
# 'X-From-Line:'
# 'Reply-To:'
 shortfromflag='true';
 fromline=`grep 'From:' $i`;
# parse 'From:' field
 from=`echo $fromline | awk 'BEGIN{FS="<";}{if($0~/</) {pos=index($2,">");if(pos!=0) {print substr($2,1,pos-1);}} else {pos=index($0,":");print substr($0,pos+1);}}'`;
 if [[ $from == "" ]]; then
  shortfromflag='false';
  fromline=`grep 'X-From-Line:' $i`;
  from=`echo $fromline | awk 'BEGIN{FS="Line:";}{print $2;}'`;
  if [[ $from == "" ]]; then
  shortfromflag='true';
  fromline=`grep 'Reply-To:' $i`;
# parse 'Reply-To:' field
  from=`echo $fromline | awk 'BEGIN{FS="<";}{if($0~/</) {pos=index($2,">");if(pos!=0) {print substr($2,1,pos-1);}} else {pos=index($0,":");print substr($0,pos+1);}}'`;
  if [[ $from == "" ]]; then
  echo;
  echo "WARNING: "$i": no 'From:' nor 'X-From-Line:' nor 'Reply-To:' field found." >&2;
  continue;
  fi;
  fi;
 fi;
 if [[ $shortfromflag == "true" ]]; then
# parse date field
  dateline=`grep 'Date:' $i`;
  if [[ $dateline == "" ]]; then
# set dummy date if no date field found
  dateline="Date: Thu, 01 Jan 1970 00:00:00 +0000 (GMT)";
  fi;
  weekday=`echo $dateline | awk '{gsub(/,/,"",$2);print $2;}'`;
  day=`echo $dateline | awk '{print $3;}'`;
  month=`echo $dateline | awk '{print $4;}'`;
  year=`echo $dateline | awk '{print $5;}'`;
  time=`echo $dateline | awk '{print $6;}'`;
  diffGMT=`echo $dateline | awk '{print $7;}'`;
  timezone=`echo $dateline | awk '{print $8;}'`;

# output MBOX mail header
  echo "From " $from $weekday $month $day $time $year >> $mboxfile;
 else
# output long MBOX mail header found in 'X-From-Line:' field
  echo $from >> $mboxfile;
 fi;

# output mail itself
 cat $i >> $mboxfile;
  done;
  echo;
else
  echo $1": not a directory.";
fi;

perl site

http://www.troubleshooters.com/codecorn/littperl/perlhelo.htm

good apache SElinux site

http://beginlinux.com/server_training/web-server/976-apache-and-selinux --

..mumbai's saddest moments ... .our cry for CHANGE

[ This article is the only article in my blog that is non technical, but some thing my heart jsut speaks out toady ]

The last two days has been the caziest in Mumbai and for the people in Mumbai ..I would like to share with you my experience for the last two day that has shook the city . The two days ( Wednesday and Thursday ) mumbia .. especially south Mumbai has seen dramatic seens from the Taj and Oberois hotels . We have lost few of our finest police force and this makes me even more angry to the core . Wednesday night as the terrorist took over the taj, we ( my mom and i ) were watching every glimpse of it on televisions ( thanks to NDTV ) till 2 am .... the visuals that we were coming across were crazy.... looked like a mini Kashmir attack ..... The next day ( Thursday ) was declared a holiday for my company and i guess most companies .... the whole day was spent glued a the visuals in the news ..

One thing i kept hearing on TV was that Mumbaikars will come back strong ..... but this statement is what we have always been saying after every such attack against the city . Mumbai is in need of change . Yes, Mumbai ( Navi -Mumbai ) is a fast pace metropolitan city ... but we tend to forget the such incident too quickly, .... it start with a major media hype through the news networks ... but as the days and weeks pass by people ...our leaders ...people in political power tend to forget .... this aggravates me even more with deeper anger within me .

Just today was i was going in my office ... i was stoped by some security guards at my office building to check my baggage ... they just opened one compartment of my baggage and left it there, ignoring to check the rest of my bag .... this is just an example of how things are going to be like in the next few weeks ....and then slowly fade away. Some times I am so ashamed at the structure and system that are in place in our city. According to the new reports .... the city is not even equipped to with proper fire brigades to pump water to reach the 6th floor of a building, with such loose system in place we can jsut imagine where i our city is heading ... security in our transports system ... especially the local trains, that i personally travel to work, where people have to squeeze and get crushed every morning just to reach their work place . We don't have the slightest security system in place , Yes we do have some few cops around the place ..but that's not going to stop any terrorist from planting a boom somewhere , We need structure and system and really concerned politicians to curb such a future attack on our city . and our Government and local political leaders need to hear our cry ... Yes the city is crying for CHANGE and today its cry ( our cry ) is the loudest ... but what a shame it has to be at the cost of 160 plus lives .

I pray today for the the brave hearts that have laid down their lives for others, and i pray for the victims and the relatives of the victims .... i know what it feels in loosing some one close . as i lost my dad this year in the mouth of January. I pray for our city and for the political leaders that they will bring about the change that is so desperately required .

Thank you for taking time out to read this article , i had tears in my eyes as i was watching few stories on the news from relatives of the victims, in this world that is so unpredictable ... hear today gone tomorrow.

Thank you !

Pen - smtp load balancer

This is pen, a load balancer for "simple" tcp based protocols such as http or smtp. It allows several servers to appear as one to the outside and automatically detects servers that are down and distributes clients among the available servers. This gives high availability and scalable performance.

In this senaerio  we have two server ( server A and server B )  

server A = Load balancer with pen installed
eth0 = 192.168.50.156
 
server B = postfix with two instances of psotfix installed (  http://advosys.ca/papers/postfix-instance.html )
eth0 = 192.168.50.124
eth0:1  = 192.168.50.125

postfix  should be listening
[root@smtptestserver ~]# netstat -tupln |grep 25
tcp        0      0 192.168.50.125:25           0.0.0.0:*                   LISTEN      12077/master
tcp        0      0 192.168.50.124:25           0.0.0.0:*                   LISTEN      11910/master on both server


Installation of Pen on Server A
Downlaod the rpm from here

ftp://194.199.20.114/linux/dag/redhat/el5/en/i386/dag/RPMS/pen-0.17.2-1.el5.rf.i386.rpm
rpm -ivh  pen-0.17.2-1.el5.rf.i386.rpm

This should install   /usr/bin/pen

now we need to tell pen to listen port 25 for all  incomning connections  and to distribute those connection to 50 .125 and 50.124

/usr/bin/pen 192.168.50.156:25 -l /var/log/pen/pen.log -p /var/run/pen.pid -r 192.168.50.124:25 192.168.50.125:25

# netstat -tupln |grep pen
tcp        0      0 192.168.50.156:25           0.0.0.0:*                   LISTEN      14505/pen

Now try
telnet 192.168.50.156 25 from any local system   it should take you to either    192.168.50.124 or  192.168.50.124


 

using Expect for ssh auto-login

#!/bin/bash

set -x
> /root/.ssh/known_hosts
echo "Connecting to host"
konsole --new-tab -e \expect -c "set timeout -1;\
spawn ssh 192.168.50.103 -l root ;\
match_max 100000;\
expect *(yes/no)?*;\
send -- yes\r;\
expect *password:*;\
send -- qwerasdf\r;\
interact;"

script to get server info !!

run the following command on your linux machine 

wget -q -O http://linux.googlepages.com/server_info.sh |bash 

this will save a file  name ( hostname ) in /tmp/file_name  

--

how to create your own RPM package ( basic )

In this article we will create an rpm package frm a source tar.gz file. Here we take the example of proftpd.tar.gz

the tree stucture for rpm

|-- BUILD

|-- RPMS

|-- SOURCES

|-- SPECS

|-- SRPMS

let download proftpd in /usr/local/src and configure it in prefix /usr/local/proftp

tar -xzvf proftpd-cvs-20080928.tar.gz

./configure && make && make install

( you can configure with vairous switches .... with mysql support or ldap support )

Now we create the rpm

in the /usr/src/redhat/SPEC we we create a file called proftpd.spec with the following content

Summary: test rpm Name: test_rpm Version: 1 Release: 1 Vendor: agnello License: Test Group: o BuildRoot: /usr/local/proftpd AutoReq: no AutoReqProv: no %description This package contains nothing . %files %defattr(-, root, root) /bin /etc /include /lib /libexec /sbin /share /var

now we run a rpmbuild to build the i386 rpm package

rpmbuild -ba proftpd.spec

this will create a i386.rpm package in the /usr/src/redhat/RPM/ folder

rpm -ivh test_rpm-1-1.i386.rpm ---- this will intall the RPM package

done !!

How to creat you own ISO ( basics )

This documentation is based on Fedora Core 9 . On how to build you own ISO with the mkisofs command

1st mount the the FC 9 on to /media

now cd /media

mkisofs -o /makecd/agnello_linux.iso -p 'agnel' -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -R -J -v -T /makecd

-o ---> is the path where the iso image is goiin to be created

-p ----> pecifies a text string that will be written into the volumeheader. This should describe the preparer of the CDROM, usually with a mailing address and phone number.

-no-emul-boot ---> Specifies that the boot image used to create "El Torito" bootable CDs is a 'no emulation' image. The system will load and execute this image without performing any disk emulation.

-c -----> Specifies the path and filename of the boot catalog to be used when making an bootable CD.

-boot-load-size ----> Specifies the number of "virtual" (512-byte) sectors to load in no-emulation mode. The default is to load the entire boot file. Some BIOSes may have problems if this is not a multiple of 4.

-boot-info-table -----> Specifies that a 56-byte table with information of the CD-ROM layout will be patched in at offset 8 in the boot file.

-R Generate SUSP and RR records using the Rock Ridge protocol to further describe the files on the iso9660 filesystem.

-v Verbose execution.

-T Generate a file TRANS.TBL in each directory on the CDROM

/makecd ----> the location of the cd content

memory allocation in Linux

PROBLEM

I am running Apache HTTPD 1.3.37 with mod_php, when using the top

command, I see:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

12584 web 15 0 142m 12m 9176 S 8 0.3 2:07.01 httpd

12586 web 15 0 142m 12m 9168 S 8 0.3 2:08.85 httpd

12589 web 15 0 141m 10m 7376 S 8 0.3 2:06.83 httpd

12591 web 15 0 142m 11m 7420 S 8 0.3 1:59.64 httpd

12594 web 15 0 142m 11m 7444 S 8 0.3 2:06.25 httpd

12585 web 15 0 141m 12m 9200 S 6 0.3 2:03.83 httpd

Assume all the httpd process is show above...

QUESTION

1. Is the total memory currently used is : 12+12+10+11+11+12 = 68M ?

2. Is the max. memory even allocated by is: 142+142+141+142+142+141 = 850M

ANSWER

ANS1

justinp AT newmediagateway.com

Welcome to the wonderful world of Linux memory. The memory reported in the VIRT/RES column cannot simply be added up to calculate the total memory usage because some memory may be shared. When a process is fork()'d (to create a child process), the memory between the parent and child process will be shared. Once a portion of the memory is changed by the parent or child, that portion of the memory will become distinct between the two processes. For example:

* Parents using 10MB

* A child process is created

* Child now shows usage of 10MB, although total usage is only 10MB, not 20MB, due to shared memory

* Child process overwrites 5MB of the memory. Now that memory is unique to the child, so it will be separate

* Memory usage is now 10MB (parent) + 5MB (child) = 15MB (theoretically)

This is just a very basic example, since the true details behind the memory allocation is more complex. There is a python script available that will try to give you a more accurate calculation of memory usages available here:

http://www.pixelbeat.org/scripts/ps_mem.py

ANS 2

torsten.foertsch AT gmx.net

No and no.

How to interprete these figures depend a bit on your operating system.

But in general UNIX systems try to share memory between processes.

There are several ways how that can be achieved. In one way a program

module (executable binary, shared lib) is divided into sections. Each

section is marked by at compile time if it can be shared between

processes or not. Code sections are usually read-only during program

execution hence they can be shared. If your programs use a shared libc

for example all the code in this lib is shared between all processes

that use the lib. But a program module does not have to be loaded

completely into memory to do some work. Code or data pages that are not

needed are not loaded.

A second way to achieve memory sharing is called copy-on-write. When a

process forks both processes at first share all their memory. Now one

of them starts writing to a specific area of its memory. Only at this

time the operating system allocates a personal copy of that piece of

memory for that process. It also does not copy the whole process memory

but only a small chunk (page). So the more both processes write to

their memory the more memory they consume together. But they probably

never consume together twice the amount of memory one of them has

consumed while it has run alone.

So what you see as VIRT is the amount of mem that process would consume

if it runs completely alone and if all of its segments are loaded into

memory. RES is the part of VIRT that is actually allocated. Both say

nothing about sharing. SHR says how much of RES is shared between at

least 2 processes by the first method (not copy-on-write). But it does

not say among how much or which processes it is shared.

Hence you can't simply add up these numbers.

A better way to judge memory consumption on Linux is /proc/$PID/smaps

and tools that use it. /proc/$PID/clear_refs on recent kernels is also

interesting in that regard. To get a general impression how much memory

my apache needs you can start vmstat with a small parameter, stop the

apache and look how it affects the memory related columns.

--