As a system administrator you often will study the contents of logfiles. Logfiles are your main source of information when something is not (quite) working. Alas, often these logfiles are riddled with information you do not need (yet). Therefore, it is useful to be able to filter the data. For example, the syslogd daemon logs so-called marks - lines containing the string “-- MARK --”. It pumps out one such line every 20 minutes^[24]. This is useful to check whether or not the daemon is running successfully, but adds a huge number of lines to the logfile. You can remove these lines in many ways: Unix allows many roads to a given goal. A simple sed script could do the trick:

 
sed '/-- MARK --$/d' /var/log/messages 

or you could use another essential Unix tool:

 
grep -v -- "-- MARK --" /var/log/messages 

Note the double dash that tells grep where its options end. This is necessary to prevent grep from parsing the searchstring as if it were an option. Yet another way to obtain the same goal:

 
awk '!/-- MARK --$/ { print }' /var/log/messages 

And finally, using perl:

 
perl -ne '/-- MARK --$/ || print;' /var/log/messages

Or, in the true tradition of perl that even within perl there is more than one way to do things:

 
perl -ne 'print unless /-- MARK --$/;' /var/log/messages

Another example: suppose you are interested in those lines from the messages file that were written during lunch breaks (noon to 14.00). You could filter out these lines using sed:

sed -n '/^[A-Z][a-z]\{2\} [ 1-3][0-9] 1[23]:/p' /var/log/messages

The -n option tells sed not to print any lines, unless specifically told so using the 'p' command. The regular expression looks like stiff swearing on first sight. It isn't. Ample analysis reveals that it instructs sed to print all lines that begin (^) with an uppercase letter ([A-Z]), followed by 2 lowercase letters ([a-z]\{2\}), a space, either a space or one of the digits 1-3 ([ 1-3]), yet another digit ([0-9]), another space, the digit 1 and either the digit 2 or the digit 3 ([23]), followed by a colon. Phew.

Or you could use this awk one-liner:

awk '/^[A-Z][a-z][a-z] [ 1-3][0-9] 1[23]:/' /var/log/messages

Note that this time we did not use the interval expression ({2}), but simply duplicated the expression. By default gawk (which we are using throughout this book) does not support interval expressions. However, by specifying the --posix flag it can be done:

awk --posix '/^[A-Z][a-z]{2} [ 1-3][0-9] 1[23]:/' /var/log/messages

Alternately, you could use a simple awk script:

awk '{
        split($3,piece, ":")
        if ( piece[1] ~ /1[23]/ ) { print }

}' /var/log/messages

which renders the same result, using a different method. It simply looks for the third (white-space delimited) field, which is the time-specification, splits that into 3 subfields (piece), using the colon as delimiter (split) and instructs awk to print all lines that have a match (~) with the third field from the timefield.

Finally, we can use perl to obtain the same results:

perl -ne '/^[A-Z][a-z]{2} [ 1-3][0-9] 1[23]:/ && print;' /var/log/messages 

As an exercise, you could try some other approaches yourself. If you want to be reasonably sure that your do-it-yourself approach renders the same result as the methods tested above, make a copy of (a part of) the messages file, run one of the commands we gave as an example on it and pipe the result through sum:

$ cp /var/log/messages ./myfile
$ sed -n '/^[A-Z][a-z]\{2\} [ 1-3][0-9] 1[23]:/p' ./myfile |sum
48830   386 []

Your home-brewn script should render the exact same numbers you get from one of the example scripts when its results are piped through sum.

Often, you'll need to combine information from several files into one report. Again, this can be done in many ways. There is no such thing as the right way - it all depends on your skills, preferences and the available toolsets. You are free to choose whichever method you want. However, to get you started, some examples follow.

The command groups will list the groups you are in. It accepts parameters, which should be usernames, and will display the groups for these users. By the way: on most systems groups itself is a shell script. An example of typical use and output for this command follows:

$ groups nobody root uucp
nobody : nogroup
root : root bin uucp shadow dialout audio nogroup
uucp : uucp
$ _

The information displayed here is based on two publicly readable files: /etc/passwd and /etc/group. The groups command internally uses id to access the information in these files, but nothing stops us from accessing it directly. The following sample shell script consists mainly of an invocation of awk. It will display a list of all users and the groups they are in, and, as an extra, prints the full-name of the user.

   /----------------------------------------------------------------------
01 | #!/bin/sh
02 | #
03 | 
04 | # This script displays the groups system users
05 | # are in, on a per user basis.
06 | 
07 | awk -F: 'BEGIN {
08 | 
09 |         # Read the comment field / user name field from the
10 |         # passwd file into the array 'fullname', indexed by
11 |         # the username. If the comment field was empty, use
12 |         # the username itself as 'comment'.
13 |         #
14 |         while (getline < "/etc/passwd") { 
15 |                 if ($5=="") {
16 |                         fullname[$1] = $1
17 |                 } else {
18 |                         fullname[$1] = $5 
19 |                 }
20 |         }
21 | } 
22 | 
23 | {
24 |         # Build up an array, indexed by user, that contains a
25 |         # space concatenated list of groups for that user
26 | 
27 |         number_of_users=split($NF,users,",")
28 | 
29 |         for(count=0; count < number_of_users; count++) {
30 |                 this_user=users[count+1]
31 |                 logname[this_user] = logname[this_user] " " $1 
32 |         }
33 | }
34 | 
35 | END {
36 |         # List the array
37 |         #
38 |         for (val in logname) {
39 |                 printf "%-10s %-20s %s\n", 
40 |                         substr(val,1,10), 
41 |                         substr(fullname[val],1,20), 
42 |                         logname[val]
43 |         }
44 | }' /etc/group
   \----------------------------------------------------------------------

An example of how the output might look:

at         at                    at
bin        bin                   bin
db2as      DB2 Administration    db2iadm1
named      Nameserver Daemon     named
oracle     Oracle User           dba
db2inst1   DB2 Instance main us  db2asgrp

... output truncated ... 

On line 01 we state that the interpreter for this script is /bin/sh. That may seem surprising at first, since the script actually consists of awk statements. However, there are two exceptions: the command-line parameter (-F:), (line 07) and the filename /etc/group (line 44). It is possible to rewrite this program as a pure awk script - we leave it up to you as an exercise. The awk construction consists of three groups:

the BEGIN block.  Line 14 shows a method to sequentially read a file within an awk program block: the getline function. getline fetches an input-line (record) from a file and, if used in this context, will fill the variables (e.g., $0..$NF) accordingly. By putting getline in the decision part of a while loop, the entire file will be read, one record (line) per iteration, until getline returns 0 (which signals “end of file”) and the while loop is terminated. In our case, the file /etc/passwd is opened and read, line by line. We set the field-delimiter to a colon (-F:, line 07), which is indeed the field-separator for fields in the /etc/passwd file. Within the loop we now can access the username ($1) and full name of (or a comment about) that user ($5). The construct used on line 16 (and 18) creates an array, indexed by username.

the main block.  In the main block, awk will loop through the /etc/group file (which was specified on the command line as input file, line 44). The variable $NF always contains the contents of the last field, which, in this case, is a comma-delimited list of users. Line 27 splits up that field and puts the usernames found in the last field into an array users. The for loop on lines 29-32 adds the group to the proper elements of array logname, which is indexed by username.

the END block.  The END block will be executed when the input-file is exhausted. The logname array was previously indexed by username: each array-element contains a space-delimited list of group-names (e.g., the array-element logname[root] could contain the string “root bin shadow nogroup”). The “for (val in logname)” loop (line 38) will walk through all elements of the array logname, whilst putting the name of the element in the variable val. Line 39 defines the output format to use: first a left-aligned string-field with a length of 10 characters, if necessary padded to the left with spaces (%-10s); next, another left-aligned string-field with a length of 20 characters, also padded to the left with spaces (%-10s), and finally a string of variable length. To prevent field overflow, the substr function is used to trim fields to their maximum allowed length.

Of course, you can do the very same thing using a perl script. You can start from scratch, but since we already have a fully functional awk program that does the trick, we can also use a utility to convert that script to a perl program. On most Unix systems it's named a2p. It takes the name of the awk script as its argument and prints the resulting output to standard output:

 
a2p userlist.awk > userlist.pl

The results are often pretty good, but in our case some fine-tuning was needed, both to the originating and resulting scripts, since the awk program in our example was embedded in a shell-script. The example below lists our result:

   /----------------------------------------------------------------------
01 | #!/usr/bin/perl
02 | 
03 | open(_ETC_PASSWD, '/etc/passwd') || die 'Cannot open file "/etc/passwd".';
04 | open(_ETC_GROUP,  '/etc/group') || die 'Cannot open file "/etc/group".';
05 | 
06 | $[ = 1;                    # set array base to 1
07 | 
08 | $FS = ':';
09 | 
10 | # Read the comment field / user name field from the
11 | # passwd file into the array 'fullname', indexed by
12 | # the username. If the comment field was empty, use
13 | # the username itself as 'comment'.
14 | #
15 | while (($_ = &Getline2('_ETC_PASSWD'),$getline_ok)) {
16 |     if ($Fld[5] eq '') {
17 |        $fullname{$Fld[1]} = $Fld[1];
18 |     }
19 |     else {
20 |        $fullname{$Fld[1]} = $Fld[5];
21 |     }
22 | }
23 | 
24 | while (($_ = &Getline2('_ETC_GROUP'),$getline_ok)) {
25 |     chomp; # strip record separator
26 |     @Fld = split(/[:\n]/, $_, 9999);
27 | 
28 |     # Build up an array, indexed by user, that contains a
29 |     # space concatenated list of groups for that user
30 | 
31 |     $number_of_users = (@users = split(/,/, $Fld[$#Fld], 9999));
32 | 
33 |     for ($count = 0; $count < $number_of_users; $count++) {
34 |        $this_user = $users[$count + 1];
35 |        $logname{$this_user} = $logname{$this_user} . ' ' . $Fld[1];
36 |     }
37 | }
38 | 
39 | # List the array
40 | #
41 | foreach $val (keys %logname) {
42 |     printf "%-10s %-20s %s\n", 
43 |     substr($val, 1, 10), 
44 |     substr($fullname{$val}, 1, 20), 
45 |     $logname{$val};
46 | }
47 | 
48 | sub Getline2 {
49 |     ($fh) = @_;
50 |     if ($getline_ok = (($_ = <$fh>) ne '')) {
51 |        chomp;  # strip record separator
52 |        @Fld = split(/[:\n]/, $_, 9999);
53 |     }
54 |     $_;
55 | }
   \----------------------------------------------------------------------

As you might expect, this code works in a manner similar to the previously used awk example. The most striking difference is the addition of the Getline2 function. This function reads a line from the file specified by the file handle (which was passed as argument) and splits the records into fields, which are put in the array Fld. The chomp function (line 51) removes the record separator from the input-line, since perl does not do that automatically (awk does). Also, the perl program was rewritten to remove its dependency on a shell, therefore, the program opens both input files within (lines 03 and 04). Line 06 sets the array base to 1, which is the default for awk arrays, as opposed to perl arrays, which are base 0. The loop in line 15-22 reads the information from the passwd file and puts the full user names in the array fullname again, the loop on lines 24-37 reads in the group-file and builds the array that contains the lists of groups for each user. Finally, the code in lines 41-46 is almost identical to that of the END block in the awk example (lines 38-44) and works likewise.

The automatic generation of code does not necessarily deliver very clean code. In our example, some code was included that is not necessary to make this program work properly. As an exercise, try to find those lines.

In most cases, it suffices to construct reports and mail them to the system administrator. However, when an event occurs which requires immediate attention another mechanism should be used. In these cases, you could use SMS-messaging or page the sysadmin. You also could send a message to a terminal or have a pop-up box appear somewhere. Again, there are many ways to achieve your goal. SMS and (alphanumeric) pagers are used most frequently.

Sending mail to somebody on a Unix system from within a process is very simple provided that you have set up e-mail correctly, of course. For example, to mail from within a shell program, use this command:

echo "Hi there!" | mail jones -s "Hi there!" 

Or, to send longer texts, you could use a “here document” (described in the section called “Here documents”):

mail jones -s "Hi there!" <<EOF

#         #
#      #  #
#         #
# ###  #  #
##   # #  #
#    # #   
#    # #  #    

EOF 

To mail something from within Perl, a similar concept can be used:

 
open(MAIL , "|mail jones -s \"Hi There!\"") || die "mail failed";
print MAIL "Hi, Paula.";
close(MAIL); 

To print more than one line, you can use may different techniques, for example, you could use an array and print it like this:

 
@text[0]="Hello, Paula\n";
@text[1]="\n";
@text[2]="Greetings!\n";
 
open(MAIL , "|mail jones -s \"Hi There!\"") || die "mail failed";
print MAIL @text;
close(MAIL);   

Note that you need to insert newline characters at the end of the array elements if you want to start a new line. Alternatively, you could use a “here document”), as we did in the shell:

 
open(MAIL, "|mail root -s \"Hi There!\"") || die "mail failed";
print MAIL <<EOF;
 
#         #
#      #  #
#         #
# ###  #  #
##   # #  #
#    # #
#    # #  #
 
EOF
close(MAIL);           

Pager-service providers provide PSTN gateways: you dial in to these systems using a modem, over public phone-lines. Some special software needs to be written or downloaded to enable you to page someone. Often some kind of terminal based protocol is used. Other, systems use a protocol called TAP (Telocator Alpha-entry Protocol), formerly known as PET (Personal Entry Terminal) or IXO (Motorola invented it, including its name). Sometimes (in Europe) the UCP protocol (Universal Communication Protocol) is used. There are a number of software programs for Unix systems to enable them to use PSTN gateways, the most frequently used ones are Hylafax, beepage, qpage, tpage and sendpage. The latter is written in Perl and released under the GPL.

Pager-services are often replaced by SMS services nowadays, especially in Europe, where the GSM network is available almost everywhere. SMS offers the advantage of guaranteed delivery to the recipient. Additionally, most system administrators already have mobile phones. If SMS is used, they do not need to carry around an additional pager.

To enable computers to send SMS messages, special add-on GSM devices are available. They either accept serial input or are placed in a PCMCIA slot. They are capable of sending SMS messages directly. Some software has been written for them: you may want to look at http://smslink.sourceforge.org. By adding a computer that addresses such a device to your network, you can set up your own gateway(s) for your own network.

Apart from installing your own software/gateway, you can use external gateways, some of which are available on the Internet, others are provided by your Telecom operator:

The objectives state that you should be able to write a script that notifies (the) administrator(s) when specified users log in or out. There are, as always, many ways to accomplish this.

One solution would be to give a user a special login “shell”, that issues a warning before starting up the “real” shell. This could be a simple shell-script (don't forget to chmod +x it) and named something like /bin/prebash. An example of such a script:

#!/bin/bash
#
DATE=`/bin/date`
#
mail root -s "USER $USER LOGGED IN" <<EOF

Attention!

The user $USER has logged in at $DATE.
EOF
exec /bin/bash --login

The corresponding line in the /etc/passwd file for this user could look like this:

user:x:501:100::/home/user:/bin/prebash

When the user logs in, the initiating script will be run. It will send the mail and then overwrite the current process with the shell. The user will never see that the warning has been issued, unless he or she checks out his/her entry in the /etc/passwd file. Alternately, you could choose to leave the “initiating” shell in place (i.e., not to issue the exec command) which lets you use the initiating shell to report to you when a user logs out:

#!/bin/bash
#
DATE=`/bin/date`

/usr/bin/mail root -s "USER $USER LOGGED IN" <<EOF
 
Attention!
The user $USER has logged in at $DATE.
EOF

/bin/bash --login

DATE=`/bin/date`
nohup /usr/bin/mail root -s "USER $USER LOGGED OUT" 2>/dev/null 1>&2 <<EOF2
 
Attention!
 
The user $USER has logged out at $DATE.
EOF2

This script consists of three parts: a part that is executed before the actual shell is executed, the execution of the shell and a part that will be executed after the shell has terminated. Note that the last mail command is protected from hangup signals by a nohup statement. If you leave out that statement, the mail process will receive a terminating signal before it would be able to send the mail.

Now, root will receive mail every time the user logs in or out. Of course, you can issue any command you want instead of sending mail. Note that the user could easily detect this guardian script is running by issuing a ps command. He could even kill the guardian script, which would also terminate his script, but the logout notification would never arrive.

Another method would be to invoke logger, which logs a warning to the syslog facility (e.g., for a script that just warns you when somebody has logged in):

#!/bin/bash
#
/usr/bin/logger "logged in"
exec /bin/bash --login

logger, by default, will log a line to the file that has been connected to the “user.notice” syslog facility (see: syslog.conf(5) and logger(1)). Note, that logger automatically prepends the line in the logfile with the date and the username, hence you do not need to specify it yourself.

We have described two ways to record a user logging in or out, using the Bourne shell to glue standard Unix/Linux commands together. There are many more possibilities: to use a script that acts as a daemon and that regularly checks the process-table of any processes run on behalf of the user or to add a line to /etc/profile that alerts the sysadmin. All of these methods have various pros and cons. As usual, there are many ways to achieve what you want.

To demonstrate the use of awk to parse information, we will tackle this problem another way. On Linux systems, the last(1) command shows a listing of the logging behavior of users. The command obtains its information from the wtmp file (often found in /var/log). The wtmp file is used by a number of programs to records logins and logouts; login, init and some versions of getty. An example of its output follows:

piet     tty1                          Tue Nov 27 18:04   still logged in   
piet     tty1                          Tue Nov 27 18:03 - 18:03  (00:00)    
henk     :0           console          Thu Nov 15 16:38   still logged in   
henk     :0           console          Wed Oct 24 17:02 - 18:33 (8+02:31)   
reboot   system boot  2.2.18           Wed Oct 24 17:01         (8+02:33)   

... lines truncated ...

wtmp begins Fri Sep  7 17:48:50 2001

The output displays the user, the (pseudo)terminal the user used to log in, the address of the host from which the user made the connection (if available) and finally a set of fields that specify how long the user has been logged in.

By running last every - let's say - five minutes and checking if anything has changed since the last time we checked, we can generate a report that tells us who has logged in/out since the last time we checked. An outline of the flow:

       (begin)
          |
          +- store current situation in a file
          |
          <is there a history file?>
          |
         / \
        n   y
        |   |
        |   +- make a diff between history file and new file
        |   |
        |   <were there changes since last time?>
        |   |
        |  / \
        | n   y
        | |   |
        | |   +- mail report
        | |   |
        |  \ /
        |   |
         \ /
          |
          +- put current situation into history file
          |
        (end)

and a possible implementation, marked with line-numbers for later reference:

   /----------------------------------------------------------------------
01 | #!/bin/sh
02 |  
03 | # The base directory to put our logs in 
04 | # 
05 | BASEDIR=/var/log/checklogin  
06 |  
07 | # The name of the history- and current file, in which the 
08 | # login information is stored 
09 | # 
10 | HISTORY=${BASEDIR}/history  
11 | CURRENT=${BASEDIR}/current 
12 |  
13 | # A simple generic failure function, aborts after displaying 
14 | # its arguments. 
15 | # 
16 | fail() 
17 | { 
18 |    echo "Failed: $*" 
19 |    exit 1 
20 | } 
21 |  
22 | # This function cleans up the output of 'last', by 
23 | # eliminating empty lines, reboot lines and wtmp  
24 | # lines. 
25 | # 
26 | clean_last() 
27 | { 
28 |    /usr/bin/last | sed '{  
29 |      /^reboot /d 
30 |      /^$/d 
31 |      /^wtmp begins /d 
32 |    }'                           
33 |  
34 | } 
35 |  
36 | MYGROUP=`id -gn` 
37 | MYIDENT=`id -un` 
38 |  
39 | # Check our environment: 
40 | # 
41 | [ -d ${BASEDIR} ] || mkdir -p ${BASEDIR} 
42 | [ -d ${BASEDIR} ] || fail could not create ${BASEDIR} 
43 | [ -G ${BASEDIR} ] || fail ${BASEDIR} not owned by ${MYGROUP} 
44 | [ -O ${BASEDIR} ] || fail ${BASEDIR} not owned by ${MYIDENT} 
45 |  
46 | # store current situation in a file 
47 |  
48 | clean_last >${CURRENT} 
49 | 
50 |  
51 |  
52 | # Is there a history file? 
53 | # 
54 | if [ -f ${HISTORY} ] 
55 | then 
56 |    # Yes - has the situation changed since last time? 
57 |    # 
58 |    if ! `cmp --silent $CURRENT $HISTORY`  
59 |    then 
60 |         # Yes - mail root about it 
61 |         # 
62 |         diff $HISTORY $CURRENT |mail root -s "Login report" 
63 |    fi 
64 | fi 
65 | # 
66 | # Put current situation into history file 
67 | # 
68 | mv ${CURRENT} ${HISTORY} 
69 | [ $? -eq 0 ] || fail mv ${CURRENT} ${HISTORY} 
70 | exit 0 
   \----------------------------------------------------------------------

This script could be run by cron, let's say, every 5 minutes. At line 01, we see the sequences (“hash-bang”) that tell the shell that this script should be run by /bin/sh. On most Linux systems, this will be a link to the bash shell. The script uses a file to maintain its state between invocations (the history file) and a workfile. The history file will contain the login information as it was written at the previous invocation of the script and the workfile will contain the most recent login information. The locations for these files are specified on lines 01-12. Using variables to configure a script is both a common and time-honored practice. That way, you only need to change these variables should you ever desire another location for these files, instead of having to change the names throughout the entire script.

On lines 13-20, a rudimentary fail-function is defined. It prints its arguments and exits, passing the value 1 to the calling shell. Passing a non-zero value to the calling program is, by convention, a signal that some error occurred whilst executing the program. In the rest of the script, we can use the construct:

fail {the text to print before exiting}

which will print:

Failed: {the text to print before exiting}

and will abort the script. If the script is run by cron, any (error)output will be mailed by default to the owner of the entry.

Lines 22-34 contain a function that serves as a wrapper around the last command. Its output is filtered: it removes lines that start with reboot (line 29), empty lines (line 30) and the line that reports when the wtmp file was created (line 31). All other lines are sent through unchanged.

In lines 36 and 37, the identity of the user running the script is determined. These values used to create readable reports on failure. Lines 39-44 check the environment. Line 41 checks if the base directory exists and, if not, tries to create it. Line 42 rechecks the existence of the directory and exits if it (still) is not there. Lines 43 and 44 check whether or not the base-directory is owned by the current user and group. If one of these tests fails, an error messages is created by the fail function and the script is aborted.

Lines 48 and 49 execute our special filtered version of last and send the output to our workfile. That file now contains the most current log information. Lines 52-64 checks whether a history file already exists. This will always be the case, unless this is the first invocation of the script. On line 58, a comparison between the old and new database is made. If there is no difference, nothing is done. Otherwise, the command diff is used to mail the differences to root. Finally, in line 68, the history file is overwritten by the current database.

The aforementioned method has plenty of flaws: not every program logs to wtmp, and the reports will be delayed for, at most, the length of the interval you specified to run the checkscript. Also, the output format of diff is not the most user-friendly output. You may find it a rewarding operation to refine this script yourself.