All things sysadmin

Just thought I’d share a little python script I wrote the other day and have found helpful on several occasions already. It’s basically the same as tail -f file | grep pattern, but, if it finds a match, it allows you to execute an arbitrary command.

For instance, the yesterday a customer had problems with Apache infrequently segfaulting. The problem was temporarily fixed by simply restarting Apache, so I used this script to automatically restart apache each time children started segfaulting while we looked for a more long term solution.

Here is the script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#!/usr/bin/python

# This script mimics the behaviour of tail -f , but has the capablity to execute
# a command whenver a line with a matching pattern in it is found.
# As you probably see, there's no support for regex or anything of that
# flashy nature.
#
# Erik Ljungstrom 15/10/07

import time
import sys
import os
import socket

file = open(sys.argv[2],"r");
node=socket.gethostbyaddr(socket.gethostname())[0]
setdef = os.path.getsize(sys.argv[2])
file.seek(setdef-1)
while 1:
    where = file.tell()
    line = file.readline()
    if not line:
        time.sleep(1)
        file.seek(where)
    else:
        pos = line.find(sys.argv[1])
        if pos >= 0:
            os.system("logger -p local6.crit -t postfix amavisd has bailed out on %s!" % (node))
            sys.exit(1)

You start it such as ./pytail.py pattern file. As my comment says, it’s very basic, but useful! Of course it would be nice if it forked into the background and could do regexp matching etc. But I simply haven’t had use for that yet, and screen is a good enoguh tool. The above ‘version’ was used under pretty similar circumstances. A customer’s amavisd kept bailing out due to BDB 3 limtiations (running out of locker entries). So I ran that script while we upgraded it on all their mail servers. So I ran

1

./pytail.py child_init_hook /var/log/messages

The command being executed is the bit between the quotation marks in os.system(). In that case, I had previously set up syslog to log to a MySQL instance on one of my own servers which is then polled from our monitoring panel (2 x 50″ LCD screens in my corner of the office). So whenever this happened, we knew which server had the issue and could mitigate the problem.