All things sysadmin

mod_log_forensic is an often forgotten yet very handy tool in debugging webservers. It gives each request a unique ID which you can then track through your log. It first writes the request prefixed with the unique ID, then it writes the same ID once the request is completed. Very useful to spot scripts which never finishes, be it due to client or server issues. You see the entire request including browser information,  cookies etc. Like this:

1
2

+sS6NLH8AAAEAAHoSUdUAAAAB|GET /favicon.ico HTTP/1.1|User-Agent:Opera/9.26 (Windows NT 5.1; U; en)|Host:northernmost.org|Accept:text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1|Accept-Language:en-GB,en;q=0.9|Accept-Charset:iso-8859-1, utf-8, utf-16, *;q=0.1|Accept-Encoding:deflate, gzip, x-gzip, identity, *;q=0|Referer:http%3a//northernmost.org/test.html|Connection:Keep-Alive, TE|TE:deflate, gzip, chunked, identity, trailers
-sS6NLH8AAAEAAHoSUdUAAAAB

When investigating the logs, you usually want to look for any + without a corresponding -, that means that that request was never finalised. You can then look through the script/file in question and see what may have gone wrong.

mod_log_forensic is included in most distribution packages of Apache and comes with the source tarball download, but if you compile Apache 2.x.x from source, you need to add --enable-log-forensic and --enable-unique-id to the configure line.

 In httpd.conf, you will need to have the following lines (preferably in their respective context):

1
2
3

LoadModule log_forensic_module modules/mod_log_forensic.so
LoadModule unique_id_module modules/mod_unique_id.so
ForensicLog logs/forensic_log

If you want, you can have the request ID added to your normal access_log as well by simply editing the LogFormat to look something along the lines of:

1

LogFormat "%{forensic-id}n %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Note that this will most likely break any log-based webstats package you might be using, such as awstats, webalizer etc.

There is a tool supplied with the source distribution of Apache called check_forensic which takes the forensics log as input and outputs any inomplete request. If I for instance were to remove -sS6NLH8AAAEAAHoSUdUAAAAB line from /var/log/httpd/forensics_log, I’d get this from a run:

1
2

# sh check_forensic /var/log/httpd/forensic_log
+sS6NLH8AAAEAAHoSUdUAAAAB|GET /favicon.ico HTTP/1.1|User-Agent:Opera/9.26 (Windows NT 5.1; U; en)|Host:northernmost.org|Accept:text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1|Accept-Language:en-GB,en;q=0.9|Accept-Charset:iso-8859-1, utf-8, utf-16, *;q=0.1|Accept-Encoding:deflate, gzip, x-gzip, identity, *;q=0|Referer:http%3a//northernmost.org/test.html|Connection:Keep-Alive, TE|TE:deflate, gzip, chunked, identity, trailers

You would however have rather serious issues if you had failed requests for favicon.ico ;) Another overlooked useful module you might want/need to use in combination with mod_log_forensic is mod_whatkilledus.