mod_log_forensic is an often forgotten yet very handy tool in debugging webservers. It gives each request a unique ID which you can then track through your log. It first writes the request prefixed with the unique ID, then it writes the same ID once the request is completed. Very useful to spot scripts which never finishes, be it due to client or server issues. You see the entire request including browser information, cookies etc. Like this:
When investigating the logs, you usually want to look for any +
mod_log_forensic is included in most distribution packages of Apache and comes with the source tarball download, but if you compile Apache 2.x.x from source, you need to add
--enable-unique-id to the configure line.
In httpd.conf, you will need to have the following lines (preferably in their respective context):
1 2 3
If you want, you can have the request ID added to your normal access_log as well by simply editing the LogFormat to look something along the lines of:
Note that this will most likely break any log-based webstats package you might be using, such as awstats, webalizer etc.
There is a tool supplied with the source distribution of Apache called check_forensic which takes the forensics log as input and outputs any inomplete request. If I for instance were to remove -sS6NLH8AAAEAAHoSUdUAAAAB line from /var/log/httpd/forensics_log, I’d get this from a run:
You would however have rather serious issues if you had failed requests for favicon.ico ;) Another overlooked useful module you might want/need to use in combination with mod_log_forensic is mod_whatkilledus.