Unfortunately I set about this task on an RHEL 5.4 box, and it hasn’t been a walk in the park. Quite a few dependencies were out of date or didn’t exist in the repositories, libicu, boost, onig, tbb etc.

Though, CMake did a good job of telling me what was wrong, so it wasn’t a huge deal, I just compiled the missing pieces from source and put them in $CMAKE_PREFIX_PATH. One thing CMake didn’t pick up on however, was that the flex version shipped with current RHEL is rather outdated. Once I thought I had everything configured, I set about the compilation, and my joy was swiftly abrupted by this:

[  3%] [FLEX][XHPScanner] Building scanner with flex /usr/bin/flex version 2.5.4
/usr/bin/flex: unknown flag '-'.  For usage, try /usr/bin/flex --help

Not entirely sure what it was actually doing here, I took the shortcut of replacing /usr/bin/flex with a shell script which just exited after putting $@ in a file in /tmp/ and re-ran `make`. Looking in the resulting file, this is the argument flex is given:

-C --header-file=scanner.lex.hpp -o/home/erik/dev/hiphop-php/src/third_party/xhp/xhp/scanner.lex.cpp /home/erik/dev/hiphop-php/src/third_party/xhp/xhp/scanner.l

To me that looks quite valid, and there’s certainly no single – in that command line.

Long story short, flex introduced –header-file in a relatively “recent” version (2.5.33 it seems, but I may be wrong on that one, doesn’t matter). Unlike most other programs (using getopt), it won’t tell you “Invalid option ‘–header-file’”. So after compiling a newer version of flex, I was sailing again.

So as of now, you can safely use one varnish instance for several front-ends, thus eliminate double-caching (memory waste, unnecessary load on back-ends), reduce network traffic, do rudimentary load balancing, ease management etc.
With the obscene amount of traffic Varnish can push without putting a fairly basic system under any load worth mentioning, you can use a single front-end to serve several nodes in most setups.

Here’s an elementary sample VCL for how to do this:

backend node0 {
  .host = "127.0.0.1";
  .port = "80";
  .probe = {
           .url = "/";
           .timeout = 50 ms;
           .interval = 1s;
           .window = 10;
           .threshold = 8;

  }
}

backend node1 {
  .host = "10.0.0.2";
  .port = "80";
  .probe = {
#           .url = "/";
           .timeout = 100 ms;
           .interval = 1s;
           .window = 10;
           .threshold = 8;
        .request =
            "GET /healthcheck.php HTTP/1.1"
            "Host: 10.0.0.2"
            "Connection: close"
            "Accept-Encoding: foo/bar" ;
  }
}
director cl1 random {
    { .backend = node0; .weight = 1; }
    { .backend = node1; .weight = 1; }
}

#director cl1 round-robin {
#   { .backend = node1; }
#   { .backend = node0; }
#}

sub vcl_recv {
        set req.backend = cl1;
}

As you can see I’m defining the backends slightly differently. You need to define one of .url or .request, but not both for obvious reasons. If you go for the slighly simpler .url the default request looks like this:

GET / HTTP/1.1
Host: something
Connection: close

If this does not suit your need, comment out .url and use .request to roll your own. This aspect of Varnish is actually quite well documented, so I won’t repeat what’s on the trac page.

There is clearly a lot more you can and, more often than not, should do in the VCL than the above. This is a stripped down version which only pertains to the backend polling functionality.

While I’ve experienced some of the “oddities” they refer to, I have every bit of confidence in the developers. Even so – it’s a risky path to go down. They will most likely iron out the current shortcomings and oddities, but it’s fairly likely that a few new will be introduced along the way. I do however believe the planned use of the well proven glib is likely to prevent some of them.

It will be mighty interesting to see the impact of using libev for managing events. It certainly helps when “flying light”. Another interesting thing will be to see what plugins people come up with!

Graceful restarts will be most welcomned as well!

Go go lighttpd!

+sS6NLH8AAAEAAHoSUdUAAAAB|GET /favicon.ico HTTP/1.1|User-Agent:Opera/9.26 (Windows NT 5.1; U; en)|Host:northernmost.org|Accept:text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1|Accept-Language:en-GB,en;q=0.9|Accept-Charset:iso-8859-1, utf-8, utf-16, *;q=0.1|Accept-Encoding:deflate, gzip, x-gzip, identity, *;q=0|Referer:http%3a//northernmost.org/test.html|Connection:Keep-Alive, TE|TE:deflate, gzip, chunked, identity, trailers
-sS6NLH8AAAEAAHoSUdUAAAAB


When investigating the logs, you usually want to look for any +<ID> without a corresponding -<ID>, that means that that request was never finalised. You can then look through the script/file in question and see what may have gone wrong.

mod_log_forensic is included in most distribution packages of Apache and comes with the source tarball download, but if you compile Apache 2.x.x from source, you need to add --enable-log-forensic and --enable-unique-id to the configure line.

 In httpd.conf, you will need to have the following lines (preferably in their respective context):

LoadModule log_forensic_module modules/mod_log_forensic.so
LoadModule unique_id_module modules/mod_unique_id.so
ForensicLog logs/forensic_log

If you want, you can have the request ID added to your normal access_log as well by simply editing the LogFormat to look something along the lines of:

LogFormat "%{forensic-id}n %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Note that this will most likely break any log-based webstats package you might be using, such as awstats, webalizer etc.

There is a tool supplied with the source distribution of Apache called check_forensic which takes the forensics log as input and outputs any inomplete request. If I for instance were to remove -sS6NLH8AAAEAAHoSUdUAAAAB line from /var/log/httpd/forensics_log, I’d get this from a run:

# sh check_forensic /var/log/httpd/forensic_log
+sS6NLH8AAAEAAHoSUdUAAAAB|GET /favicon.ico HTTP/1.1|User-Agent:Opera/9.26 (Windows NT 5.1; U; en)|Host:northernmost.org|Accept:text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1|Accept-Language:en-GB,en;q=0.9|Accept-Charset:iso-8859-1, utf-8, utf-16, *;q=0.1|Accept-Encoding:deflate, gzip, x-gzip, identity, *;q=0|Referer:http%3a//northernmost.org/test.html|Connection:Keep-Alive, TE|TE:deflate, gzip, chunked, identity, trailers

You would however have rather serious issues if you had failed requests for favicon.ico ;) Another overlooked useful module you might want/need to use in combination with mod_log_forensic is mod_whatkilledus.

/usr/local/src/php-5.1.4/ext/mysqli/mysqli_api.c: In function 'zif_mysqli_stmt_bind_param':
/usr/local/src/php-5.1.4/ext/mysqli/mysqli_api.c:145: error: 'gptr' undeclared (first use in this function)
/usr/local/src/php-5.1.4/ext/mysqli/mysqli_api.c:145: error: (Each undeclared identifier is reported only once
/usr/local/src/php-5.1.4/ext/mysqli/mysqli_api.c:145: error: for each function it appears in.)
/usr/local/src/php-5.1.4/ext/mysqli/mysqli_api.c: In function 'zif_mysqli_stmt_execute':
/usr/local/src/php-5.1.4/ext/mysqli/mysqli_api.c:601: error: 'gptr' undeclared (first use in this function)

To fix this, either upgrade to 5.2.4, or if you’re left with no choice such as myself edit php-5.1.4/ext/mysqli/mysqli_api.c and replace all occurrances of gptr with char*

That ought to get you through the rest of the compilation provided that you’ve got the rest of your setup correct.

This post might not be very relevant to the audience I mainly write this blog for, but if it can save me or someone else with on-call duties just one sleepless night, it’ll be worth it!

Today it happened again. At 2 am an SMS message from our beloved friend and saviour NetFlow woke me up from my much needed beauty sleep. 120 000 outbound packets per second from a customer’s machine. Null routed, great! As I made my way to the datacenter I thought “I bet it’s another one of those web-hack”

So I figured I should write about it here in hope that some web hoster (a lot of our customers are) reads this and clocks on!

The ‘hack’ was indeed a classic one, a non-patched/outdated Joomla install with a cross-site scripting vulnerability. This post will only rant about this particular type of intrusion, and by no means web app hacks in general, although most of it will be useful in other contexts as well.

I think there are somewhere around 19 different worms/scanners for Joomla out in the wild. I’m guessing it’s because the installation requires some knowledge of apache, php and elementary understanding of access control. Almost more often than not, I see these installations ending up with register_globals turned on, allow_url_fopen turned on, safe_mode off and chmod -R 777 /var/www/

Brilliant, isn’t it?

Since 9 out of 10 hacks we’ve seen lately happens due to insecure web applications which typically run as the Apache user which has limited write privileges, the first thing I do when investigating a hacked server is:

ls -al /tmp ; ls -al /var/tmp

This is a popular place for script kiddies to put their (read other people’s) scripts of doom. Nothing there! It was now clear to me that I was dealing with a mighty level 2 script kiddie!

So I ran:

find /var/www/htdocs/ -name "*.php" | xargs grep -i r57 > /root/r57

r57 is the spawn of satan on your harddrive. It’s a single-file PHP script which allows you to do pretty much everything you would if you had shell access with the permissions of apache (you technically do since you’ve even got a command line). It’s got a database section which allows you to dump databases and run arbitrary queries, an FTP interface for uploading and downloading files, a callback functionality etc. In a nutshell; not something you’ll want anywhere under your DocumentRoot! Turns out the 120 000 packets per second originated from a script hidden well inside the directory structure of the site called ‘udp.pl’. This was downloaded and executed through the r57 shell. Another drawback of 777 permissions, files such as those can be hidden anywhere!

So, in order to stop this particular type hack, you’d want to keep your privileges strict! Do not allow apache to write anywhere where it’s not strictly necessary! Although I do realise that some directories requires write permissions if you think (S)FTP is too much like voodoo to get your images onto your site, most directories don’t!

The second thing is to disable some of the functions these scripts love to use. In your php.ini (php -i |grep .ini will tell you where it is), find the line starting with disable_functions, and make it look like this:

disable_functions = system,exec,execl,shell_exec,passthru,ftp_login,popen

r57 and its derivatives (c99 etc.) needs these to do most of its tasks. Since these scripts has a front-end for emailing, in order to prevent ending up in a handful of blacklists, you could add ‘mail’ to the list as well if your website doesn’t need to send emails or use a socket based solution (a la phpmailer). If at all possible, you will want open_basedir and safe_mode enforcing! register_globals should be Off as well. If it requires you to rewrite part of your website to cater for this, you still might want to consider it. It could save you a lot of work in the future. The above mentioned customer had 6 hours of downtime during a reinstall and checking over their scripts (their backups were contaminated as well and the above command does obviously not find everything malicious!).

Another PHP setting you should turn off if possible is allow_url_fopen which basically means that if your webapp doesn’t do proper input validation and opens a file requested by a user, a URL could be passed instead of a local file! The URL could contain a script which the visitor then can use to do as he wishes.

But, the number one precaution; write good code!

Another useful thing you can do to if you suspect there’s been some foul play going on is:

netstat -tpanu |grep 666[78]

which will tell you if there are any in- or outbound IRC connections from the server. If there are, it is quite likely that it’s compromised as a majority of hacks are made by script kiddies looking for a place to put their IRC bots.

Well, I should say, a majority of the hacks that are discovered are.

Keeping track of your bandwidth consumption and webstats is useful as well as a lot of hacked servers acts as filedumps for less able warez groups. If you one month see a 40% increase in bandwidth usage, but no corresponding increase in hits, you either have an employee taking liberties, or a 14 year old doing the same from his home PC.

If your site is fairly static in terms of files in the directory structure, you can use ‘find’ to list all files created after a certain date. If you haven’t uploaded these, chances are above mentioned 14 year old has. This command will show you all files which has been modified in the last 24 hours:

find /var/www/htdocs/ -type f -mtime -1

So, to sum up; As a very basic start; write proper code, restrict filesystem permissions, enable open_basedir and safe_mode, turn allow_url_fopen off, apply the above mentioned disable_functions and disable register_globals.

There are obviously a lot more you can do than the above, those are just the very basics! If you want to go a bit more hardcore, you could look into running PHP as CGI with proper OS level user separation, using SuExec, noexec mounted partitions, chroot environments etc. But that’s perhaps another post!

This particular solution has got five loadbalanced webservers and two database back-ends designed to cater for a reasonably high amount of traffic and shorter bursts of traffic surges.

The servers did not seem to be under any particular load, and I could connect to localhost just fine (telnet is a magnificent troubleshooting tool, is it not?). I looked at the cacti graphs on all four servers in the cluster, and noticed in the apache scoreboard that there were more or less exactly 256 threads running on all five servers. `ps ax|grep httpd | wc -l`confirmed this. So I went and edited httpd.conf and raised ServerLimit and MaxClients to 750 (from 256).

Then here’s my mistake – being the nice guy that I am – I did /usr/local/apache/bin/apachectl graceful . But got this in error_log:

WARNING: MaxClients of 750 exceeds ServerLimit value of 256 servers,lowering MaxClients to 256. To increase, please see the ServerLimit directive.

Hm, why would maxclients be respected, but not serverlimit? Short answer is; when doing graceful, the parent apache uber-process is not killed and restarted and this value can not be changed without a full blown restart. Pretty obvious once you know about it! I sifted through apachectl, and all it did was:…

case $ARGV instart|stop|restart|graceful|graceful-stop)
$HTTPD -k $ARGV ERROR=$? ;;

…so obviously an `apachectl restart` would not be sufficient either. So long story short, when changing the ServerLimitdirective, a graceful-stop and a start is necessary! At least this is true for this custom compiled apache 2.2.3.

The tests were made using apachebench on two dedicated machines with no other tasks and over a dedicated 100 mbit network. Some figures really surprises me/strikes me as odd, such as the varied php performance of lighttpd utilising fastcgi!

I suppose it would only have been fair to test apache with fastcgi as well, but thatwas out of scope for their request.Should you be interested in the full AB output, you can download a tarball containing all the data here. See attachment for diagrams and figures.