Hacked webapps

Disclaimer: I say ‘hack’ and ‘hacker’, both you jargon puritans out there and myself know they’re not. Really. But it’s a short and easy to spell word and with this disclaimer, everyone will know what I mean. Including the audience I mainly target with this post!

This post might not be very relevant to the audience I mainly write this blog for, but if it can save me or someone else with on-call duties just one sleepless night, it’ll be worth it!

Today it happened again. At 2 am an SMS message from our beloved friend and saviour NetFlow woke me up from my much needed beauty sleep. 120 000 outbound packets per second from a customer’s machine. Null routed, great! As I made my way to the datacenter I thought “I bet it’s another one of those web-hack”

So I figured I should write about it here in hope that some web hoster (a lot of our customers are) reads this and clocks on!

The ‘hack’ was indeed a classic one, a non-patched/outdated Joomla install with a cross-site scripting vulnerability. This post will only rant about this particular type of intrusion, and by no means web app hacks in general, although most of it will be useful in other contexts as well.

I think there are somewhere around 19 different worms/scanners for Joomla out in the wild. I’m guessing it’s because the installation requires some knowledge of apache, php and elementary understanding of access control. Almost more often than not, I see these installations ending up with register_globals turned on, allow_url_fopen turned on, safe_mode off and chmod -R 777 /var/www/

Brilliant, isn’t it?

Since 9 out of 10 hacks we’ve seen lately happens due to insecure web applications which typically run as the Apache user which has limited write privileges, the first thing I do when investigating a hacked server is:

ls -al /tmp ; ls -al /var/tmp

This is a popular place for script kiddies to put their (read other people’s) scripts of doom. Nothing there! It was now clear to me that I was dealing with a mighty level 2 script kiddie!

So I ran:

find /var/www/htdocs/ -name "*.php" | xargs grep -i r57 > /root/r57

r57 is the spawn of satan on your harddrive. It’s a single-file PHP script which allows you to do pretty much everything you would if you had shell access with the permissions of apache (you technically do since you’ve even got a command line). It’s got a database section which allows you to dump databases and run arbitrary queries, an FTP interface for uploading and downloading files, a callback functionality etc. In a nutshell; not something you’ll want anywhere under your DocumentRoot! Turns out the 120 000 packets per second originated from a script hidden well inside the directory structure of the site called ‘udp.pl’. This was downloaded and executed through the r57 shell. Another drawback of 777 permissions, files such as those can be hidden anywhere!

So, in order to stop this particular type hack, you’d want to keep your privileges strict! Do not allow apache to write anywhere where it’s not strictly necessary! Although I do realise that some directories requires write permissions if you think (S)FTP is too much like voodoo to get your images onto your site, most directories don’t!

The second thing is to disable some of the functions these scripts love to use. In your php.ini (php -i |grep .ini will tell you where it is), find the line starting with disable_functions, and make it look like this:

disable_functions = system,exec,execl,shell_exec,passthru,ftp_login,popen

r57 and its derivatives (c99 etc.) needs these to do most of its tasks. Since these scripts has a front-end for emailing, in order to prevent ending up in a handful of blacklists, you could add ‘mail’ to the list as well if your website doesn’t need to send emails or use a socket based solution (a la phpmailer). If at all possible, you will want open_basedir and safe_mode enforcing! register_globals should be Off as well. If it requires you to rewrite part of your website to cater for this, you still might want to consider it. It could save you a lot of work in the future. The above mentioned customer had 6 hours of downtime during a reinstall and checking over their scripts (their backups were contaminated as well and the above command does obviously not find everything malicious!).

Another PHP setting you should turn off if possible is allow_url_fopen which basically means that if your webapp doesn’t do proper input validation and opens a file requested by a user, a URL could be passed instead of a local file! The URL could contain a script which the visitor then can use to do as he wishes.

But, the number one precaution; write good code!

Another useful thing you can do to if you suspect there’s been some foul play going on is:

netstat -tpanu |grep 666[78]

which will tell you if there are any in- or outbound IRC connections from the server. If there are, it is quite likely that it’s compromised as a majority of hacks are made by script kiddies looking for a place to put their IRC bots.

Well, I should say, a majority of the hacks that are discovered are.

Keeping track of your bandwidth consumption and webstats is useful as well as a lot of hacked servers acts as filedumps for less able warez groups. If you one month see a 40% increase in bandwidth usage, but no corresponding increase in hits, you either have an employee taking liberties, or a 14 year old doing the same from his home PC.

If your site is fairly static in terms of files in the directory structure, you can use ‘find’ to list all files created after a certain date. If you haven’t uploaded these, chances are above mentioned 14 year old has. This command will show you all files which has been modified in the last 24 hours:

find /var/www/htdocs/ -type f -mtime -1

So, to sum up; As a very basic start; write proper code, restrict filesystem permissions, enable open_basedir and safe_mode, turn allow_url_fopen off, apply the above mentioned disable_functions and disable register_globals.

There are obviously a lot more you can do than the above, those are just the very basics! If you want to go a bit more hardcore, you could look into running PHP as CGI with proper OS level user separation, using SuExec, noexec mounted partitions, chroot environments etc. But that’s perhaps another post!