It turns out that the grass actually IS greener. Tonight I stumbled upon this. It’s a patched version of freetype. For what I assume are political reasons (free as in speech), Fedora ships a Freetype version without subpixel rendering. These patches fixes that and other things.

With a default configuration file of 407 lines, it’s quite extensible and configurable as well. Lucky, I quite like the default!

If you’re not entirely happy with the way your fonts look on Fedora – it’s well worth a look

Either way, starting from kernel 2.6.16, we can find out using smaps which can be found in the proc filesystem. I’ve written a simple bash script which prints out all running processes and their swap usage.
It’s quick and dirty, but does the job and can easily be modified to work on any info exposed in /proc/$PID/smaps
If I find the time and inspiration, I might tidy it up and extend it a bit to cover some more alternatives. The output is in kilobytes.

#!/bin/bash
# Get current swap usage for all running processes
# Erik Ljungstrom 27/05/2011
SUM=0
OVERALL=0
for DIR in `find /proc/ -maxdepth 1 -type d | egrep "^/proc/[0-9]"` ; do
PID=`echo $DIR | cut -d / -f 3`
PROGNAME=`ps -p $PID -o comm --no-headers`
for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'`
do
let SUM=$SUM+$SWAP
done
echo "PID=$PID - Swap used: $SUM - ($PROGNAME )"
let OVERALL=$OVERALL+$SUM
SUM=0

done
echo "Overall swap used: $OVERALL"

This will need to be ran as root for it to be able to gather accurate numbers. It will still work even if you don’t, but it will report 0 for any processes not owned by your user.
Needless to say, it’s Linux only. The output is ordered alphabetically according to your locale (which admittedly isn’t a great thing since we’re dealing with numbers), but you can easily apply your standard shell magic to the output. For instance, to find the process with most swap used, just run the script like so:

$ ./getswap.sh | sort -n -k 5
Don’t want to see stuff that’s not using swap at all?
$ ./getswap.sh | egrep -v "Swap used: 0" |sort -n -k 5

… and so on and so forth

So far so good! A week or so later, we often get the call “Our page load time is higher now than before the upgrade! We’ve got twice as much hardware, and it’s slower! You have broken it!”
It’s easy to see where they’re coming from. It makes sense, right?

That is until you factor in the newly introduced network topology! Today it’s not unusual (that’s not to say it’s acceptable or optimal) for your average
wordpress/drupal/joomla/otherspawnofsatan site to run 40-50 queries per page load. Quite often even more!

Based on a tcpdump session of a reasonably average query (if there is such a thing), connecting to a server, authenticating, sending a query and receiving a 5 row result set of 1434 bytes yields 25 packets being sent between my laptop and a remote DB server on the same wired, non-congested network. A normal, average latency of TCP/IP over Ethernet is ~0.2 ms for the size of packets we’re talking here.
So, doing the maths, you’re seeing 25*0.2*50= 250ms in just network latency per page load for your SQL queries. This is obviously a lot more than you see over a local UNIX socket.

This is inevitable, laws of physics. It is nothing you, your sysadmin and/or your hosting company can do anything about. There may however be something your developer can do about the amount of queries!
You also shouldn’t confuse response-times with availability. Your response times may be slower, but you can (hopefully) serve a lot more users with this setup!

Sure, there are technologies out there which have considerably less latency than ethernet, but they come with quite the price-tag, and there are more often than not quite a few avenues to go down before it makes sense to start looking at that kind of thing.

You could also potentially looking at running the full stack on both machines using master/master replication for your DBs, and load balance your front-ends and have them both read locally, but only write to one node at a time! That kind of DB scenario is something fairly easily set up using mmm for MySQL. But in my experience, this often ends up more costly and potentially introducing more complexities than it solves.
I’m an avid advocate for keeping server roles separate as much as possible!

I got most of it set up, and got started on writing up the glusterfs Puppet module. Fairly straight forward, a few directories, configuration files and a mount point. Then I came to the Service declaration, and of course we want this to be running at all times, so I went on and wrote:

service { "glusterfsd":
ensure => running,
enable => true,
hasrestart => true,
hasstatus => true,
}

expecting glusterfsd to be running shortly after I purposefully stopped it. But it didn’t. So I dove into puppet (Yay Ruby!) and deduced that the way it determines whether something is running or not is the return code of:
/sbin/service servicename status

So a quick look in the init script which ships with glusterfs-server shows that it calls the stock init function “status” on glusterfsd, which is perfectly fine, but then it doesn’t exit with the return code from this function, it simply runs out of scope and exits with the default value of 0.

So to get around this, I made a quick change to the init script and used the return code from the “status” function (/etc/rc.d/init.d/functions on RHEL5) and exited with $?, and Puppet had glusterfsd running within minutes.

I couldn’t find anything when searching for this, so I thought I’d make a note of it here.

...
[2162] dbg: learn: initializing learner
[2162] dbg: check: is spam? score=4.004 required=6
[2162] dbg: check: tests=FH_DATE_PAST_20XX,HTML_MESSAGE,SPF_HELO_PASS
...

4 is obviously quite a high score for an email whose only flaw is being in HTML. But FH_DATE_PAST_20XX caught my eye in all of the outputs. So to the rule files:

$ grep FH_DATE_PAST_20XX /usr/share/spamassassin/72_active.cf
##{ FH_DATE_PAST_20XX
header FH_DATE_PAST_20XX Date =~ /20[1-9][0-9]/ [if-unset: 2006]
describe FH_DATE_PAST_20XX The date is grossly in the future.
##} FH_DATE_PAST_20XX

Aha. This is a problem. With 50_scores.cf containing this:

$ grep FH_DATE_PAST /usr/share/spamassassin/50_scores.cf
score FH_DATE_PAST_20XX 2.075 3.384 3.554 3.188 # n=2

there’s no wonder emails are getting dropped! I guess this is a problem one can expect when running a distribution with packages 6 years old and neglect to frequently (or at least every once in a while) update the rules!

Luckily, this rule is gone altogether from RHEL6′s version of spamassassin.

It’s relatively simple to set up, and configuration can be done in two different ways. You can use the supplied cgset command, or if you’re accustomed to doing it the usual way when dealing with kernel settings, you can simply echo values into the pseudo-files under the control group.

Here’s a controlgroup in action:

[root@rhel6beta cgtest]# grep $$ /cgroup/gen/group1/tasks
1138
[root@rhel6beta cgtest]# cat /cgroup/gen/group1/memory.limit_in_bytes
536870912
[root@rhel6beta cgtest]# gcc alloc.c -o alloc && ./alloc
Allocating 642355200 bytes of RAM,,,
Killed
[root@rhel6beta cgtest]# echo `echo 1024*1024*1024| bc` > /cgroup/gen/group1/memory.limit_in_bytes
[root@rhel6beta cgtest]# ./alloc
Allocating 642355200 bytes of RAM,,,
Successfully allocated 642355200 bytes of RAM, captn' Erik...
[root@rhel6beta cgtest]#

The first line shows that the shell which launches the app is under the control of the cgroup group1, so subsequently all it’s child processes are subject to the same restrictions.

As you can also see, the initial memory limit in the group is 512M. Alloc is a simple C app I wrote which calloc()s 612M of RAM (for demonstrative purposes, I’ve disabled swap on the system altogether). At the first run, the kernel kills the process in the same way it would if the whole system had run out of memory. The kernel message also indicates that the control group ran out of memory, and not the system as a whole:

...
May 13 17:56:20 rhel6beta kernel: Memory cgroup out of memory: kill process 1710 (alloc) score 9861 or a child
May 13 17:56:20 rhel6beta kernel: Killed process 1710 (alloc)

Unfortunately it doesn’t indicate which cgroup the process belonged to. Maybe it should?

cgroups doesn’t just give you the ability to limit the amount of RAM, it has a lot of tuneables. You can even set swappiness on a per-group basis! You can limit the devices applications are allowed to access, you can freeze processes as well as tag outgoing network packets with a class ID, in case you want to do shaping or profiling on your network! Perfect if you want to prioritise SSH traffic over anything else, so you can comfortably worked even when your uplink is saturated. Furthermore, you can easily get an overview of memory usage, CPU accounting etc. of applications in any given group.

All this means you can clearly separate resources and to quite a large extent ensure that some applications won’t starve the whole system, or each other from resources. Very handy, no more waiting for half an hour for the swap to fill up and OOM to kick (and often chose the wrong PID) in when customer’s applications have run astray.

A much welcomed addition to RHEL!

Unfortunately I set about this task on an RHEL 5.4 box, and it hasn’t been a walk in the park. Quite a few dependencies were out of date or didn’t exist in the repositories, libicu, boost, onig, tbb etc.

Though, CMake did a good job of telling me what was wrong, so it wasn’t a huge deal, I just compiled the missing pieces from source and put them in $CMAKE_PREFIX_PATH. One thing CMake didn’t pick up on however, was that the flex version shipped with current RHEL is rather outdated. Once I thought I had everything configured, I set about the compilation, and my joy was swiftly abrupted by this:

[  3%] [FLEX][XHPScanner] Building scanner with flex /usr/bin/flex version 2.5.4
/usr/bin/flex: unknown flag '-'.  For usage, try /usr/bin/flex --help

Not entirely sure what it was actually doing here, I took the shortcut of replacing /usr/bin/flex with a shell script which just exited after putting $@ in a file in /tmp/ and re-ran `make`. Looking in the resulting file, this is the argument flex is given:

-C --header-file=scanner.lex.hpp -o/home/erik/dev/hiphop-php/src/third_party/xhp/xhp/scanner.lex.cpp /home/erik/dev/hiphop-php/src/third_party/xhp/xhp/scanner.l

To me that looks quite valid, and there’s certainly no single – in that command line.

Long story short, flex introduced –header-file in a relatively “recent” version (2.5.33 it seems, but I may be wrong on that one, doesn’t matter). Unlike most other programs (using getopt), it won’t tell you “Invalid option ‘–header-file’”. So after compiling a newer version of flex, I was sailing again.

So I put some effort into finding a solution to this, and whilst reading through the bash manpage, I saw PROMPT_COMMAND. *pling*
export PROMPT_COMMAND='history -a'

To quote the manpage: “If set, the value is executed as a command prior to issuing each primary prompt.”
So every time my command has finished, it appends the unwritten history item to ~/.bash_history before displaying the prompt (only $PS1) again.

So after putting that line in /etc/bashrc I don’t have to find myself reinventing wheels or lose valuable seconds re-typing stuff just because I was lazy with my terminals.

This is one of those things that I should have done ages ago, but never took the time to.

btrfs, like ZFS, implements copy-on-write model, so yes – it will be able to do snapshots! Writeable ones at that. In fact, it’s got the ability to do snapshots of snapshots! Quasi-MVC filesystem!
COW unfortunately makes a filesystem more prone to fragmentation, but luckily btrfs comes with online defragmentation and fs check abilities. The speed of read and write operations will obviously be impaired during such operations, but there’s always ways around that in most performance sensitive setups! If not, there should be!
Sadly, COW isn’t that good of a choice for database workloads. But fret not, COW can be disabled with a mount option (-o nodatacow). This doesn’t mean you will lose the snapshot ability, as btrfs ignores this option if a data extent is referenced by more than one snapshot, so COW will, as far as I understand, be enabled from that you initiate a snapshot and stay that way until you’re done with it.

Early benchmarks show that btrfs is extremely fast at writing, and a little poorer at reading. It will be interesting to see how these numbers change as development proceeds. If added features will have any negative impact on performance. As a side note – I was quite surprised to see the poor numbers for ext3 in these benchmarks!

So if you’re a DBA and your data fits in memory, this filesystem will be right up your alley. With a reasonable amount of tables and some proper values for innodb_open_files and table_cache, I wouldn’t expect any remarkable difference in day-to-day database operation since the real bottleneck usually is in the hardware.
This is generally speaking of course. I’m sure there are workloads out there which will benefit a lot more than “the norm”. Likewise, people with awkward read heavy setups with a lot of data in a lot of files may probably be better off not using btrfs.
If you, like myself, often use blinks of an eye as a unit, you know what I’m talking about.

Yet another interesting functionality built in is the multiple device support. I will not call it a substitute for proper hardware based RAID, but could well be one for LVM (bearing the snapshots in mind as well)!

Another thing worth keeping an eye on is a related project; CRFS which may turn out to be a worthy NFS replacement. While it’s planned to get failover capabilities, I would much rather have seen a client-agnostic MogileFS- style implementation.

Sadly, they are not production ready yet. By far. But it’s something to look forward to. I’ll give it a version or two until I will put it under the microscope further and chuck some real world load onto it. Can’t wait!

‘restart’ will effectively increase a counter by 1 and re-run vcl_recv(). You can set how many times a restart should take place before giving up entirely – should you not use the counter in a condition prior to it reaching the limit – by starting varnishd with -p max_restarts=n or ‘param.set max_restarts 1′ on the CLI. This variable defaults to 4, and you can of course set conditions depending on the number of restarts.

Here’s a sample VCL to do this:

backend be1 {
.host = "127.0.0.1";
.port = "81";
}
backend be2 {
.host = "10.0.0.2";
.port = "81";
}

sub vcl_fetch {
if (obj.status != 200 && obj.status != 302) {
restart;
}
}


In this simple VCL, a request destined for this instance of Varnish which doesn’t return 200 or 302 from the backend, is effectively sent to 10.0.0.2 which may have something else in store for the visitor!

If I for instance use the above VCL and set be1 to return a 301 for / and send a request to Varnish, this is what shows up in varnishlog:

…
10 ObjProtocol c HTTP/1.1
10 ObjStatus c 301
10 ObjResponse c Moved Permanently
10 ObjHeader c Date: Tue, 22 Jul 2008 00:25:29 GMT
10 ObjHeader c Server: Apache/2.0.59 (CentOS)
10 ObjHeader c X-Powered-By: PHP/5.1.6
10 ObjHeader c Location: http://be1.northernmost.org:6081/links.php/
10 ObjHeader c Content-Type: text/html; charset=UTF-8
13 BackendClose b be1
10 TTL c 1839681264 RFC 120 1216686329 1216686329 0 0 0
10 VCL_call c fetch
10 VCL_return c restart
10 VCL_call c recv
10 VCL_return c lookup
10 VCL_call c hash
10 VCL_return c hash
10 VCL_call c miss
10 VCL_return c fetch
12 BackendClose b be2
12 BackendOpen b be2 10.0.0.1 38478 10.0.0.2 81
12 TxRequest b GET
12 TxURL b /
12 TxProtocol b HTTP/1.1
…
10 ObjProtocol c HTTP/1.1
10 ObjStatus c 200
10 ObjResponse c OK
10 ObjHeader c Date: Mon, 21 Jul 2008 23:37:24 GMT
10 ObjHeader c Server: Apache/2.2.6 (FreeBSD) mod_ssl/2.2.6 OpenSSL/0.9.8e DAV/2
10 ObjHeader c Last-Modified: Thu, 10 Jul 2008 14:26:46 GMT
10 ObjHeader c ETag: “35e801-3-3702d580″
10 ObjHeader c Content-Type: text/html
12 BackendReuse b be2
…

You can of course use this for very basic resilience as well, but that’s definitely a job for your load balancer. Also be aware about the overhead in this, since the request after all is sent to the backend and processed before passed on to the other node.

Maybe it’s not the most useful feature in the world, but I thought it was nifty!