Linked from a post on the lighttpd blog is a page outlining the plans for lighttpd 2.0

While I’ve experienced some of the “oddities” they refer to, I have every bit of confidence in the developers. Even so - it’s a risky path to go down. They will most likely iron out the current shortcomings and oddities, but it’s fairly likely that a few new will be introduced along the way. I do however believe the planned use of the well proven glib is likely to prevent some of them.

It will be mighty interesting to see the impact of using libev for managing events. It certainly helps when “flying light”. Another interesting thing will be to see what plugins people come up with!

Graceful restarts will be most welcomned as well!

Go go lighttpd!

As of now, Varnish has no means to detect whether a backend is available or at good health before sending a request (periodic checking is scheduled for ver 2.0 and will presumably work with the cluster mode as well). So if you’ve got two or more backends, and under some condition can’t or won’t serve a request immediately or want to send it elsewhere depending on some circumstance, you can do this using HTTP return code or header with the not-so-well-documented feature ‘restart’ (then again, what feature is well documented in Varnish?).

‘restart’ will effectively increase a counter by 1 and re-run vcl_recv(). You can set how many times a restart should take place before giving up entirely - should you not use the counter in a condition prior to it reaching the limit - by starting varnishd with -p max_restarts=n or ‘param.set max_restarts 1′ on the CLI. This variable defaults to 4, and you can of course set conditions depending on the number of restarts.

Here’s a sample VCL to do this:

backend be1 {
.host = "127.0.0.1";
.port = "81";
}
backend be2 {
.host = "10.0.0.2";
.port = "81";
}

sub vcl_fetch {
if (obj.status != 200 && obj.status != 302) {
restart;
}
}


In this simple VCL, a request destined for this instance of Varnish which doesn’t return 200 or 302 from the backend, is effectively sent to 10.0.0.2 which may have something else in store for the visitor!

If I for instance use the above VCL and set be1 to return a 301 for / and send a request to Varnish, this is what shows up in varnishlog:

…
10 ObjProtocol c HTTP/1.1
10 ObjStatus c 301
10 ObjResponse c Moved Permanently
10 ObjHeader c Date: Tue, 22 Jul 2008 00:25:29 GMT
10 ObjHeader c Server: Apache/2.0.59 (CentOS)
10 ObjHeader c X-Powered-By: PHP/5.1.6
10 ObjHeader c Location: http://be1.northernmost.org:6081/links.php/
10 ObjHeader c Content-Type: text/html; charset=UTF-8
13 BackendClose b be1
10 TTL c 1839681264 RFC 120 1216686329 1216686329 0 0 0
10 VCL_call c fetch
10 VCL_return c restart
10 VCL_call c recv
10 VCL_return c lookup
10 VCL_call c hash
10 VCL_return c hash
10 VCL_call c miss
10 VCL_return c fetch
12 BackendClose b be2
12 BackendOpen b be2 10.0.0.1 38478 10.0.0.2 81
12 TxRequest b GET
12 TxURL b /
12 TxProtocol b HTTP/1.1
…
10 ObjProtocol c HTTP/1.1
10 ObjStatus c 200
10 ObjResponse c OK
10 ObjHeader c Date: Mon, 21 Jul 2008 23:37:24 GMT
10 ObjHeader c Server: Apache/2.2.6 (FreeBSD) mod_ssl/2.2.6 OpenSSL/0.9.8e DAV/2
10 ObjHeader c Last-Modified: Thu, 10 Jul 2008 14:26:46 GMT
10 ObjHeader c ETag: “35e801-3-3702d580″
10 ObjHeader c Content-Type: text/html
12 BackendReuse b be2
…

You can of course use this for very basic resilience as well, but that’s definitely a job for your load balancer. Also be aware about the overhead in this, since the request after all is sent to the backend and processed before passed on to the other node.

Maybe it’s not the most useful feature in the world, but I thought it was nifty!

When adding a new disk for a customer running CentOS 4.7 on severely old hardware, I bumped into something I’ve never had happening to me before. Basically the system wouldn’t let me create the Physical Volume and gave me this message:

[root@gwyneth ~]# pvcreate /dev/hdc1
Can’t open /dev/hdc1 exclusively. Mounted filesystem?

hdc1 was obviously not mounted or in any other way used. Or so I thought. As I was flicking through the loaded kernel modules, I saw the dm_* modules being loaded and I was quite sure I knew what it was at that point. dmraid is hogging the disk. I verified that dmraid was indeed aware of the new disk with:

[root@gwyneth ~]# dmraid -r
/dev/hdc: pdc, “pdc_hceidaeha”, mirror, ok, 78125000 sectors, data@ 0

So deactivating the (in)appropriate RAID set:

[root@gwyneth ~]# dmraid -a no pdc
[root@gwyneth ~]# pvcreate /dev/hdc1
Physical volume “/dev/hdc1″ successfully created

Obviously, if you currently aren’t or don’t plan on ever run any software RAID setup (which you shouldn’t on a server in my opinion), you do best in removing the dmraid package altogether to avoid the same or similar problems in the future.

When dealing with replicated setups with two or more slaves sharing a master, it appears as if a lot of people overlook the obvious. You don’t need to take your master down to resync a slave. I was hoping I wouldn’t need to post about this, but I see people taking down their masters when they have perfectly healthy slaves way too often to let it slip.

You’ve got everything you need on the other slave(s). Provided that it’s in good health, you’ve got all the data, the master’s binlog file name and position. Run SHOW SLAVE STATUS\G on the slave, take note of Relay_Master_Log_File and Exec_Master_Log_Pos which are the same as what you’d get from SHOW MASTER STATUS\G on the master instance, minus the lag which is irrelevant in this case. Then proceed to sync the data from the healthy slave and use the above values in the CHANGE MASTER TO statement (obviously setting MASTER_HOST to the real master, not the other slave).

Happy higher availability!

When your app outgrows its single DB, the next logical step is attaching a slave to your current DB server to spread the read load. To do this efficiently, you will need a load balancer. If your app and growth is somewhat normal, you will at this point already have at least two front-end servers. Chances are that these are load balanced as well. So when it’s time to load balance your DB servers, you’ll already have the means to do this at hand.

In the highly likely - yet unfortunate - event that you have a load balancer without MySQL capabilities, you can always set up a generic TCP cluster (all traffic on port 3306 goes there and there). The down side of this is that there is no L7 checks - something you really want when load balancing your backend. The best you can do is L4 (is there anything accepting connections on this port?) Needless to say, there are a lot of problems that can impact your application which doesn’t make MySQL stop listen on its port. Table corruption, accidentally dropped tables, permission issues, max_connection hit, privilege problems etc.

A customer of ours was in this particular situation. They had a very decent hardware load balancer for their webservers with capacity to spare. So they ended up load balancing the mysql instances through the same device and using a piece of software I’ve written called mylbhelper.

I won’t go into exactly what mylbhelper does as the project page explains that well enough (I hope). But in a nutshell, it runs as a daemon and periodically runs a custom query on the local DB server, if it fails in any way shape or form, it executes a custom script. The script which comes with mylbhelper blocks L4 access (ie. firewalls port 3306) so that the load balancer stops sending traffic to it. Of course you can write your own scripts. Once mylbhelper has successfully executed the predefined query (twice, to avoid flapping), another script runs. Obviously, the shipped script simply removes the firewall rule put in place.

And oh it’s written in C, so you’ll need libmysql in order to compile and run it. It runs on any posix compliant system and is released under the BSD license.