They reported back that everything seemed fine, and we went off to do the actual migration. Everything went according to plan and things seemed well.
After a while they started seeing some discrepancies in the stock portion of their application. The data didn’t add up with what they expected and stock levels seemed surprisingly high. A crontabbed program was responsible for periodically updating the stock count of products, so this was of course the first place I looked.
I ran it manually and looked at its output; it was very verbose and reported some 2000 products had been updated. But looking at the actual DB, this was far from the case.

Still having the test environment available, I ran it a few times against that and could see the com_update and com_insert counters being incremented, so I knew the queries were making it there. But the data remained intact. At this point, I had a gut feeling what was going on.. so to confirm this, I enabled query logging to see what was actually going on. It didn’t take me long to spot the problem. On the second line of the log, I saw this:

40 Query set autocommit=0


The program responsible for updating the stock levels was a python script using MySQLDB. I couldn’t see any traces of autocommit being set explicitly, so I went on assuming that it was off by default (which turned out to be correct). After adding cursor.commit()* after the relevant queries had been sent to the server, everything was back to normal as far as stock levels were concerned.
Since the code itself was seeing its own transaction, calls such as cursor.rowcount which the testers had relied on were all correct.

But the lesson here; when testing your software from a database point of view, don’t blindly trust what your code tells you it’s done, make sure it’s actually done it by verifying the data!
A lot of things can happen to data between your program and the platters. Its transaction can deadlock and be rolled back, it can be reading cached data, it can get lost in a crashing message queue, etc.

* As a rule of thumb, I’m rather against setting a blanket autocommit=1 in code, I’ve seen that come back to haunt developers in the past. I’m a strong advocate for explicit transaction handling.

So far so good! A week or so later, we often get the call “Our page load time is higher now than before the upgrade! We’ve got twice as much hardware, and it’s slower! You have broken it!”
It’s easy to see where they’re coming from. It makes sense, right?

That is until you factor in the newly introduced network topology! Today it’s not unusual (that’s not to say it’s acceptable or optimal) for your average
wordpress/drupal/joomla/otherspawnofsatan site to run 40-50 queries per page load. Quite often even more!

Based on a tcpdump session of a reasonably average query (if there is such a thing), connecting to a server, authenticating, sending a query and receiving a 5 row result set of 1434 bytes yields 25 packets being sent between my laptop and a remote DB server on the same wired, non-congested network. A normal, average latency of TCP/IP over Ethernet is ~0.2 ms for the size of packets we’re talking here.
So, doing the maths, you’re seeing 25*0.2*50= 250ms in just network latency per page load for your SQL queries. This is obviously a lot more than you see over a local UNIX socket.

This is inevitable, laws of physics. It is nothing you, your sysadmin and/or your hosting company can do anything about. There may however be something your developer can do about the amount of queries!
You also shouldn’t confuse response-times with availability. Your response times may be slower, but you can (hopefully) serve a lot more users with this setup!

Sure, there are technologies out there which have considerably less latency than ethernet, but they come with quite the price-tag, and there are more often than not quite a few avenues to go down before it makes sense to start looking at that kind of thing.

You could also potentially looking at running the full stack on both machines using master/master replication for your DBs, and load balance your front-ends and have them both read locally, but only write to one node at a time! That kind of DB scenario is something fairly easily set up using mmm for MySQL. But in my experience, this often ends up more costly and potentially introducing more complexities than it solves.
I’m an avid advocate for keeping server roles separate as much as possible!

This mode of replication is called semi synchronous due to the fact that it only guarantees that at least one of the slaves have written the transaction to disk in its relay log, not actually committed it to its data files. It guarantees that the data exists by some means somewhere, but not that it’s retrievable through a MySQL client.

Semi sync is available as a plugin, and if you compile from source, you’ll need to do –with-plugins=semisync….
So far, the semisync plugin can only be built as a dynamic module, so you’ll need to install it once you’ve got your instance up and running. To do this, you do as with any other plugin:
install plugin rpl_semi_sync_master soname 'semisync_master.so';
install plugin rpl_semi_sync_slave soname 'semisync_slave.so';

You might get an 1126 error and a message saying “Can’t open shared library..”, then you most likely need to set the plugin_dir variable in my.cnf and give MySQL a restart.
If you’re using a master/slave pair, you obviously won’t need to load both modules as above. You load the slave one on your slave, and the master one on your master. Once you’ve done this, you’ll have entries for these modules in the mysql.plugin table.
When you have confirmed that you do, you can safely add the pertinent variables to your my.cnf, the values I used (in addition to the normal replication settings) for my master/master sandboxes were:
plugin_dir=/opt/mysql-5.5.6-rc/lib/mysql/plugin/
rpl_semi_sync_master_enabled=1
rpl_semi_sync_master_timeout=10000
rpl_semi_sync_slave_enabled=1
rpl_semi_sync_master_trace_level=64
rpl_semi_sync_slave_trace_level=64
rpl_semi_sync_master_wait_no_slave=1

Note that you probably won’t want to use these values for _trace_level in production due to the verbosity in the log! I just enabled these while testing.
Also note that the timeout is in milliseconds.
You can also set these on the fly with SET GLOBAL (thanks Oracle!), just make sure the slave is stopped before doing this, as it needs to be enabled during the handshake with the master for the semisync to kick in.

The timeout is the amount of time the master will lock and wait for a slave to acknowledge the write before giving up on the whole idea of semi synchronous operation and continue as normal.
If you want to monitor this, you can use the status variable Rpl_semi_sync_master_status which is set to Off when this happens.
If this condition should be avoided altogether, you would need to set a large enough value for the timeout and a low enough monitoring threshold as there doesn’t seem to be a way to force MySQL to wait forever for a slave to appear.

If you’re running an automated failover setup, you’ll want to set the timeout higher than your heartbeat, so ensuring no committed data is lost. Then you might also want to set the timeout considerably lower initially on the passive master so that you don’t end up waiting on the master we know is unhealthy and have just failed over from.

Before implementing this in production, I would strongly recommend running a few performance tests against your setup as this will slow things down considerably for some workloads. Each transaction has to be written to the binlog, read over the wire and written to the relay log, and then lastly flushed to disk before each DML statement returns. You will almost definitely benefit in batching up queries into larger transactions rather than using the default auto commit mode as this will increase the frequency of the steps.
Update: Even though the manual clearly states that the event has to be flushed to disk, this doesn’t actually appear to be the case (see comments). The above still stands, but the impact may not be as great as first thought

When I find the time, I will run some benchmarks on this.

Lastly, please note that this is written while MySQL 5.5 is still in release candidate stage, so while unlikely, things are subject to change. So please be mindful of this in future comments.

They have changed their minds quite a bit. Virtualisation in production is no longer a solid no-no according to them (a lot of people would argue). Solaris containers, anyone?

As most of us know by now, MySQL struggles to utilise multiple cores efficiently. This has been the case for quite some time by now, and people like Google and Percona has grown tired of waiting for MySQL to fix it.

Sun decided to not go down the route of reviewing and accepting the patches, but are now suggesting – are you sitting down? – running multiple instances on the same hardware.
I’m not against this from a technical point of view as it currently actually does improve performance on multiple-core-multiple-disk systems (for an unpatched version) for some workloads, but the fact that they have gone to openly and officially suggest workarounds to their own problem rather than fixing the source of the problem is disturbing.

Granted, I suppose it makes sense to suggest larger boxes if you’ve been bought by a big-iron manufacturer. Also, I should be fair and note that Ivan at least didn’t say scaling out was a negative thing and that it’s still a good option.

If anyone asks me though, I think I’ll keep scaling outwards and use the more sensible version of MySQL

You’ve got everything you need on the other slave(s). Provided that it’s in good health, you’ve got all the data, the master’s binlog file name and position. Run SHOW SLAVE STATUS\G on the slave, take note of Relay_Master_Log_File and Exec_Master_Log_Pos which are the same as what you’d get from SHOW MASTER STATUS\G on the master instance, minus the lag which is irrelevant in this case. Then proceed to sync the data from the healthy slave and use the above values in the CHANGE MASTER TO statement (obviously setting MASTER_HOST to the real master, not the other slave).

Happy higher availability!

In the highly likely – yet unfortunate – event that you have a load balancer without MySQL capabilities, you can always set up a generic TCP cluster (all traffic on port 3306 goes there and there). The down side of this is that there is no L7 checks – something you really want when load balancing your backend. The best you can do is L4 (is there anything accepting connections on this port?) Needless to say, there are a lot of problems that can impact your application which doesn’t make MySQL stop listen on its port. Table corruption, accidentally dropped tables, permission issues, max_connection hit, privilege problems etc.

A customer of ours was in this particular situation. They had a very decent hardware load balancer for their webservers with capacity to spare. So they ended up load balancing the mysql instances through the same device and using a piece of software I’ve written called mylbhelper.

I won’t go into exactly what mylbhelper does as the project page explains that well enough (I hope). But in a nutshell, it runs as a daemon and periodically runs a custom query on the local DB server, if it fails in any way shape or form, it executes a custom script. The script which comes with mylbhelper blocks L4 access (ie. firewalls port 3306) so that the load balancer stops sending traffic to it. Of course you can write your own scripts. Once mylbhelper has successfully executed the predefined query (twice, to avoid flapping), another script runs. Obviously, the shipped script simply removes the firewall rule put in place.

And oh it’s written in C, so you’ll need libmysql in order to compile and run it. It runs on any posix compliant system and is released under the BSD license.

With querysniffer you get a real time overview of all queries which are running. It’s a simple perlscript and is easy enough to get going with. On RedHat/CentOS, you’d go about it like this:

wget http://iank.org/querysniffer/mysqlsniff-0.10.pl.txt -O mysqlsniff.pl
yum install libpcap-devel
cpan -i Net::PcapUtils
cpan -i NetPacket::Ethernet
perl mysqlsniff.pl eth0

And now you should see any query sent to the server over eth0 on your terminal.

Being at the peak of the “scalability hype”, where everyone pretends to be a mini-livejournal or flickr , the vast amount of companies of a more humble size out there is often forgotten about. The type of companies that easily get away with a decent master and a slave or two hanging off of it, and most likely will for a very long time to come.

So, why is it flawed? My major beef with it is that it’s not self-healing. Sure, you can monitor, script and re-jig things to a certain extent. But this is why I was thrilled when read the MySQL Forge suggestions for Google Summer of Code. One of the suggestions is to enable self-healing replication using components, or at least concepts, from maatkit and Google’s MMRM.  

Interesting! While I can’t see it become completely fool proof, I’m sure it would help in the majority of scenarios I’ve seen where replication has broken.

As the linked forum post says, it is a bit of a shame that the tools to make this has been conceived by people outside of MySQL when it really should have been a part of the server’s implementation a long time ago!

Let’s just hope someone talented with some spare time steps up to the challenge!

Basically it’s a multithreaded client. Most servers these days have, or certainly should have, multiple disks and multiple CPU cores and reasonably fast storage. So using a single threaded client to insert those hundreds of thousands or millions of records doesn’t make that much sense today. There’s quite a lot of time to save. I’ve ran some tests on MySQL 5.1.24 and compared the results with those from the stock mysql client:

1 million rows, 15k SAS drives in RAID 10, 4 cores @ 2.33GHz, 6G of ram, 4 threads for paramy. Table is int, char(8), char(8), InnoDB plugin 1.0.1 (given enough time, I’ll compare these to MySQL 5.0 and 4.0 as well at a later stage):

Extended insert format:
paramy: 58786.74 inserts/s – 0m4.977s
stock: 25898.35 inserts/s – 0m9.588s
————————————-
51% gain

Single row format:
paramy: 24329.54 inserts/s – 0m25.221s
stock: 16642.02 inserts/s – 0m52.753s
————————————–
47% gain

So, effectively cutting the insert time in half. It can’t handle auto incremented fields very well yet, but domas’ working code (not publicly available yet as far as I know) handles these with 1.3 core speed.

Be careful not to over-do it though, as more threads doesn’t necessarily equal more throughput. Take CPU cores and disks into account when setting the amount. I ran paramy with a few different number of threads. As stated above I ran these test on a 4 core system:

n threads, single row format:
2 threads: 0m39.700s
4 threads: 0m25.221s
6 threads: 0m31.889s
8 threads: 0m32.761s
 

I tested paramy on an RHEL4 system, and experienced an issue to begin with. After reporting this, it was concluded that it was related to the LOCK TABLES, so after a swish grep -v “LOCK TABLES” paramy.sql > paramy_nolock.sql, it was good to go.

All in all – this tool is bound to save me hours and hours of time in the future! Good work indeed! 

As I am quite intrigued by the compression facility, I shortly after the compilation proceeded by trying to create a table:

mysql> CREATE TABLE compressed_4 (id int(11) primary key, txt char(8)) Engine=InnoDB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=4;
Query OK, 0 rows affected, 5 warnings (0.00 sec)

mysql> show warnings;
+---------+------+----------------------------------------------------------------+
| Level | Code | Message |
+---------+------+----------------------------------------------------------------+
| Warning | 1478 | InnoDB: KEY_BLOCK_SIZE requires innodb_file_per_table. |
| Warning | 1478 | InnoDB: KEY_BLOCK_SIZE requires innodb_file_format > Antelope. |
| Warning | 1478 | InnoDB: ignoring KEY_BLOCK_SIZE=4. |
| Warning | 1478 | InnoDB: ROW_FORMAT=COMPRESSED requires innodb_file_per_table. |
| Warning | 1478 | InnoDB: assuming ROW_FORMAT=COMPACT. |
+---------+------+----------------------------------------------------------------+
5 rows in set (0.00 sec)
Primary mistake: Not reading the docs properly. So after sticking innodb_file_per_table=1 and innodb_file_format=”Barracuda” into my.cnf and restart MySQL, the creation worked fine. I created three tables, 3 with compression and 1 without:

CREATE TABLE compressed_4 (id int(11) PRIMARY KEY, txt char(8)) Engine=InnoDB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=4;
CREATE TABLE compressed_8 (id int(11) PRIMARY KEY, txt char(8)) Engine=InnoDB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8;
CREATE TABLE compressed_1 (id int(11) PRIMARY KEY, txt char(8)) Engine=InnoDB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=1;
CREATE TABLE noncompressed_4 (id int(11) PRIMARY KEY, txt char(8)) Engine=InnoDB;

I then proceeded to insert 50000 rows and inspect the on-disk size:

# du /var/lib/mysql_i/testing/*.ibd
3076 /var/lib/mysql_i/testing/compressed_1.ibd
4100 /var/lib/mysql_i/testing/compressed_4.ibd
6156 /var/lib/mysql_i/testing/compressed_8.ibd
10256 /var/lib/mysql_i/testing/noncompressed_4.ibd

So far so good! 30% with 1K page size (referred to as KEY_BLOCK_SIZE at table creation because MySQL doesn’t allow storage engines to add their own syntax (another FIXME for our friends at MySQL? At least one should get the option in my opinion)).

But what is the performance like? Both the insert and select benchmarks are done with to a ramdisk. I want to know the impact on the CPUs and the difference between the different compression rates, not the performance of the disks:

1K insert time: 7.4322438240051 seconds.
4K insert time: 5.1487679481506 seconds.
8K insert time: 4.8629088401794 seconds.
NC insert time: 3.5483801364899 seconds. (No Compression)

Pretty much what I expected for the inserts.

Selects are another story altogether though:

1K select time: 1.8464620113373 seconds.
4K select time: 1.7520101070404 seconds.
8K select time: 1.6863808631897 seconds.
NC select time: 1.631724023819 seconds. (No Compression)

1K select time: 1.5653350353241 seconds.
4K select time: 1.8094820976257 seconds.
8K select time: 1.7305459976196 seconds.
NC select time: 1.8121519088745 seconds. (No Compression)

1K select time: 1.7385520935059 seconds.
4K select time: 1.8407368659973 seconds.
8K select time: 1.8273019790649 seconds.
NC select time: 1.7326860427856 seconds. (No Compression)

Rather random from the looks of things. Not entirely sure if that’s down to poor benchmarking tools or if I’ve overseen something in regards to how select queries are performed. Yes, the query cache was disabled and the server was rebooted between the runs. I was so perplexed that I ran the tests on two other machines, and both with and without using a ramdisk for storage, but the variation is there regardless. Does anyone have some deeper insight into how the bits and bytes of this works?

My only half-feasible theory at this point is that the reading of the data takes 2-2.5x the time it takes to decompress it. So in a scenario with reading 1K and 4K pages; while MySQL reads the additional 3K, it’s already well on the way of decompressing the 1K since it’s obviously quicker to read 1K than 4, and in the end of the day, they sum up to pretty much the same amount of time – just in different ways (CPU or storage).

So as an example – say a SELECT takes 1 second:
For the 1K page, 0.3s would be spent reading it and 0.7s would be spent decompressing it.
For the 4K page, 0.7s would be spent reading it and 0.3s would be spent decompressing it.

I suppose this could be tested with a really slow single core CPU and fast storage, or vice versa.