How does MySQL hide the command line password in ps?

I saw this question asked today, and thought I’d write a quick post about it.
Giving passwords on the command line isn’t necessarily a fantastic idea – but you can sort of see where they’re coming from. Configuration files and environment variables are better, but just slightly. Security is a night mare!

But if you do decide to write an application which takes a password (or any other sensitive information) on the command line, you can prevent other users on the system from easily seeing it like this:

#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <sys/types.h>

int main(int argc, char *argv[]){

int i = 0;
pid_t mypid = getpid();
if (argc == 1)
return 1;
printf("argc = %d and arguments are:\n", argc);
for (i ; i < argc ; i++)
printf("%d = %s\n" ,i, argv[i]);
printf("Replacing first argument with x:es... Now open another terminal and run: ps p %d\n", (int)mypid);
fflush(stdout);
memset(argv[1], 'x', strlen(argv[1]));
getc(stdin);
return 0;
}

A sample run looks like this:
$ ./pwhide abcd
argc = 2 and arguments are:
0 = ./pwhide
1 = abcd
Replacing first argument with x:es... Now run: ps p 27913

<In another terminal>
$ ps p 27913
PID TTY STAT TIME COMMAND
27913 pts/1 S+ 0:00 ./pwhide xxxx

In the interest of brevity, the above code isn't very portable - but it works on Linux and hopefully the point of it comes across. In other environments, such as FreeBSD, you have the setproctitle() syscall to do the dirty work for you. The key thing here is the overwriting of argv[1]
Because the size of argv[] is allocated when the program starts, you can't easily obfuscate the length of the password. I say easily - because of course there is a way.

Posted in misc, MySQL | Comments Off

Font rendering – no more jealousy

I suppose this kind of content is what most people use twitter for these days. But since I’ve remained strong and stayed well away from that, I suppose I will have to be a tad retro and write a short blog post about it.
If you like me are an avid Fedora user, I’m sure you’ve thrown glances at colleague’s or friend’s Ubuntu machines and thought that there was something that was slightly different about the way it looked (aside from the obvious Gnome vs Unity differences). Shinier somehow… So had I, but I mainly dismissed it as a case of “the grass is always greener…”.

It turns out that the grass actually IS greener. Tonight I stumbled upon this. It’s a patched version of freetype. For what I assume are political reasons (free as in speech), Fedora ships a Freetype version without subpixel rendering. These patches fixes that and other things.

With a default configuration file of 407 lines, it’s quite extensible and configurable as well. Lucky, I quite like the default!

If you’re not entirely happy with the way your fonts look on Fedora – it’s well worth a look

Posted in misc, Sundry sysadmin | Comments Off

Transactions and code testing

A little while ago I worked with a customer to migrate their DB from using MyISAM to InnoDB (something I definitely don’t mind doing!)
I set up a smaller test instance with all tables using the InnoDB engine as part of the testing. I instructed them to thoroughly test their application against this test instance and let me know if they identified any issues.

They reported back that everything seemed fine, and we went off to do the actual migration. Everything went according to plan and things seemed well.
After a while they started seeing some discrepancies in the stock portion of their application. The data didn’t add up with what they expected and stock levels seemed surprisingly high. A crontabbed program was responsible for periodically updating the stock count of products, so this was of course the first place I looked.
I ran it manually and looked at its output; it was very verbose and reported some 2000 products had been updated. But looking at the actual DB, this was far from the case.

Still having the test environment available, I ran it a few times against that and could see the com_update and com_insert counters being incremented, so I knew the queries were making it there. But the data remained intact. At this point, I had a gut feeling what was going on.. so to confirm this, I enabled query logging to see what was actually going on. It didn’t take me long to spot the problem. On the second line of the log, I saw this:

40 Query set autocommit=0

The program responsible for updating the stock levels was a python script using MySQLDB. I couldn’t see any traces of autocommit being set explicitly, so I went on assuming that it was off by default (which turned out to be correct). After adding cursor.commit()* after the relevant queries had been sent to the server, everything was back to normal as far as stock levels were concerned.
Since the code itself was seeing its own transaction, calls such as cursor.rowcount which the testers had relied on were all correct.

But the lesson here; when testing your software from a database point of view, don’t blindly trust what your code tells you it’s done, make sure it’s actually done it by verifying the data!
A lot of things can happen to data between your program and the platters. Its transaction can deadlock and be rolled back, it can be reading cached data, it can get lost in a crashing message queue, etc.

* As a rule of thumb, I’m rather against setting a blanket autocommit=1 in code, I’ve seen that come back to haunt developers in the past. I’m a strong advocate for explicit transaction handling.

Posted in MySQL | Comments Off

Find out what is using your swap

Have you ever logged in to a server, ran `free`, seen that a bit of swap is used and wondered what’s in there? It’s usually not very indicative of anything, or even overly helpful knowing what’s in there, mostly it’s a curiosity thing.

Either way, starting from kernel 2.6.16, we can find out using smaps which can be found in the proc filesystem. I’ve written a simple bash script which prints out all running processes and their swap usage.
It’s quick and dirty, but does the job and can easily be modified to work on any info exposed in /proc/$PID/smaps
If I find the time and inspiration, I might tidy it up and extend it a bit to cover some more alternatives. The output is in kilobytes.

#!/bin/bash
# Get current swap usage for all running processes
# Erik Ljungstrom 27/05/2011
SUM=0
OVERALL=0
for DIR in `find /proc/ -maxdepth 1 -type d | egrep "^/proc/[0-9]"` ; do
PID=`echo $DIR | cut -d / -f 3`
PROGNAME=`ps -p $PID -o comm --no-headers`
for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'`
do
let SUM=$SUM+$SWAP
done
echo "PID=$PID - Swap used: $SUM - ($PROGNAME )"
let OVERALL=$OVERALL+$SUM
SUM=0

done
echo "Overall swap used: $OVERALL"

This will need to be ran as root for it to be able to gather accurate numbers. It will still work even if you don’t, but it will report 0 for any processes not owned by your user.
Needless to say, it’s Linux only. The output is ordered alphabetically according to your locale (which admittedly isn’t a great thing since we’re dealing with numbers), but you can easily apply your standard shell magic to the output. For instance, to find the process with most swap used, just run the script like so:

$ ./getswap.sh | sort -n -k 5
Don’t want to see stuff that’s not using swap at all?
$ ./getswap.sh | egrep -v "Swap used: 0" |sort -n -k 5

… and so on and so forth

Posted in Sundry sysadmin | 40 Comments

Example using Cassandra with Thrift in C++

Due to a very exciting, recently launched project at work, I’ve had to interface with Cassandra through C++ code. As anyone who has done this can testify, the API docs are vague at best, and there are very few examples out there. The constant API changes between 0.x versions and the fact that the Cassandra API has its docs and Thrift has its own, but there is nothing bridging the two isn’t helpful either.
So at the moment it is very much a case of dissecting header files and looking at implementation in the Thrift generated source files.

The only somewhat useful example of using Cassandra with C++ one can find online is this, but due to the API changes, this is now outdated (it’s still worth a read).

So in the hope that nobody else will have to spend the better part of a day piecing things together to achieve even the most basic thing, here’s an example which works with Cassandra 0.7 and Thrift 0.6.

First of all, create a new keyspace and a column family, using cassandra-cli:

[default@unknown] create keyspace nm_example;
c647b2c0-83e2-11e0-9eb2-e700f669bcfc
Waiting for schema agreement...
... schemas agree across the cluster
[default@unknown] use nm_example;
Authenticated to keyspace: nm_example
[default@nm_example] create column family nm_cfamily with comparator=BytesType and default_validation_class=BytesType;
30466721-83e3-11e0-9eb2-e700f669bcfc
Waiting for schema agreement...
... schemas agree across the cluster
[default@nm_example]

Now go to the directory where you have cassandra installed and enter the interface/directory and run: thrift –gen cpp cassandra.thrift
This will create the gen-cpp/ directory. From this directory, you need to copy all files bar the Cassandra_server.skeleton.cpp one to wherever you intend to keep your sources.
Here’s some example code which inserts, retrieves, updates, retrieves and deletes keys:

#include "Cassandra.h"

#include <protocol/TBinaryProtocol.h>
#include <thrift/transport/TSocket.h>
#include <thrift/transport/TTransportUtils.h>

using namespace std;
using namespace apache::thrift;
using namespace apache::thrift::protocol;
using namespace apache::thrift::transport;
using namespace org::apache::cassandra;
using namespace boost;

static string host("127.0.0.1");
static int port= 9160;

int64_t getTS(){
/* If you're doing things quickly, you may want to make use of tv_usec
* or something here instead
*/
time_t ltime;
ltime=time(NULL);
return (int64_t)ltime;

}

int main(){
shared_ptr socket(new TSocket(host, port));
shared_ptr transport(new TFramedTransport(socket));
shared_ptr protocol(new TBinaryProtocol(transport));
CassandraClient client(protocol);

const string& key="your_key";

ColumnPath cpath;
ColumnParent cp;

ColumnOrSuperColumn csc;
Column c;

c.name.assign("column_name");
c.value.assign("Data for our key to go into column_name");
c.timestamp = getTS();
c.ttl = 300;

cp.column_family.assign("nm_cfamily");
cp.super_column.assign("");

cpath.column_family.assign("nm_cfamily");
/* This is required - thrift 'feature' */
cpath.__isset.column = true;
cpath.column="column_name";

try {
transport->open();
cout << "Set keyspace to 'dpdns'.." << endl;
client.set_keyspace("nm_example");

cout << "Insert key '" << key << "' in column '" << c.name << "' in column family '" << cp.column_family << "' with timestamp " << c.timestamp << "..." << endl;
client.insert(key, cp, c, org::apache::cassandra::ConsistencyLevel::ONE);

cout << "Retrieve key '" << key << "' from column '" << cpath.column << "' in column family '" << cpath.column_family << "' again..." << endl;
client.get(csc, key, cpath, org::apache::cassandra::ConsistencyLevel::ONE);
cout << "Value read is '" << csc.column.value << "'..." << endl;

c.timestamp++;
c.value.assign("Updated data going into column_name");
cout << "Update key '" << key << "' in column with timestamp " << c.timestamp << "..." << endl;
client.insert(key, cp, c, org::apache::cassandra::ConsistencyLevel::ONE);

cout << "Retrieve updated key '" << key << "' from column '" << cpath.column << "' in column family '" << cpath.column_family << "' again..." << endl;
client.get(csc, key, cpath, org::apache::cassandra::ConsistencyLevel::ONE);
cout << "Updated value is: '" << csc.column.value << "'" << endl;

cout << "Remove the key '" << key << "' we just retrieved. Value '" << csc.column.value << "' timestamp " << csc.column.timestamp << " ..." << endl;
client.remove(key, cpath, csc.column.timestamp, org::apache::cassandra::ConsistencyLevel::ONE);

transport->close();
}
catch (NotFoundException &nf){
cerr << "NotFoundException ERROR: "<< nf.what() << endl;
}
catch (InvalidRequestException &re) {
cerr << "InvalidRequest ERROR: " << re.why << endl;
}
catch (TException &tx) {
cerr << "TException ERROR: " << tx.what() << endl;
}

return 0;
}

Say we've called the file cassandra_example.cpp, and you have the files mentioned above in the same directory, you can comile things like this:

$ g++ -lthrift -Wall cassandra_example.cpp cassandra_constants.cpp Cassandra.cpp cassandra_types.cpp -o cassandra_example
$ ./cassandra_example
Set keyspace to 'nm_example'..
Insert key 'your_key' in column 'column_name' in column family 'nm_cfamily' with timestamp 1306008338...
Retrieve key 'your_key' from column 'column_name' in column family 'nm_cfamily' again...
Value read is 'Data for our key to go into column_name'...
Update key 'your_key' in column with timestamp 1306008339...
Retrieve updated key 'your_key' from column 'column_name' in column family 'nm_cfamily' again...
Updated value is: 'Updated data going into column_name'
Remove the key 'your_key' we just retrieved. Value 'Updated data going into column_name' timestamp 1306008339 ...

As my WP template isn't very suitable for code, I've put up the cpp file here for download
Another thing worth mentioning is Padraig O'Sullivan's libcassandra, which may or may not be worth a look depending on what you want to do and what versions of Thrift and Cassandra you're tied to.

Posted in misc | 4 Comments