All things sysadmin

Skip to the end for a TL;DR

I’ve neglected this blog a bit in the last year or so. I’ve written a lot of documentation and given a lot of training internally at work, so there hasn’t been an enormous amount of time I’ve been able or willing to spend on it. However;

Five years ago, I wrote an article presenting a script which lets you find what processes have pages in your swap memory, and how much they consume. This article is still by far the most popular one I’ve ever written, and it still sees a fair amount of traffic, so I wanted to write a bit of an updated version, and fill in some of the things I probably should have mentioned more in depth in the original one.

Let’s get one thing out of the way first - the script in the original article is now redundant. It actually already was redundant in certain cases when I wrote about it, but now it’s really redeundant. To get the same information in any currently supported distribution, simply launch top, press f, scroll down to where it says SWAP press space followed by q. There we go, script redundant.

With that cleared up - what I didn’t mention five years ago is why you would want to know this. The short answer is; you probably don’t! At least not for the reasons most people seem to want to.

When it comes to swap, the Linux virtual memory manager doesn’t really deal with ‘programs’. It only deals with pages of memory (commonly 4096 bytes, see getconf PAGE_SIZE to find out what your system uses).

Instead, when memory is under pressure and the kernel need to decide what pages to commit to swap, it will do so according to an LRU (Least Recently Used) algorithm. Well, that’s a gross oversimplification. There is a lot of magic going on there which I won’t pretend to know in any greater detail. There is also mlock() and madvise() which developers can use to influence these things.

But in essence - the VMM will amongst other things deliberately put pages of memory which are infrequently used to swap, to ensure that frequently accessed memory pages are kept in the much faster, residential memory.

So if you landed on my original article, wanting to find out what program was misbehaving by seeing what was in swap, like many appear to have done, please reconsider your troubleshooting strategy! Besides, a memory page being in swap isn’t a problem in and of itself - it’s only when those pages are shifted from RAM onto disk and back that you experience a slowdown. Pages that were swapped out once, and remain there for the lifetime of the process it belongs to doesn’t contribute to any noticeable issues.

It’s also worth noting, that once a page has been swapped out to disk, it will not be swapped back into RAM again until it need accessing by the process. Therefore, pages being in swap may be an indicator of a previous memory pressure issue, rather than one currently in progress.

So how do you know if you’re “swapping” ?

Most of the time you can just tell, you’ll notice. But for more modest swapping; the command sar -B 1 20 will print out some statistics each second for 20 seconds. If you observe the second and thrid column, you will see how many pages you swap in and out respectively. If this number is 0 or near 0, you’re unlikely to notice any issues due to swapping. Not everyone has sar installed - so another command you can run is vmstat 1 20 and look at the si and so columns for Swap In and Swap Out.

So in summary;

• top can show you how much swap a process is using
• A process using swap isn’t necessarily (or even is rarely) a badly behaved process
• Swap isn’t inherently bad, it’s only bad when it’s used frequently
• The presence of pages in swap doesn’t necessarily indicate a current memory resource issue