LVM thinpool for docker storage on Fedora 22

TL;DR: You can use docker-storage-setup without root fs being on LVM by passing DEVS and VG environment variables to the script or editing /etc/sysconfig/docker-storage-setup

I stumbled across this article the other day ‘Friends Don’t Let Friends Run Docker on Loopback in Production’

I also saw this bug being raised, saying docker-storage-setup doesn’t work with the Fedora 22 cloud image, as the root fs isn’t on LVM.

I decided to try this out, so I created some block storage and a Fedora 22 VM on the Rackspace cloud:

$ cinder create --display-name docker-storage --volume-type 1fd376b5-c84e-43c5-a66b-d895cb75ac2c 75
# Verify that it's built and is available
$ cinder show 359b01b7-541c-4f4d-b2e7-279d778079a4
# Build a Fedora 22 server with the volume attached
nova boot --image 2cc5db1b-2fc8-42ae-8afb-d30c68037f02 \
--flavor performance1-1 \
--block-device-mapping xvdb=359b01b7-541c-4f4d-b2e7-279d778079a4 \
docker-storage-test

Once on the machine, I followed the article above:

$ dnf -y install docker
$ systemctl stop docker
$ rm -rf /var/lib/docker/

And here’s where the bug report I linked earlier comes into play. docker-storage-setup is just a bash script, and if you just take a look at this output:

docker-storage-setup --help
Usage: /usr/bin/docker-storage-setup [OPTIONS]

Grows the root filesystem and sets up storage for docker.

Options:
  -h, --help            Print help message.

It sure gives the impresson of only doing one single thing - growing the root FS! As the bug rightly points out, the Fedora cloud image doesn’t come with LVM for the root FS (which is a good thing!), so there’s no VG for this script to grow.

So unless you read the script, or the manpage, you wouldn’t necessarily notice that what --help says is just the default behaviour, and you can use docker-storage-setup to use an emphemeral disk and leave the root fs alone. The kicker lies in two environment variables (as opposed to arguments to the script itself, like is more common); $DEVS and $VG. If you supply both of those, and the disk you give in DEVS has no partition table and the VG you supply doesn’t exist, the script will partition the disk and create all the necessary bits for LVM on that disk:

# Verify that ephemeral disk has no partition table:
$ partx -s /dev/xvdb
partx: /dev/xvdb: failed to read partition table

# Start lvmetad
$ systemctl start lvm2-lvmetad

$ DEVS="/dev/xvdb" VG="docker-data" docker-storage-setup
  Volume group "xvda1" not found
  Cannot process volume group xvda1
Checking that no-one is using this disk right now ... OK

Disk /dev/xvdb: 75 GiB, 80530636800 bytes, 157286400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Script header accepted.
>>> Created a new DOS disklabel with disk identifier 0x2b7ebb69.
Created a new partition 1 of type 'Linux LVM' and of size 75 GiB.
/dev/xvdb2:
New situation:

Device     Boot Start       End   Sectors Size Id Type
/dev/xvdb1       2048 157286399 157284352  75G 8e Linux LVM

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
  Physical volume "/dev/xvdb1" successfully created
  Volume group "docker-data" successfully created
  Rounding up size to full physical extent 80.00 MiB
  Logical volume "docker-poolmeta" created.
  Logical volume "docker-pool" created.
  WARNING: Converting logical volume docker-data/docker-pool and docker-data/docker-poolmeta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  Converted docker-data/docker-pool to thin pool.
  Logical volume "docker-pool" changed.

# Verify that the script wrote the docker-storage file
$ cat /etc/sysconfig/docker-storage
DOCKER_STORAGE_OPTIONS=--storage-driver devicemapper --storage-opt dm.fs=xfs
--storage-opt dm.thinpooldev=/dev/mapper/docker--data-docker--pool
--storage-opt dm.use_deferred_removal=true

# Verify that the LV is there:
$ lvs
  LV          VG          Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool docker-data twi-a-t--- 44.95g             0.00   0.07

So now the script has created the LV thinpool, and written the required docker configuration.

$ systemctl start docker
$ docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker--data-docker--pool
 Pool Blocksize: 524.3 kB
 Backing Filesystem: extfs
 Data file:
 Metadata file:
 Data Space Used: 19.92 MB
 Data Space Total: 48.26 GB
 Data Space Available: 48.24 GB
 Metadata Space Used: 65.54 kB
 Metadata Space Total: 83.89 MB
 Metadata Space Available: 83.82 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Library Version: 1.02.93 (2015-01-30)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.8-300.fc22.x86_64
Operating System: Fedora 22 (Twenty Two)
CPUs: 1
Total Memory: 987.8 MiB
Name: docker-storage-test
ID: EYKV:Q5D6:4F3Y:Z5X3:ZILX:ZBVI:2YF6:VHD7:RFQS:IWWO:MOFL:EWO7

No trace of /dev/loop0! And to verify that it’s actually using our thinpool:

$ lvdisplay | egrep "Allocated pool data" ; du -sh /var/lib/docker/ ; docker pull centos:6 ; du -sh /var/lib/docker ; lvdisplay | egrep "Allocated pool data"
  Allocated pool data    0.04%
5.6M    total
6: Pulling from docker.io/centos
47d44cb6f252: Pull complete
6a7b54515901: Pull complete
e788880c8cfa: Pull complete
1debf8fb53e6: Pull complete
72703a0520b7: Already exists
docker.io/centos:6: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:5436a8b20d6cdf638d936ce1486e277294f6a1360a7b630b9ef76b30d9a88aec
Status: Downloaded newer image for docker.io/centos:6
5.8M    total
  Allocated pool data    0.53%

In conclusion - the script could definitely do with being updated to using command line arguments for this, rather than environment variables, and update the –help output to highlight this.

All things sysadmin

LVM Thinpool for Docker Storage on Fedora 22