SSD caching using Linux
von Felix Mößbauer
BTRFS + bcache + RAID5
Windows users have always been used to distribute their folders over the various disks, but as a linux user I just want to store everything nice and decent in my home folder. The only problem therewith is that the SSD does not provide enough (cheap) space to store all my data on the fast memory.
To meet this limitation, I mounted my large HDDs into folders in /home/felix/. Hence I had the same problem as the windows users described above. Furthermore this solution is not suitable for a multi-user environment.
Wouldn't it be nice if I can combine the advantages of both SSD and HDD, and I don't mean just to buy a consumer grade SSHD? My idea is to create a RAID5 with 3 HDDs and cache them all with bcache.
The target audience of this blog posts are sophisticated linux users who are familiar with the command line. Hence, all commands are posted without sudo to avoid users overwriting their existing filesystems by just copying the lines.
bcache is a block device cache, backed by a fast SSD and is optimized to detect sequential accesses. Hence, I expect to get the high IOPS rates of a SSD combined with a high throughput when reading large files, as this transfer should entirely bypass the cache. In addition, the relatively small cache is not trashed by large file transfers. The cache can be used in a read and a read/write mode, where the latter is disabled by default. In my opinion the only disadvantage of bcache is that in cannot be installed in-place.
As the mean time between failure decreases massively with each disk added to a pool, I use a RAID5 to keep my data save. Using that, you need at least 3 HDDs and a single HDD can break without data loss. A nice side effect is that the throughput to the disk array is higher compared to a single disk, as the data is written to all disks in parallel.
Be aware: RAID is no alternative for backups!!!
- easily extendable
- SSD cachabel
Environment & Hardware
- 3 x 3TB HDD (7200rpm)
- 128GB SSD with OS and cache partition
- Ubuntu 15.10
If you want to try this setup using a different OS, check if your kernel version is at least 3.19. Otherwise you might end up with a corrupted filesystem due to a bug.
But moment: Why not use ZFS with an SSD L2ARC and ZIL cache? At first, ZFS is until version 16.04 not native on Ubuntu. Furthermore, the memory demands of the filesystem are huge. Another aspect is the missing capability to change settings like the RAID mode "on the fly".
/dev/sda1 ------------------- EXT4 / /dev/sda2 --------- | /dev/sdb -- /dev/bcache0 -| /dev/sdc -- /dev/bcache1 -|- BTRFS /home /dev/sdd -- /dev/bcache2 -|
Be careful, these commands reflect the setup above. Your disk lables will virtually certain differ from mine. Before doing any changes to the filesystem, make a backup to a disk and remove it from the system.
# create cache make-bcache -C /dev/sda2 make-bcache -B /dev/sdb /dev/sdc /dev/sdd #attache cache, determine cache cset.uuid bcache-super-show /dev/sda2 | grep cset.uuid echo <csetuuid> > /sys/block/bcache0/bcache/attach echo <csetuuid> > /sys/block/bcache1/bcache/attach echo <csetuuid> > /sys/block/bcache2/bcache/attach #create btrfs mkfs.btrfs /dev/bcache0 /dev/bcache1 /dev/bcache2 -d raid5 -m raid5 #mount filesystem to copy home folder mount /dev/bcache0 /mnt #if you trust in your SSD, enable write caching echo 'writeback' /sys/block/bcache*/bcache/cache_mode
I will definetly provide some IO benchmarks, but currently I do not have the time to specify and run these :( However the subjective feeling is a way higher performance. The time to the fully loaded desktop decreased from around 1 minute to approx. 15 seconds.
- Nice article about bcache (in German): https://www.thomas-krenn.com/de/wiki/Bcache
- Bcache on ArchLinux: https://wiki.archlinux.org/index.php/Bcache
- SSD caching using ZFS: https://blogs.oracle.com/brendan/entry/test
- Blogpost on a even more sophisticated setup: https://www.rath.org/ssd-caching-under-linux.html