swapfile vs swap partition vs zram vs zswap

this originally started as my research notes when deciding on a swap configuration for my nixos configuration, prawnix (https://github.com/SolidEva/prawnix)

sorry for the weird list formatting, you can blame https://pypi.org/project/Markdown/

i last examined swap configurations when working on PrawnOS (https://github.com/SolidEva/PrawnOS). On PrawnOS, i really just picked zram without further thought because the ChromeOS devs used it for the same hardware. i wanted to give this decision a bit more thought.

at then end, i want to have reasonable default swap configurations for the following types of systems i manage and encounter:

  • desktops with powerful cpus, lots of memory (>>16GB), and persistent nvme SSD storage (>1TB)
  • laptops with powerful cpus, lots of memory (>>16GB), and persistent nvme SSD storage (>1TB)
  • laptops with weak cpus, minimal memory (~8GB or less), and persistent nvme SSD storage (~128GB)
  • SBCs (like the rpi), weak cpus, minimal memory, and SD card persistent storage
  • NAS server, moderate cpu, ~32GB memory, and persistent nvme SSD storage

What does linux use swap for

Types of swap

https://wiki.nixos.org/wiki/Swap

  • swap partition (lvm or normal partition)
    • defines a partition to use as swap space
  • swapfile
    • defines a file path on a filesystem to use as swap space
  • swap on zram (often referred to as just "zram"): https://fedoraproject.org/wiki/Changes/SwapOnZRAM
    • use a ramdisk with compression as your swap device
  • zswap: https://wiki.archlinux.org/title/Zswap
    • use a ramdisk with compression as your first layer swap, then a swapfile or swap partition as your second layer swap
    • yep, the solution to full/slow caches is always just to make another cache layer (or add more memory). welcome to memory management! its turtles all the way down

hibernation on linux

a short explanation of how hibernation works

  • hibernation is a "deep sleep" state, mainly useful for laptops
  • hibernation is more power efficient than the usual suspend state, allowing the machine to stay asleep for very long periods of time
  • to hibernate, the device dumps its ram contents to some form of persisitent storage (ssd, disk, etc) and powers off
  • on "wake up" the device does a shortened boot sequence, reads the persisted memory contents, and loads them back into memory
  • the user sees exactly what they were doing before hibernation was triggered, same as normal suspend
  • more here https://en.wikipedia.org/wiki/Hibernation_(computing)

why hibernation is good

  • its a super low power usage sleep, whats not to love! Everyone wants more battery life.
  • less time spent watching the boot process!

why hibernation is complicated

security

rootfs encryption
  • i will assume that everyone already knows why an encrypted rootfs is a good thing to have.

  • so i will also assume that everyone already has an encrypted rootfs.

  • What happens when your encryption keys get dumped to swap as part of hibernate? If your swap is unencrypted, the data one your system is vulnerable during hibernate!

    • to fix this, swap must be encrypted!
secure boot
bad habits
  • leaving your machines unattended, in a powered on state for long periods is not great security practice. an encrypted system is most secure when powered off

hardware issues

Limited swap choices

  • if you use zram without a disk-based swap device, you cannot use hibernation since zram lives in ram. you now need two swap devices.
    • of course, you can use zswap instead

non trivial to setup

  • most distros don't provide a configuration with hibernation enabled by default, it is up to the user to figure it out.

many laptops already have great suspend

  • suspend "just works" out of the box on just about every laptop
  • most laptops can get over a day of suspend on a single charge
  • for most users, standard suspend is more than sufficient for their workflows. suspend just needs to last long enough to get between meetings/commuting/charging sessions.

so is hibernation worth it?

  • obviously, most of this section is going to be opinion.
  • honestly, i don't think it is. i don't bother with it on any of my machines.
  • if i had a machine with bad suspend, i might consider it.

  • if your workflow requires leaving your computer unused for a week, but shutting it down is really bad for some reason, hibernate might be worth it.

  • looking at a couple more streamlined uses of linux:

    • valve doesn't bother with hibernate for their steamdeck.
    • from what i can tell, google does seems to use hibernate on most chromebooks.

because of all of that, i won't be focusing much on hibernation when looking at the different swap setups. the focus of the comparison will be the resource constraints, security, and performance of each swap implementation

swapfile and swap partition

  • a swapfile and swap partition are very similar, so we will talk about them together

similarities

differences

swapfile

swap parititon

  • annoying to resize if you aren't using LVM
  • hard to create after initial system setup if you aren't using LVM

swap on zram

  • compressing/decompressing the data is significantly faster than read/writing from persistent storage, even on slow/older cpus.
    • effectively gives you more ram at the cost of extremely-extremely minor CPU usage.
  • good for systems with sd cards as it doesn't contribute to sd card wear
  • your memory contents never touch persistent storage, so no security risk on unencrypted systems
  • some notes on zram configuration:

zswap

  • https://www.kernel.org/doc/html/latest/admin-guide/mm/zswap.html
  • provides the advantages of zram, while allowing additional swap space for extremely infrequently used pages
  • security concerns of traditional swapfile/swap partition apply, if you don't use encryption, your memory content end up readable from your persistent storage device
  • does incur more wear on your persistent storage than zram, but much less than traditional swapfiles/swap partitions
    • might not be worth it on systems using SD cards
  • allows for more swap room than zram when necessary, both by expanding the in ram compressed cache, and by offloading to a compressed swapfile/swappartition
  • expands the in ram compressed cache when necessary, vs zram which has a static in-ram cache size
  • causes additional wear on persistent storage
    • this is especially bad on SD cards, but also impacts SSDs
  • can use up precious persistent storage space on systems with small ssds/sds/hdds/etc

final configurations

For the following configurations:

  • desktops with powerful cpus, lots of memory (>>16GB), and persistent nvme SSD storage (>1TB)
  • laptops with powerful cpus, lots of memory (>>16GB), and persistent nvme SSD storage (>1TB)
  • laptops with weak cpus, minimal memory (~8GB or less), and persistent nvme SSD storage (~128GB)
  • NAS server, moderate cpu, ~32GB memory, and persistent nvme SSD storage

i settled on the following zswap configuration:

"zswap.enabled=1" # enables zswap "zswap.compressor=zstd" # compression algorithm "zswap.max_pool_percent=20" # maximum percentage of RAM that zswap is allowed to use "zswap.shrinker_enabled=1" # whether to shrink the pool proactively on high memory pressure

and combined it with a 16GB swapfile on my rootfs, which means its automatically encrypted.

nixos config example here: https://github.com/SolidEva/prawnix/blob/main/modules/swap/zswap.nix

Some rationale: zstd seems to be the best compromise between compression speed and ratio for this use case. lzo and lz4 are both faster than zstd, but zstd gets much better compression ratios. the more we can compress, the more we can fit in the zswap pool and the faster we can copy the swapped data. there isn't a lot of data available comparing the compression algos for this tasks unfortunately. enabling the shrinker seems to be better at punting cold pages to persistent storage according to https://github.com/torvalds/linux/commit/b5ba474f3f518701249598b35c581b92a3c95b48 which is nice for things like browser pages

For the last configuration:

  • SBCs (like the rpi), weak cpus, minimal memory, and SD card persistent storage

i settled on using the following zram configuration:

ALGO=zstd PERCENTAGE=30

nixos config example here: https://github.com/SolidEva/prawnix/blob/main/modules/swap/zram.nix