Does Kubernetes node swap pay off on a Raspberry Pi?

The Kubernetes blog reports that swap let a kernel build run in half the memory, and finish faster. I repeated the build on a Raspberry Pi 5 in my k3s cluster, with swap on NVMe and on zram. The memory halved, but no build got faster, and the bill came as drive writes on NVMe and CPU time on zram.

I run a small Kubernetes cluster at home. It is four Raspberry Pi 5 boards, each with 16 GB of RAM and an NVMe drive on a combined PoE and NVMe HAT, so there is not a single SD card in it. Three of the Pis form the k3s control plane with embedded etcd, and the fourth is an agent node that gets the experiments. Ansible configures the hosts, Flux keeps the cluster in step with a Git repository, and Longhorn replicates storage across the NVMe drives.

It started as a place to try things, and it still is one. I deploy my own projects on it, break things on purpose, and learn how Kubernetes behaves when it is not running in someone else’s cloud. A Raspberry Pi is not a powerful machine. Four Cortex-A76 cores do not get you far. But 16 GB of RAM per node leaves plenty of headroom, and headroom is what makes the cluster fun to experiment with.

So when the Kubernetes blog published a post saying that swap can fit up to three times as many pods on a node, I wanted to see what it does on mine. One result in that post caught my eye in particular: a kernel build that ran in half the memory with swap also finished faster than the build without it. Swap is the slow path. Making things faster is not what it’s known for.

Kubernetes can swap now

Scaling Kubernetes Workloads with Node Swap appeared on 5 October. Swap on nodes has been generally available since Kubernetes v1.34, and the post benchmarks it on Google Cloud machines that swap to local NVMe SSDs. Depending on the workload and the sandbox runtime, a node held between 25% and 200% more pods with swap. The headline figure of three times is the top of that range: 80 isolated Python sandboxes per node without swap, 240 with it. The first benchmark is a kernel build. Linux 6.1.1 needed a 600 MB memory limit to build without swap and only 300 MB with it, and the build with swap took 374 seconds against 433 seconds without.

By default the kubelet still refuses to start on a node with swap, and for years there was no way around it. One reason was accounting: cgroup v1, the kernel interface Kubernetes uses to limit containers, had a single limit for memory and swap combined. cgroup v2 tracks swap on its own, in memory.swap.max, and node swap is built on top of that. The day after the swap post, the blog published a reminder that cgroup v1 has been in maintenance mode since v1.31, and that since v1.35 the kubelet does not start on a cgroup v1 node by default. Checking a node takes one command:

$ stat -fc %T /sys/fs/cgroup
cgroup2fs

All four of my nodes answered cgroup2fs. No surprise there: k3s v1.36 would not have started on them otherwise.

The surprise was elsewhere. Raspberry Pi OS already comes with swap. Its rpi-swap service sets up 2 GiB of zram on every node, a block device in RAM that compresses whatever is written to it. And k3s already runs the kubelet with failSwapOn: false, which is why it starts on such a node at all. What keeps the pods away from that swap is the kubelet’s default swap behaviour, NoSwap, which writes a memory.swap.max of 0 into every container’s cgroup. The k3s agent and containerd processes on the host could swap all along. The pods could not.

Who gets swap: the request decides

Switching the kubelet to LimitedSwap does not open all of the swap to every pod. Each container gets a fixed allowance, and the kubelet calculates it from the container’s memory request:

swap allowance = memory request / node memory * node swap

A container gets the same share of the swap as its request is of the node’s RAM. Request 1/64 of the memory and you get 1/64 of the swap. Only containers in Burstable pods get any at all. Guaranteed and BestEffort pods get none, and neither do pods with the system-node-critical or system-cluster-critical priority.

There is one more case that is easy to miss. A container whose memory request equals its limit gets no swap either, even inside a Burstable pod. I checked with a handful of idle pods on a node with 16 GiB of swap:

The first three pods are Burstable, the middle two only because they have a CPU request and no CPU limit. Where the kubelet gives swap, it is the formula to the decimal: 256 * 16384 / 16218 = 258.6 MiB. A memory limit without a request lands in a zero row too, because the request then defaults to the limit. So the common advice to set memory requests equal to limits opts a container out of swap. It nearly opted my benchmark out as well: the first version of the test pods did exactly that, and the swap runs would have quietly run without swap.

It also works the other way round. A namespace with a LimitRange that fills in a default request below the default limit makes every pod that sets no resources of its own Burstable. My development namespace does that, so a pod there with no resources at all would get 258.6 MiB of swap. Under LimitedSwap, everything nobody thought about gets swap.

With the stock 2 GiB of zram, a container gets 12.6% of its request as swap, far too little to halve anything. To give a pod about as much swap as its request, the node needs about as much swap as RAM, so the benchmark used 16 GiB. And the request is the only knob. Raising it is the one way to give a container more swap, and it also reserves more RAM at the scheduler. We will come back to that.

The test

I repeated the kernel build from the Kubernetes post on the agent node, natively on arm64: Linux 6.1.1, defconfig, GCC 12.2 from Debian bookworm, make -j4 Image. A full build with modules took 41 minutes, too long for the number of runs I wanted, so every build stops at the kernel image. The pod got a 3-CPU limit, so that a build could not starve everything else on the node, and one build took about 20 minutes.

There were three setups, one swap device at a time:

• no swap: the node as it was, with NoSwap
• NVMe: a 16 GiB swapfile on the root partition of the node’s drive, a Kingston FURY Renegade 2 TB on the Pi’s single PCIe lane, set to Gen3
• zram: a 16 GiB zram device with zstd, the compressor Raspberry Pi OS uses by default, and later the same with lz4

Without swap, I narrowed down the smallest limit at which the build still finished. With swap, I ran the build at fractions of that limit until it was OOM-killed, mostly with two runs per limit, in shuffled order, and lz4 at two of those limits. In the swap runs the memory request was 1 MiB below the limit, for the reason above. Swap for pods takes a kubelet configuration drop-in, which k3s reads from its agent directory, and a restart of k3s-agent:

# /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/90-swap.conf
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memorySwap:
  swapBehavior: LimitedSwap

failSwapOn: false is the k3s default already. All of this was done by hand on the one node, with every change logged and reverted at the end. A permanent setup would go through Ansible like the rest of the host configuration.

That gives four claims from the Kubernetes post to test:

1. With swap, the build needs half the memory.
2. With swap, it runs faster.
3. Squeezing the limit further makes it much slower: over 40% at 200 MB.
4. Swap can triple the number of pods on a node.

The fourth one I did not measure, so it gets arithmetic instead. On top of those come two questions the post has no reason to ask on cloud machines: is zram a better place to swap to than the NVMe, and what does swap cost the drive?

The post gives memory in MB, and my limits are in MiB, the way Kubernetes counts them. The 5% difference changes nothing below.

The memory floor halves, and the formula explains why

Without swap, the smallest limit at which every build finished was 576 MiB. With swap on the NVMe it fell to 320 MiB, and 288 MiB, exactly half, finished once in two tries. On zram both builds at 288 MiB finished. That matches the Kubernetes post’s result.

The reason is the formula, not the drive. The pod requested 1 MiB less than its limit, so with 16 GiB of swap on a 16 GiB node it got about as much swap as its limit. Memory plus swap was then about twice the limit, and the floor sat where twice the limit meets the no-swap floor: twice 288 is 576. Any device that can hold the pages would have done the same.

To see what those 576 MiB are, we need to look at the two kinds of memory a process uses. Page cache holds file data, here the kernel sources, the headers and the object files, and the kernel can drop it at any time and read it from disk again later. Anonymous memory is what a program builds for itself, such as the compiler’s working state, and it has no file to go back to. With an 8 GiB limit, the pod’s memory peaked at about 3.8 GiB, and over 3 GiB of that was page cache. The anonymous part peaked at 500 to 540 MiB, and that is what sets the floor. Without swap, the only way for the kernel to make room is to drop cache. Swap gives anonymous memory somewhere to go.

The floor is not a sharp line either. At 560 MiB the build finished once in three runs. About two minutes in, four compiler processes are briefly alive at the same time, and whether they fit depends on which four files make -j4 happens to compile side by side. When they do not fit, the whole container dies, every make, gcc and cc1 in it. That is another cgroup v2 default: the kubelet sets memory.oom.group on each container, so an OOM kill takes all of its processes instead of leaving half a build running.

Swap did not make anything faster

This is the result I was most curious about, and it did not reproduce. At 576 MiB the build took 1218 seconds without swap and 1218 to 1226 seconds with it, on either device. At about half the memory, 288 and 320 MiB, which is the comparison the post makes, the NVMe builds took 4 to 9% longer than that, and the zram builds 11 to 33% longer. Swap did not make the build faster at any limit.

I expected it to. A build squeezed to its floor without swap keeps dropping page cache and reading the same sources back from disk, while with swap the kernel can park idle compiler memory instead. The first half is true. At 576 MiB the build read 2.2 to 2.3 GiB from the drive, against 0.2 GiB at most with 8 GiB of memory. But on NVMe those re-reads are nearly free: the squeezed build still finished within 1% of the 8 GiB one.

And the 8 GiB build itself took 1207 seconds. Fourteen times the memory the build needed made no difference either. The build is limited by its three CPUs. In every run that finished, with or without swap, the wall time was the pod’s CPU time divided by three, plus 13 to 49 seconds of waiting. There was nothing left for swap to win back.

Why the build in the post got faster, I do not know for sure. The post gives no number of parallel jobs, kernel configuration, CPU or swap size, and it does not say how many times each build ran. Its example repository does describe the nodes behind the browser benchmarks, though. The baseline nodes have no local SSD and run on network persistent disk, which the repository itself names as a bottleneck for re-reading evicted page cache. The swap nodes have a local SSD and tuned vm.swappiness and vm.watermark_scale_factor. If the kernel build ran on the same two kinds of node, the baseline paid for every re-read over the network, and swap had real work to save. That is a guess. The repository does not cover the kernel build.

Below the floor: OOM killer

The post reports that a 200 MB limit forced the build’s working set into swap and made it over 40% slower. On the Pi, there was no such slope. At 176 and 144 MiB the pod was OOM-killed 7 to 10 seconds in, on both NVMe and zram. Twice 176 MiB is 352 MiB of memory and swap, and the build needs more than that within its first few seconds.

The slowest build that did finish took 10% longer than the 8 GiB build on NVMe and 34% longer on zram. That is the other side of LimitedSwap. The allowance is a hard cap, and a pod that needs more than its memory plus its swap is killed rather than left to thrash. Whether a quick death beats a slow build depends on what you run.

NVMe swap pays in drive writes

Swap traffic has to go somewhere, and on NVMe it goes to the drive. A build without swap wrote 2.2 GB, mostly its own output. With NVMe swap at 432 MiB it wrote 14 GB, and at 320 MiB between 30 and 81 GB. The worst single build wrote as much as three and a half normal days of everything on that node. None of it shows up in the build time.

How much gets written has little to do with how much swap the pod uses, either. In the first build at 288 MiB, the pod held a median of 39 MiB in swap, and its swap allowance was full for under 1% of the build. Yet over the whole build, 58 GiB went out to the drive and 24 GiB was read back. Swap never held much at a time, but pages kept passing through it, out to the drive and often back again, and every page that goes out is a write. That traffic is what wears the drive, and it is hard to predict: two builds at the same 320 MiB limit differed almost threefold.

Reading pages back is where swap costs time. The Pi 5 kernel uses 16 KiB pages, four times the 4 KiB of x86, and every page brought back from swap is one 16 KiB read, about 80 microseconds on this drive.

That leaves wear. The drive is rated for 2,000 TB written, and at the worst build’s 81 GB that is about 25,000 such builds. That build took 22 minutes, so the drive would last about a year of building around the clock.

zram pays in CPU

zram keeps the drive out of it. With zram, every build wrote 2.3 to 3.2 GB, about what a build without swap writes, while up to 100 GiB went through swap. But zram was never faster than the NVMe. It tied at 576 MiB, then lost 2 to 3% at 432 MiB, 5 to 7% at 320 MiB and 5 to 24% at 288 MiB, where the slowest build of the whole benchmark took 1621 seconds.

The whole difference is compression, and the pod pays for it. When a pod sits at its memory limit, every page it brings back pushes another one out, and on zram pushing out means compressing, in the pod’s own threads and on its own CPU limit. Every GiB swapped out cost the pod about 10.5 seconds of CPU with zstd, against about 3 on NVMe. In a build already bound by its three CPUs, that CPU time is wall time.

lz4 compresses faster. It halved the CPU cost to about 5 seconds per GiB, and at 432 MiB it matched the NVMe build to within a few seconds. The price is RAM: lz4 packed the compiler’s memory about four to one, zstd about seven and a half to one, so lz4 needs about twice the RAM for the same swapped pages.

That RAM is the second thing to know about zram: it is not charged to the pod. The pod’s memory.current stayed at its limit while zram held up to 40 MiB for it with zstd and 78 MiB with lz4, so the pod really used up to 14% more RAM than its limit said with zstd and up to 24% with lz4, and neither the kubelet nor the scheduler sees it. Compiler memory compresses well. Data that does not compress is stored as it is, and a pod full of it could keep its whole swap allowance in RAM outside its limit: up to twice its limit in total.

Against the Kubernetes blog

Halving the memory holds, and the formula explains it, whatever the swap device. The speed-up did not reproduce. The 200 MB result may come down to the amount of swap. The post’s example repository dedicates a whole local SSD to swap, 375 GiB, on machines with 30 to 120 GB of RAM, and on such a node the formula hands a container several times its request in swap. That would be enough to keep a 200 MB build alive, where mine got its limit’s worth and was killed. Whether the kernel build ran on such a node, the post does not say.

The density claim I did not test, but we can still put a ceiling on it. The scheduler keeps the sum of memory requests below the node’s RAM, so the allowances add up to the node’s swap at most, and only when the requests fill the node. Pods can then use RAM plus swap at best, or 1 + swap/RAM times what fits without swap. On a 16 GiB Pi with 16 GiB of swap that is twice as many pods, and only if every pod’s idle memory can actually leave RAM. Three times needs twice as much swap as RAM, as on the Google Cloud nodes in the post. On zram the ceiling is lower still, because the compressed pages stay in RAM.

This is where the request works against density. Enough swap needs high requests, and high requests are what limits how many pods the scheduler will place. The post says the node “automatically rations fast swap space based on idle application memory usage”. It rations by memory request, however much of a container’s memory sits idle.

To be fair to the post, a busy compiler is the opposite of what it wants swap for. Its main workloads are agent sandboxes and browser sessions that load a lot into memory and then sit idle waiting for the next prompt, and idle memory is exactly what swap is good at moving out of the way. The post itself calls swap “an insurance policy for burst memory, not a replacement for active RAM”. A kernel build is all active RAM.

Before you turn it on

A few things that bit me, or nearly did, on k3s and Raspberry Pi OS:

• Restart the kubelet after every swap change. The kubelet reads the node’s swap capacity once, when it starts, and divides that among the pods. After I switched swap devices it reported 18 GiB, 16 for the swapfile plus the 2 GiB zram that had come back in the meantime, and only a second restart fixed it. status.nodeInfo.swap.capacity on the node shows what the kubelet believes.
• rpi-swap does not stay off. Stopping dev-zram0.swap swapped zram off, and within a second systemd set it up and swapped it on again, at the same priority as the swapfile. Two devices at equal priority share the pages, so half of “NVMe swap” would have gone to zram. What worked was systemctl mask --runtime dev-zram0.swap followed by swapoff /dev/zram0.
• k3s has no memory-pressure eviction. Its kubelet sets hard eviction thresholds for disk space only, and setting any hard threshold drops the defaults, memory.available among them. When a node runs out of memory, the kernel OOM killer decides, and it picks BestEffort pods first. On my agent node that would be Traefik and the Longhorn CSI pods.
• PSI is off on Raspberry Pi OS. Pressure Stall Information, which the kubelet reports per pod, needs psi=1 on the kernel command line. On zram it also counts compression as waiting for memory: at 432 MiB the zstd builds showed 28.5 seconds of full memory stall against 6.2 on NVMe, at about the same build time.

So, does it pay off?

On my cluster, partly. Swap made room. It did not make anything faster, and on a Pi the room is paid for either in drive writes or in CPU time. So NoSwap stays the default, and LimitedSwap goes on where fitting pods in matters more than how fast they run.

When it does go on, it will be swap on the NVMe rather than zram. That is not where I expected to end up, because zram is what Raspberry Pi OS ships and it spares the drive. But CPU is what a Pi has least of, and zram spends it inside the pod that swaps, on top of the RAM it takes without anyone noticing. The NVMe writes are real, but a drive rated for 2,000 TB can take a lot of them. What I have not measured is what heavy swap traffic does to Longhorn on the same drive. That is the next experiment.

A kernel build is also a hard case for swap: a busy program with little idle memory, bound by CPU from start to finish. Plenty of what runs on my cluster looks different, such as services that load a lot at start-up and then mostly wait, or that need a burst of memory now and then. That is what swap is meant for, and I suspect that on a machine with more and faster cores the results look rather different, as the Kubernetes post suggests. Whether swap gets me more pods on a Pi, and how many, is the one claim I have not tested. The arithmetic says twice as many at most.