· firecracker · mangofaas · linux
Firecracker, part two: one rootfs, many microVMs
OverlayFS, sparse writable layers, and how to stop a not-so-micro VM running out of disk.
Previously
I explored Firecracker, compared it with containers, chose to build a snapshot-based mini-FaaS, and got a first microVM running with a custom rootfs and a prebuilt kernel. Networking fixed, VMs booting, and a plan to manage them from C#.
Preparation
I have a proof-of-concept API gateway that matches an HTTP request to a (FunctionId, FunctionVersion) pair using a route table in the database with in-memory caching.
Function id? Function version? Wat?
Fair. I still have no way to create a function. A function, in FaaS terms, is a small, stateless, event-triggered piece of code executed on demand in an ephemeral runtime that scales per invocation. To run that code I need an image — which is where part one left off.
And here’s the new problem: as soon as I tried to run many microVMs on my not-so-micro VM, I ran out of disk.
OverlayFS
OverlayFS is a Linux kernel feature that combines several mount points into one directory tree containing the files from all of them.
The classic use is overlaying a read/write partition on a read-only one. Which means every VM can share the same read-only rootfs, and each gets its own thin read-write layer on top. Disk problem solved.
The code
Base image, once:
- Create your base rootfs image.
- Inside it:
mkdir -p /mnt/rootfs/overlay/root /mnt/rootfs/overlay/work /mnt/rootfs/mnt /mnt/rootfs/rom - Drop in the
overlay-initscript from firecracker-containerd — a separate init that runs before your main init and does the preparation (you can drop the line that creates the volumes folder). chmod +xit.- Unmount. Done.
Per-VM overlay:
dd if=/dev/zero of=overlay.ext4 conv=sparse bs=1M count=1024
mkfs.ext4 overlay.ext4
That’s it. Meanwhile the C# side of the platform is taking shape in MangoFaaS.