· firecracker · mangofaas · linux
Firecracker, part four: BusyBox init, SquashFS, and a day lost to systemd
Replacing a fragile init, building rootfs images in Docker, and ending up exactly where I started — but better.
Previously
Firecracker microVMs running .NET functions per HTTP request, storage optimised with OverlayFS, sparse function images compressed for transfer, requests routed through a gateway and queued in Kafka, microVMs launched on demand and talked to over vsock. Cold starts around ten seconds, much faster warm.
I promised a few things in part three and then got derailed doing cleanup instead.
Let’s be real
- That crappy init script couldn’t take it any more. It panicked when
dotnetdied. Not ideal. - Creating runtime images by hand was a massive pain. Also not ideal.
Fixing it
I tried compiling systemd for a full day. Even with its most minimal options it came out at 200 MB. Screw that — I compiled BusyBox init as a static binary instead.
Then I wrote a Dockerfile that builds the rootfs inside Docker, so I can use it directly for my VMs.
It also turns out ext4 is a pain as a read-only disk, so I switched to SquashFS, which is built for exactly that.
Tried it. Crashed.
BusyBox’s init does nothing unless you tell it to, in inittab. So, after more
negotiation with Copilot, I added the mounts .NET actually needs:
::sysinit:/bin/mount -t devtmpfs devtmpfs /dev
::sysinit:/bin/mkdir -p /dev/pts
::sysinit:/bin/mount -t devpts devpts /dev/pts
::sysinit:/bin/mount -t proc proc /proc
::sysinit:/bin/mount -t sysfs sys /sys
::sysinit:/bin/mkdir -p /tmp /dev/shm /var/tmp /run
::sysinit:/bin/mount -t tmpfs -o mode=1777,nosuid,nodev tmpfs /tmp
::sysinit:/bin/mount -t tmpfs -o nosuid,nodev tmpfs /dev/shm
::sysinit:/bin/mount -t tmpfs -o mode=0755,nosuid,nodev tmpfs /run
::sysinit:/bin/chmod 1777 /var/tmp
::respawn:/bin/sh -c 'exec socat TCP-LISTEN:55505,fork,reuseaddr VSOCK-CONNECT:2:80'
::once:/bin/sh -c '/bin/dotnet "/app/$(cat /entrypoint.txt)"; rc=$?; echo "dotnet exited rc=$rc, powering off" >&2; sync; poweroff -f'
Back where we started — but better
Theoretically I can now convert any Docker container into a runtime by injecting my static init binaries.
Elsewhere:
- I started implementing a SquashFS driver in .NET for no good reason, and halfway through discovered I needed ext4, not SquashFS.
- In-process DEFLATE (de)compression replaced the finicky tar shell-out.
- Endpoints for adding runtimes — thanks to SquashFS I don’t need to compress them at all.
Roadmap
- Binary protocol for runtime communication
- Dogfooding: FaaS-based overlay image creation
- Non-HTTP triggers (MinIO events and friends)
- A frontend, so no more Postman
- Authorization, authorization, authorization
- Snapshots
- Replace the terrible Kafka consuming logic with kafka-offset-manager