Firecracker microVMs running .NET functions per HTTP request, storage optimised with OverlayFS, sparse function images compressed for transfer, requests routed through a gateway and queued in Kafka, microVMs launched on demand and talked to over vsock. Cold starts around ten seconds, much faster warm.

I promised a few things in part three and then got derailed doing cleanup instead.

Let’s be real

• That crappy init script couldn’t take it any more. It panicked when dotnet died. Not ideal.
• Creating runtime images by hand was a massive pain. Also not ideal.

Fixing it

I tried compiling systemd for a full day. Even with its most minimal options it came out at 200 MB. Screw that — I compiled BusyBox init as a static binary instead.

Then I wrote a Dockerfile that builds the rootfs inside Docker, so I can use it directly for my VMs.

It also turns out ext4 is a pain as a read-only disk, so I switched to SquashFS, which is built for exactly that.

Tried it. Crashed.

BusyBox’s init does nothing unless you tell it to, in inittab. So, after more negotiation with Copilot, I added the mounts .NET actually needs:

::sysinit:/bin/mount -t devtmpfs devtmpfs /dev
::sysinit:/bin/mkdir -p /dev/pts
::sysinit:/bin/mount -t devpts devpts /dev/pts
::sysinit:/bin/mount -t proc proc /proc
::sysinit:/bin/mount -t sysfs sys /sys
::sysinit:/bin/mkdir -p /tmp /dev/shm /var/tmp /run
::sysinit:/bin/mount -t tmpfs -o mode=1777,nosuid,nodev tmpfs /tmp
::sysinit:/bin/mount -t tmpfs -o nosuid,nodev tmpfs /dev/shm
::sysinit:/bin/mount -t tmpfs -o mode=0755,nosuid,nodev tmpfs /run
::sysinit:/bin/chmod 1777 /var/tmp

::respawn:/bin/sh -c 'exec socat TCP-LISTEN:55505,fork,reuseaddr VSOCK-CONNECT:2:80'
::once:/bin/sh -c '/bin/dotnet "/app/$(cat /entrypoint.txt)"; rc=$?; echo "dotnet exited rc=$rc, powering off" >&2; sync; poweroff -f'

Back where we started — but better

Theoretically I can now convert any Docker container into a runtime by injecting my static init binaries.

Elsewhere:

• I started implementing a SquashFS driver in .NET for no good reason, and halfway through discovered I needed ext4, not SquashFS.
• In-process DEFLATE (de)compression replaced the finicky tar shell-out.
• Endpoints for adding runtimes — thanks to SquashFS I don’t need to compress them at all.

Roadmap

• Binary protocol for runtime communication
• Dogfooding: FaaS-based overlay image creation
• Non-HTTP triggers (MinIO events and friends)
• A frontend, so no more Postman
• Authorization, authorization, authorization
• Snapshots
• Replace the terrible Kafka consuming logic with kafka-offset-manager

An API gateway mapping HTTP routes to (FunctionId, FunctionVersion), disk bloat from giving every microVM its own full rootfs, and the fix: OverlayFS with one shared read-only base and a per-VM sparse writable ext4 layer.

Enough preparation. Let’s walk through MangoFaaS from the point of view of everyone involved.

I’m an HTTP request. My journey starts in MangoFaaS.Gateway. I carry a Host and a Path; the gateway uses them to work out the tenant and which function version to invoke.

I’m a function owner. My journey starts in MangoFaaS.Functions. I register a function by uploading a ZIP of my .NET application to S3 with metadata: entrypoint and runtime. A background job discovers new version ZIPs in the bucket, builds a function overlay image, and pushes it to an images bucket.

Worth noting: those images are sparse files. To avoid shipping all the zero-filled regions we compress them — currently with GNU tar, because .NET’s Tar implementation doesn’t preserve sparseness. So we shell out to the lovely GNU tar.

I’m the HTTP request again. I’ve been routed and I’m sitting in Kafka. I move toward the first free MangoFaaS.Firecracker.Node. (Future improvement: prefer nodes that already have my function cached.)

I’m MangoFaaS.Firecracker.Node. I:

• put the request in a pending list;
• download, if not cached, the latest kernel, the runtime base image and the function’s overlay image;
• launch or prepare a Firecracker microVM and wait for the function runtime to connect to my Kestrel server over vsock and ask for work.

I’m the function runtime. I connect and poll the Kestrel server over vsock, process the request, and send the response back down the same channel.

The response then travels back through the node and out via the gateway that took the inbound request.

Early timings

Next: snapshots, prewarming, smarter scheduling, cache affinity.

I explored Firecracker, compared it with containers, chose to build a snapshot-based mini-FaaS, and got a first microVM running with a custom rootfs and a prebuilt kernel. Networking fixed, VMs booting, and a plan to manage them from C#.

Preparation

I have a proof-of-concept API gateway that matches an HTTP request to a (FunctionId, FunctionVersion) pair using a route table in the database with in-memory caching.

Function id? Function version? Wat?

Fair. I still have no way to create a function. A function, in FaaS terms, is a small, stateless, event-triggered piece of code executed on demand in an ephemeral runtime that scales per invocation. To run that code I need an image — which is where part one left off.

And here’s the new problem: as soon as I tried to run many microVMs on my not-so-micro VM, I ran out of disk.

OverlayFS

OverlayFS is a Linux kernel feature that combines several mount points into one directory tree containing the files from all of them.

The classic use is overlaying a read/write partition on a read-only one. Which means every VM can share the same read-only rootfs, and each gets its own thin read-write layer on top. Disk problem solved.

The code

Base image, once:

1. Create your base rootfs image.
2. Inside it: mkdir -p /mnt/rootfs/overlay/root /mnt/rootfs/overlay/work /mnt/rootfs/mnt /mnt/rootfs/rom
3. Drop in the overlay-init script from firecracker-containerd — a separate init that runs before your main init and does the preparation (you can drop the line that creates the volumes folder).
4. chmod +x it.
5. Unmount. Done.

Per-VM overlay:

dd if=/dev/zero of=overlay.ext4 conv=sparse bs=1M count=1024
mkfs.ext4 overlay.ext4

That’s it. Meanwhile the C# side of the platform is taking shape in MangoFaaS.

Sources

• https://github.com/aarani/mangofaas
• https://github.com/njapke/overlayfs-in-firecracker
• https://en.wikipedia.org/wiki/OverlayFS

What is AWS Firecracker?

Firecracker is an open-source virtual machine monitor from AWS that runs lightweight microVMs on KVM, optimised for secure multi-tenancy, fast startup and low overhead. It’s what powers Lambda and Fargate: VM-grade isolation with startup times closer to containers.

So why not just a container?

• Firecracker uses KVM hardware virtualisation with its own guest kernel; containers share the host kernel. The multi-tenant isolation story is much stronger.
• Containers start in tens of milliseconds. Firecracker cold starts are ~100–300 ms — and you can optimise the heck out of that down to single-digit ms.
• MicroVMs are light, but they still carry a guest kernel, so density is lower.
• Containers use OCI images; Firecracker wants a kernel + rootfs. You build your own images. firecracker-containerd exists, but I wanted to start from scratch.

Picking a project

I asked Copilot what to build and got three options: a declarative launchpad CLI, a snapshot-accelerated mini-FaaS, or a hardened multi-tenant code runner. The CLI was too easy. The code runner reminded me of ACM contest auto-judges — fun, but niche. So: mini-FaaS.

I promptly ignored the plan and started with a WebAPI project that swallows every request — all paths, hosts and methods — and forwards them to Kafka.

Firecracker comes into play

I’m on a Mac, which can’t run Firecracker, so I rented a Linux VM from DigitalOcean (Hetzner doesn’t do nested virtualisation). I needed a kernel and a rootfs. Building the kernel — even a shallow clone — took forever on a cheap VM, so I gave up and took Amazon’s prebuilt one:

ARCH="$(uname -m)"
release_url="https://github.com/firecracker-microvm/firecracker/releases"
latest_version=$(basename $(curl -fsSLI -o /dev/null -w  %{url_effective} ${release_url}/latest))
CI_VERSION=${latest_version%.*}
latest_kernel_key=$(curl "http://spec.ccfc.min.s3.amazonaws.com/?prefix=firecracker-ci/$CI_VERSION/$ARCH/vmlinux-&list-type=2" \
    | grep -oP "(?<=<Key>)(firecracker-ci/$CI_VERSION/$ARCH/vmlinux-[0-9]+\.[0-9]+\.[0-9]{1,3})(?=</Key>)" \
    | sort -V | tail -1)

wget "https://s3.amazonaws.com/spec.ccfc.min/${latest_kernel_key}"

Rootfs time — exported straight out of a container image:

docker pull mcr.microsoft.com/dotnet/aspnet:9.0
CONTAINER_ID=$(docker create mcr.microsoft.com/dotnet/aspnet:9.0)
docker export $CONTAINER_ID -o rootfs.tar
dd if=/dev/zero of=rootfs.ext4 bs=1M count=2048
mkfs.ext4 rootfs.ext4
mkdir /mnt/rootfs
sudo mount -o loop rootfs.ext4 /mnt/rootfs
sudo tar -xf rootfs.tar -C /mnt/rootfs

There’s a twist: containers don’t use an init process, so most container images don’t ship one. I wrote a sample init and moved on.

It works. It has no internet.

Neither container image I tried had ip installed, so I configured networking through the kernel command line instead, pointed the resolver at 1.1.1.1, and this time it really worked.

Next

Each Firecracker process should serve exactly one tenant — which means I now need to launch the Firecracker binary from C#, talk to its API, and boot VMs programmatically.