Files

Jan Novak 640bf5065a feat: initial Firecracker snapshot orchestrator (fc-orch)

A "poor man's" Firecracker VM orchestrator that boots a single golden VM,
snapshots it, then restores N clone VMs from that snapshot with minimal
per-clone overhead.

How it works:
- `init`   — downloads a Linux 6.1 kernel and builds a minimal Alpine 3.20
             rootfs (512 MiB ext4) with a basic init script
- `golden` — boots the golden VM, lets it settle, then pauses and snapshots
             it (vmstate + memory file); the golden VMM is then terminated
             since only the artifacts are needed
- `spawn N` — restores N clone VMs concurrently from the golden snapshot:
               * rootfs: filesystem-level COW copy via `cp --reflink` (falls
                 back to a plain copy if reflinks are not supported)
               * memory: shared golden `mem` file; Firecracker's MAP_PRIVATE
                 lets the kernel handle COW page-by-page at no up-front cost
               * vmstate: small file, cheap regular copy per clone
               * networking: per-clone TAP device (fctapN) bridged to fcbr0
                 with iptables MASQUERADE NAT on the default route interface
- `status`  — reads PID files and checks /proc to report alive/dead clones
- `kill`    — stops in-memory clones, kills any stragglers via PID files,
              and tears down all fctap* devices
- `cleanup` — kill + remove all state dirs and the bridge

All tunables (binary path, base dir, kernel/rootfs paths, vCPUs, memory,
bridge name/CIDR) are configurable via environment variables.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-04-09 09:53:16 +02:00

4.4 KiB

Raw Blame History

Dev VM Setup Guide

Ubuntu 24 setup for running firecracker-orchestrator.

Prerequisites on your hypervisor

Before creating the VM, enable nested virtualization (expose KVM to the guest):

VMware Fusion: VM Settings → Processors & Memory → Advanced → enable "Enable hypervisor applications in this virtual machine"
Parallels: VM Config → CPU & Memory → Advanced → enable "Enable nested virtualization"
UTM / QEMU: Add -cpu host flag or enable "Force multi-core" + "Enable Hypervisor"

The VM must be x86_64 — the orchestrator downloads an x86_64 kernel and Alpine rootfs at init time.

Step 1 — Verify KVM is accessible

ls -la /dev/kvm
# Expected: crw-rw---- 1 root kvm ...

If /dev/kvm doesn't exist, nested virtualization isn't enabled on your hypervisor — go back and fix that first.

Add your user to the kvm group so firecracker doesn't require root:

sudo usermod -aG kvm $USER
# Log out and back in, then verify:
groups | grep kvm

Step 2 — Install system dependencies

sudo apt update && sudo apt install -y \
  e2fsprogs \
  iproute2 \
  iptables \
  tar \
  wget \
  curl \
  btrfs-progs

Package	Used for
`e2fsprogs`	`mkfs.ext4` — formats the Alpine rootfs image
`iproute2`	`ip` — manages TAP devices and bridge
`iptables`	NAT masquerade for VM networking
`tar`	Extracts the Alpine minirootfs tarball
`btrfs-progs`	`mkfs.btrfs` — enables COW reflink copies (see Step 5)

Step 3 — Install Go 1.26.2

wget https://go.dev/dl/go1.26.2.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.26.2.linux-amd64.tar.gz
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
source ~/.bashrc
go version
# Expected: go version go1.26.2 linux/amd64

Step 4 — Install Firecracker 1.15.1

FIRECRACKER_VERSION=1.15.1
wget https://github.com/firecracker-microvm/firecracker/releases/download/v${FIRECRACKER_VERSION}/firecracker-v${FIRECRACKER_VERSION}-x86_64.tgz
tar -xzf firecracker-v${FIRECRACKER_VERSION}-x86_64.tgz
sudo mv release-v${FIRECRACKER_VERSION}-x86_64/firecracker-v${FIRECRACKER_VERSION}-x86_64 /usr/local/bin/firecracker
sudo chmod +x /usr/local/bin/firecracker
firecracker --version

Step 5 — Set up btrfs working directory

The orchestrator uses cp --reflink=always for COW rootfs copies when spawning clones. This only works on btrfs or xfs — on the default ext4 it silently falls back to a full file copy, which wastes disk I/O and space.

Create a btrfs volume mounted at the orchestrator's working directory:

sudo dd if=/dev/zero of=/var/fc-orch.img bs=1M count=8192   # 8 GB
sudo mkfs.btrfs /var/fc-orch.img
sudo mkdir -p /tmp/fc-orch
sudo mount -o loop /var/fc-orch.img /tmp/fc-orch
sudo chown $USER /tmp/fc-orch

To persist across reboots, add to /etc/fstab:

/var/fc-orch.img  /tmp/fc-orch  btrfs  loop  0  0

Step 6 — Enable IP forwarding persistently

The orchestrator enables this at runtime, but it resets on reboot:

echo 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/99-firecracker.conf
sudo sysctl -p /etc/sysctl.d/99-firecracker.conf

Step 7 — Build and run

Option A — cross-compile on your Mac and copy over:

# On macOS:
GOOS=linux GOARCH=amd64 go build -o fc-orch .
scp fc-orch user@your-vm:~/

Option B — build directly on the VM:

git clone <your-repo> && cd firecracker-orchestrator
go build -o fc-orch .

Run:

sudo ./fc-orch init      # download kernel + build Alpine rootfs
sudo ./fc-orch golden    # boot golden VM, snapshot it
sudo ./fc-orch spawn 5   # restore 5 clones
sudo ./fc-orch status    # list running clones
sudo ./fc-orch kill      # terminate all VMs
sudo ./fc-orch cleanup   # full teardown

TAP/bridge/iptables operations require root (or CAP_NET_ADMIN). Running via sudo is the simplest approach.

Quick checklist

Check	Command	Expected
KVM accessible	`ls /dev/kvm`	file exists
In kvm group	`groups \| grep kvm`	`kvm` listed
Go version	`go version`	`go1.26.2`
Firecracker	`firecracker --version`	`1.15.1`
mkfs.ext4	`which mkfs.ext4`	path printed
ip / iptables	`which ip && which iptables`	both paths printed
IP forwarding	`sysctl net.ipv4.ip_forward`	`= 1`
btrfs mount	`df -T /tmp/fc-orch`	type `btrfs`

4.4 KiB Raw Blame History