kacerr/firecracker-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: initial Firecracker snapshot orchestrator (fc-orch)

Browse Source 
A "poor man's" Firecracker VM orchestrator that boots a single golden VM,
snapshots it, then restores N clone VMs from that snapshot with minimal
per-clone overhead.

How it works:
- `init`   — downloads a Linux 6.1 kernel and builds a minimal Alpine 3.20
             rootfs (512 MiB ext4) with a basic init script
- `golden` — boots the golden VM, lets it settle, then pauses and snapshots
             it (vmstate + memory file); the golden VMM is then terminated
             since only the artifacts are needed
- `spawn N` — restores N clone VMs concurrently from the golden snapshot:
               * rootfs: filesystem-level COW copy via `cp --reflink` (falls
                 back to a plain copy if reflinks are not supported)
               * memory: shared golden `mem` file; Firecracker's MAP_PRIVATE
                 lets the kernel handle COW page-by-page at no up-front cost
               * vmstate: small file, cheap regular copy per clone
               * networking: per-clone TAP device (fctapN) bridged to fcbr0
                 with iptables MASQUERADE NAT on the default route interface
- `status`  — reads PID files and checks /proc to report alive/dead clones
- `kill`    — stops in-memory clones, kills any stragglers via PID files,
              and tears down all fctap* devices
- `cleanup` — kill + remove all state dirs and the bridge

All tunables (binary path, base dir, kernel/rootfs paths, vCPUs, memory,
bridge name/CIDR) are configurable via environment variables.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Jan Novak
2026-04-09 09:53:16 +02:00
commit 640bf5065a
8 changed files with 2061 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

161
dev-vm-guide.md Normal file
View File

@@ -0,0 +1,161 @@
# Dev VM Setup Guide
Ubuntu 24 setup for running `firecracker-orchestrator`.
## Prerequisites on your hypervisor
Before creating the VM, **enable nested virtualization** (expose KVM to the guest):
- **VMware Fusion**: VM Settings → Processors & Memory → Advanced → enable "Enable hypervisor applications in this virtual machine"
- **Parallels**: VM Config → CPU & Memory → Advanced → enable "Enable nested virtualization"
- **UTM / QEMU**: Add `-cpu host` flag or enable "Force multi-core" + "Enable Hypervisor"
> The VM must be **x86_64** — the orchestrator downloads an x86_64 kernel and Alpine rootfs at init time.
---
## Step 1 — Verify KVM is accessible
```bash
ls -la /dev/kvm
# Expected: crw-rw---- 1 root kvm ...
```
If `/dev/kvm` doesn't exist, nested virtualization isn't enabled on your hypervisor — go back and fix that first.
Add your user to the `kvm` group so firecracker doesn't require root:
```bash
sudo usermod -aG kvm $USER
# Log out and back in, then verify:
groups | grep kvm
```
---
## Step 2 — Install system dependencies
```bash
sudo apt update && sudo apt install -y \
e2fsprogs \
iproute2 \
iptables \
tar \
wget \
curl \
btrfs-progs
```
| Package | Used for |
|---|---|
| `e2fsprogs` | `mkfs.ext4` — formats the Alpine rootfs image |
| `iproute2` | `ip` — manages TAP devices and bridge |
| `iptables` | NAT masquerade for VM networking |
| `tar` | Extracts the Alpine minirootfs tarball |
| `btrfs-progs` | `mkfs.btrfs` — enables COW reflink copies (see Step 5) |
---
## Step 3 — Install Go 1.26.2
```bash
wget https://go.dev/dl/go1.26.2.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.26.2.linux-amd64.tar.gz
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
source ~/.bashrc
go version
# Expected: go version go1.26.2 linux/amd64
```
---
## Step 4 — Install Firecracker 1.15.1
```bash
FIRECRACKER_VERSION=1.15.1
wget https://github.com/firecracker-microvm/firecracker/releases/download/v${FIRECRACKER_VERSION}/firecracker-v${FIRECRACKER_VERSION}-x86_64.tgz
tar -xzf firecracker-v${FIRECRACKER_VERSION}-x86_64.tgz
sudo mv release-v${FIRECRACKER_VERSION}-x86_64/firecracker-v${FIRECRACKER_VERSION}-x86_64 /usr/local/bin/firecracker
sudo chmod +x /usr/local/bin/firecracker
firecracker --version
```
---
## Step 5 — Set up btrfs working directory
The orchestrator uses `cp --reflink=always` for COW rootfs copies when spawning clones. This only works on **btrfs** or **xfs** — on the default ext4 it silently falls back to a full file copy, which wastes disk I/O and space.
Create a btrfs volume mounted at the orchestrator's working directory:
```bash
sudo dd if=/dev/zero of=/var/fc-orch.img bs=1M count=8192 # 8 GB
sudo mkfs.btrfs /var/fc-orch.img
sudo mkdir -p /tmp/fc-orch
sudo mount -o loop /var/fc-orch.img /tmp/fc-orch
sudo chown $USER /tmp/fc-orch
```
To persist across reboots, add to `/etc/fstab`:
```
/var/fc-orch.img /tmp/fc-orch btrfs loop 0 0
```
---
## Step 6 — Enable IP forwarding persistently
The orchestrator enables this at runtime, but it resets on reboot:
```bash
echo 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/99-firecracker.conf
sudo sysctl -p /etc/sysctl.d/99-firecracker.conf
```
---
## Step 7 — Build and run
**Option A — cross-compile on your Mac and copy over:**
```bash
# On macOS:
GOOS=linux GOARCH=amd64 go build -o fc-orch .
scp fc-orch user@your-vm:~/
```
**Option B — build directly on the VM:**
```bash
git clone <your-repo> && cd firecracker-orchestrator
go build -o fc-orch .
```
**Run:**
```bash
sudo ./fc-orch init # download kernel + build Alpine rootfs
sudo ./fc-orch golden # boot golden VM, snapshot it
sudo ./fc-orch spawn 5 # restore 5 clones
sudo ./fc-orch status # list running clones
sudo ./fc-orch kill # terminate all VMs
sudo ./fc-orch cleanup # full teardown
```
> TAP/bridge/iptables operations require root (or `CAP_NET_ADMIN`). Running via `sudo` is the simplest approach.
---
## Quick checklist
| Check | Command | Expected |
|---|---|---|
| KVM accessible | `ls /dev/kvm` | file exists |
| In kvm group | `groups \| grep kvm` | `kvm` listed |
| Go version | `go version` | `go1.26.2` |
| Firecracker | `firecracker --version` | `1.15.1` |
| mkfs.ext4 | `which mkfs.ext4` | path printed |
| ip / iptables | `which ip && which iptables` | both paths printed |
| IP forwarding | `sysctl net.ipv4.ip_forward` | `= 1` |
| btrfs mount | `df -T /tmp/fc-orch` | type `btrfs` |