Jan Novak
640bf5065a
A "poor man's" Firecracker VM orchestrator that boots a single golden VM,
snapshots it, then restores N clone VMs from that snapshot with minimal
per-clone overhead.
How it works:
- `init` — downloads a Linux 6.1 kernel and builds a minimal Alpine 3.20
rootfs (512 MiB ext4) with a basic init script
- `golden` — boots the golden VM, lets it settle, then pauses and snapshots
it (vmstate + memory file); the golden VMM is then terminated
since only the artifacts are needed
- `spawn N` — restores N clone VMs concurrently from the golden snapshot:
* rootfs: filesystem-level COW copy via `cp --reflink` (falls
back to a plain copy if reflinks are not supported)
* memory: shared golden `mem` file; Firecracker's MAP_PRIVATE
lets the kernel handle COW page-by-page at no up-front cost
* vmstate: small file, cheap regular copy per clone
* networking: per-clone TAP device (fctapN) bridged to fcbr0
with iptables MASQUERADE NAT on the default route interface
- `status` — reads PID files and checks /proc to report alive/dead clones
- `kill` — stops in-memory clones, kills any stragglers via PID files,
and tears down all fctap* devices
- `cleanup` — kill + remove all state dirs and the bridge
All tunables (binary path, base dir, kernel/rootfs paths, vCPUs, memory,
bridge name/CIDR) are configurable via environment variables.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
162 lines
4.4 KiB
Markdown
162 lines
4.4 KiB
Markdown
# Dev VM Setup Guide
|
|
|
|
Ubuntu 24 setup for running `firecracker-orchestrator`.
|
|
|
|
## Prerequisites on your hypervisor
|
|
|
|
Before creating the VM, **enable nested virtualization** (expose KVM to the guest):
|
|
|
|
- **VMware Fusion**: VM Settings → Processors & Memory → Advanced → enable "Enable hypervisor applications in this virtual machine"
|
|
- **Parallels**: VM Config → CPU & Memory → Advanced → enable "Enable nested virtualization"
|
|
- **UTM / QEMU**: Add `-cpu host` flag or enable "Force multi-core" + "Enable Hypervisor"
|
|
|
|
> The VM must be **x86_64** — the orchestrator downloads an x86_64 kernel and Alpine rootfs at init time.
|
|
|
|
---
|
|
|
|
## Step 1 — Verify KVM is accessible
|
|
|
|
```bash
|
|
ls -la /dev/kvm
|
|
# Expected: crw-rw---- 1 root kvm ...
|
|
```
|
|
|
|
If `/dev/kvm` doesn't exist, nested virtualization isn't enabled on your hypervisor — go back and fix that first.
|
|
|
|
Add your user to the `kvm` group so firecracker doesn't require root:
|
|
|
|
```bash
|
|
sudo usermod -aG kvm $USER
|
|
# Log out and back in, then verify:
|
|
groups | grep kvm
|
|
```
|
|
|
|
---
|
|
|
|
## Step 2 — Install system dependencies
|
|
|
|
```bash
|
|
sudo apt update && sudo apt install -y \
|
|
e2fsprogs \
|
|
iproute2 \
|
|
iptables \
|
|
tar \
|
|
wget \
|
|
curl \
|
|
btrfs-progs
|
|
```
|
|
|
|
| Package | Used for |
|
|
|---|---|
|
|
| `e2fsprogs` | `mkfs.ext4` — formats the Alpine rootfs image |
|
|
| `iproute2` | `ip` — manages TAP devices and bridge |
|
|
| `iptables` | NAT masquerade for VM networking |
|
|
| `tar` | Extracts the Alpine minirootfs tarball |
|
|
| `btrfs-progs` | `mkfs.btrfs` — enables COW reflink copies (see Step 5) |
|
|
|
|
---
|
|
|
|
## Step 3 — Install Go 1.26.2
|
|
|
|
```bash
|
|
wget https://go.dev/dl/go1.26.2.linux-amd64.tar.gz
|
|
sudo tar -C /usr/local -xzf go1.26.2.linux-amd64.tar.gz
|
|
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
|
|
source ~/.bashrc
|
|
go version
|
|
# Expected: go version go1.26.2 linux/amd64
|
|
```
|
|
|
|
---
|
|
|
|
## Step 4 — Install Firecracker 1.15.1
|
|
|
|
```bash
|
|
FIRECRACKER_VERSION=1.15.1
|
|
wget https://github.com/firecracker-microvm/firecracker/releases/download/v${FIRECRACKER_VERSION}/firecracker-v${FIRECRACKER_VERSION}-x86_64.tgz
|
|
tar -xzf firecracker-v${FIRECRACKER_VERSION}-x86_64.tgz
|
|
sudo mv release-v${FIRECRACKER_VERSION}-x86_64/firecracker-v${FIRECRACKER_VERSION}-x86_64 /usr/local/bin/firecracker
|
|
sudo chmod +x /usr/local/bin/firecracker
|
|
firecracker --version
|
|
```
|
|
|
|
---
|
|
|
|
## Step 5 — Set up btrfs working directory
|
|
|
|
The orchestrator uses `cp --reflink=always` for COW rootfs copies when spawning clones. This only works on **btrfs** or **xfs** — on the default ext4 it silently falls back to a full file copy, which wastes disk I/O and space.
|
|
|
|
Create a btrfs volume mounted at the orchestrator's working directory:
|
|
|
|
```bash
|
|
sudo dd if=/dev/zero of=/var/fc-orch.img bs=1M count=8192 # 8 GB
|
|
sudo mkfs.btrfs /var/fc-orch.img
|
|
sudo mkdir -p /tmp/fc-orch
|
|
sudo mount -o loop /var/fc-orch.img /tmp/fc-orch
|
|
sudo chown $USER /tmp/fc-orch
|
|
```
|
|
|
|
To persist across reboots, add to `/etc/fstab`:
|
|
|
|
```
|
|
/var/fc-orch.img /tmp/fc-orch btrfs loop 0 0
|
|
```
|
|
|
|
---
|
|
|
|
## Step 6 — Enable IP forwarding persistently
|
|
|
|
The orchestrator enables this at runtime, but it resets on reboot:
|
|
|
|
```bash
|
|
echo 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/99-firecracker.conf
|
|
sudo sysctl -p /etc/sysctl.d/99-firecracker.conf
|
|
```
|
|
|
|
---
|
|
|
|
## Step 7 — Build and run
|
|
|
|
**Option A — cross-compile on your Mac and copy over:**
|
|
|
|
```bash
|
|
# On macOS:
|
|
GOOS=linux GOARCH=amd64 go build -o fc-orch .
|
|
scp fc-orch user@your-vm:~/
|
|
```
|
|
|
|
**Option B — build directly on the VM:**
|
|
|
|
```bash
|
|
git clone <your-repo> && cd firecracker-orchestrator
|
|
go build -o fc-orch .
|
|
```
|
|
|
|
**Run:**
|
|
|
|
```bash
|
|
sudo ./fc-orch init # download kernel + build Alpine rootfs
|
|
sudo ./fc-orch golden # boot golden VM, snapshot it
|
|
sudo ./fc-orch spawn 5 # restore 5 clones
|
|
sudo ./fc-orch status # list running clones
|
|
sudo ./fc-orch kill # terminate all VMs
|
|
sudo ./fc-orch cleanup # full teardown
|
|
```
|
|
|
|
> TAP/bridge/iptables operations require root (or `CAP_NET_ADMIN`). Running via `sudo` is the simplest approach.
|
|
|
|
---
|
|
|
|
## Quick checklist
|
|
|
|
| Check | Command | Expected |
|
|
|---|---|---|
|
|
| KVM accessible | `ls /dev/kvm` | file exists |
|
|
| In kvm group | `groups \| grep kvm` | `kvm` listed |
|
|
| Go version | `go version` | `go1.26.2` |
|
|
| Firecracker | `firecracker --version` | `1.15.1` |
|
|
| mkfs.ext4 | `which mkfs.ext4` | path printed |
|
|
| ip / iptables | `which ip && which iptables` | both paths printed |
|
|
| IP forwarding | `sysctl net.ipv4.ip_forward` | `= 1` |
|
|
| btrfs mount | `df -T /tmp/fc-orch` | type `btrfs` |
|