kacerr/firecracker-orchestrator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: multi-distro support and tagged golden snapshots

Browse Source 
Add Alpine, Debian, and Ubuntu rootfs support to `init [distro]`.
Golden snapshots are now namespaced under `golden/<tag>/` so multiple
baselines can coexist. `spawn [tag] [N]` selects which snapshot to
clone from. Systemd-based distros (Debian, Ubuntu) get a fc-net-init
systemd unit; Alpine keeps its inittab-based init.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Honza Novak
2026-04-14 20:48:43 +00:00
parent bfc1f47287
commit fb1db7c9ea
10 changed files with 576 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
docs/commands.md
View File

@@ -56,10 +56,12 @@ After running all commands, `$FC_BASE_DIR` (`/tmp/fc-orch` by default) contains:
├── vmlinux # kernel image (shared, immutable)
├── rootfs.ext4 # base Alpine rootfs (shared, immutable)
├── golden/
│ ├── api.sock # Firecracker API socket (golden VM, transient)
│ ├── rootfs.ext4 # COW copy of base rootfs used by golden VM
│ ├── mem # memory snapshot (read by all clones, never written)
│ └── vmstate # VM state snapshot (golden reference)
│ ├── default/ # "default" tag directory
│ │ ├── api.sock # Firecracker API socket (golden VM, transient)
│ │ ├── rootfs.ext4 # COW copy of base rootfs used by golden VM
│ │ ├── mem # memory snapshot (read by all clones, never written)
│ │ └── vmstate # VM state snapshot (golden reference)
│ └── <tag>/ # other tagged snapshots
├── clones/
│ ├── 1/
│ │ ├── api.sock # Firecracker API socket (clone 1)
@@ -145,13 +147,14 @@ Within ~1–2 seconds of clone start, `eth0` inside the VM will have the assigne
### Purpose
Downloads the Linux kernel image and builds a minimal Alpine Linux ext4 rootfs. This command only needs to run once; both artifacts are reused by all subsequent `golden` invocations. `init` is idempotent — it skips any artifact that already exists on disk.
Downloads the Linux kernel image and builds a minimal filesystem (Alpine, Debian, or Ubuntu). This command only needs to run once per distro; both artifacts are reused by `golden` invocations. `init` is idempotent — it skips any artifact that already exists on disk.
### Usage
```sh
sudo ./fc-orch init
sudo ./fc-orch init [distro]
```
Where `[distro]` can be `alpine` (default), `debian`, or `ubuntu`.
Optional overrides:
@@ -265,13 +268,15 @@ This command always recreates the golden directory from scratch, discarding any
### Usage
```sh
sudo ./fc-orch golden
sudo ./fc-orch golden [tag] [distro]
```
Where `[tag]` identifies the snapshot baseline name (default `default`), and `[distro]` dictates the source `.ext4` image to use (default: `alpine`).
Optional overrides:
```sh
sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden ubuntu
```
### Prerequisites
@@ -289,14 +294,14 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
2. **Recreate golden directory**
```sh
rm -rf /tmp/fc-orch/golden
mkdir -p /tmp/fc-orch/golden /tmp/fc-orch/pids
rm -rf /tmp/fc-orch/golden/<tag>
mkdir -p /tmp/fc-orch/golden/<tag> /tmp/fc-orch/pids
```
3. **COW copy of base rootfs**
```sh
cp --reflink=always /tmp/fc-orch/rootfs.ext4 /tmp/fc-orch/golden/rootfs.ext4
cp --reflink=always /tmp/fc-orch/rootfs.ext4 /tmp/fc-orch/golden/<tag>/rootfs.ext4
```
On filesystems that do not support reflinks (e.g. ext4), this falls back to a regular byte-for-byte copy via `io.Copy`. On btrfs or xfs, the reflink is instant and consumes no additional space until the VM writes to the disk.
@@ -330,7 +335,7 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
5. **Build Firecracker machine configuration** (passed to the SDK in memory):
```
SocketPath: /tmp/fc-orch/golden/api.sock
SocketPath: /tmp/fc-orch/golden/<tag>/api.sock
KernelImagePath: /tmp/fc-orch/vmlinux
KernelArgs: console=ttyS0 reboot=k panic=1 pci=off i8042.noaux quiet loglevel=0
MachineCfg:
@@ -339,7 +344,7 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
TrackDirtyPages: true ← required for snapshot support
Drives:
- DriveID: rootfs
PathOnHost: /tmp/fc-orch/golden/rootfs.ext4
PathOnHost: /tmp/fc-orch/golden/<tag>/rootfs.ext4
IsRootDevice: true
IsReadOnly: false
NetworkInterfaces:
@@ -352,7 +357,7 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
The Firecracker Go SDK spawns:
```sh
firecracker --api-sock /tmp/fc-orch/golden/api.sock
firecracker --api-sock /tmp/fc-orch/golden/<tag>/api.sock
```
The SDK then applies the machine configuration via HTTP calls to the Firecracker API socket.
@@ -385,13 +390,13 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
```go
m.CreateSnapshot(ctx,
"/tmp/fc-orch/golden/mem",
"/tmp/fc-orch/golden/vmstate",
"/tmp/fc-orch/golden/<tag>/mem",
"/tmp/fc-orch/golden/<tag>/vmstate",
)
// SDK call — PUT /snapshot/create
// {
// "mem_file_path": "/tmp/fc-orch/golden/mem",
// "snapshot_path": "/tmp/fc-orch/golden/vmstate",
// "mem_file_path": "/tmp/fc-orch/golden/<tag>/mem",
// "snapshot_path": "/tmp/fc-orch/golden/<tag>/vmstate",
// "snapshot_type": "Full"
// }
```
@@ -417,16 +422,16 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
| Path | Description |
|---|---|
| `/tmp/fc-orch/golden/mem` | Full memory snapshot (~`FC_MEM_MIB` MiB) |
| `/tmp/fc-orch/golden/vmstate` | VM state snapshot (vCPU registers, device state) |
| `/tmp/fc-orch/golden/rootfs.ext4` | COW copy of base rootfs (not needed after snapshotting, kept for reference) |
| `/tmp/fc-orch/golden/<tag>/mem` | Full memory snapshot (~`FC_MEM_MIB` MiB) |
| `/tmp/fc-orch/golden/<tag>/vmstate` | VM state snapshot (vCPU registers, device state) |
| `/tmp/fc-orch/golden/<tag>/rootfs.ext4` | COW copy of base rootfs (not needed after snapshotting, kept for reference) |
### Error conditions
| Error | Cause | Resolution |
|---|---|---|
| `kernel not found — run init first` | `FC_KERNEL` path does not exist | Run `init` first |
| `rootfs not found — run init first` | `FC_ROOTFS` path does not exist | Run `init` first |
| `rootfs not found — run init first` | Ext4 file does not exist | Run `init [distro]` first |
| `firecracker binary not found` | `FC_BIN` not in `$PATH` | Install Firecracker or set `FC_BIN` |
| `create bridge: ...` | `ip link add` failed | Check if another bridge with the same name exists with incompatible config |
| `start golden VM: ...` | Firecracker failed to boot | Check Firecracker logs; verify kernel and rootfs are valid |
@@ -446,8 +451,8 @@ Clone IDs are auto-incremented: if clones 1–3 already exist, the next `spawn 2
### Usage
```sh
sudo ./fc-orch spawn # spawn 1 clone (default)
sudo ./fc-orch spawn 10 # spawn 10 clones
sudo ./fc-orch spawn # spawn 1 clone from the "default" golden snapshot
sudo ./fc-orch spawn ubuntu 10 # spawn 10 clones from the "ubuntu" golden snapshot
```
### Prerequisites
@@ -462,7 +467,7 @@ The following steps are performed once for each requested clone. Let `{id}` be t
1. **Verify golden artifacts exist**
Checks for both `/tmp/fc-orch/golden/vmstate` and `/tmp/fc-orch/golden/mem`. Exits with an error if either is missing.
Checks for both `/tmp/fc-orch/golden/<tag>/vmstate` and `/tmp/fc-orch/golden/<tag>/mem`. Exits with an error if either is missing.
2. **Create directories**
@@ -478,20 +483,20 @@ The following steps are performed once for each requested clone. Let `{id}` be t
4. **COW copy of golden rootfs**
```sh
cp --reflink=always /tmp/fc-orch/golden/rootfs.ext4 /tmp/fc-orch/clones/{id}/rootfs.ext4
cp --reflink=always /tmp/fc-orch/golden/<tag>/rootfs.ext4 /tmp/fc-orch/clones/{id}/rootfs.ext4
```
Falls back to a full copy if reflinks are unsupported.
5. **Shared memory reference** (no copy)
The clone's Firecracker config will point directly at `/tmp/fc-orch/golden/mem`. No file operation is needed here — the kernel's MAP_PRIVATE ensures each clone's writes are private.
The clone's Firecracker config will point directly at `/tmp/fc-orch/golden/<tag>/mem`. No file operation is needed here — the kernel's MAP_PRIVATE ensures each clone's writes are private.
6. **Copy vmstate**
```sh
# implemented as io.Copy in Go
cp /tmp/fc-orch/golden/vmstate /tmp/fc-orch/clones/{id}/vmstate
cp /tmp/fc-orch/golden/<tag>/vmstate /tmp/fc-orch/clones/{id}/vmstate
```
The vmstate file is small (typically < 1 MiB), so a full copy is cheap.
@@ -524,7 +529,7 @@ The following steps are performed once for each requested clone. Let `{id}` be t
- MacAddress: AA:FC:00:00:00:{id:02X}
HostDevName: fctap{id}
Snapshot:
MemFilePath: /tmp/fc-orch/golden/mem ← shared, read-only mapping
MemFilePath: /tmp/fc-orch/golden/<tag>/mem ← shared, read-only mapping
SnapshotPath: /tmp/fc-orch/clones/{id}/vmstate
ResumeVM: true ← restore instead of fresh boot
```
@@ -543,7 +548,7 @@ The following steps are performed once for each requested clone. Let `{id}` be t
m.Start(ctx)
// SDK call — POST /snapshot/load
// {
// "mem_file_path": "/tmp/fc-orch/golden/mem",
// "mem_file_path": "/tmp/fc-orch/golden/<tag>/mem",
// "snapshot_path": "/tmp/fc-orch/clones/{id}/vmstate",
// "resume_vm": true
// }