- Add docs/commands.md with per-command purpose, step-by-step shell/SDK
call sequences, config tables, outputs, and error conditions
- Rename module from github.com/you/fc-orchestrator to github.com/kacerr/fc-orchestrator
- Add KernelURL field to Config so the download URL is configurable via
FC_KERNEL_URL instead of being hardcoded in Init()
- Expose FC_KERNEL_URL in the usage string
- Add verbose logging of dd/mkfs.ext4/mount/tar calls in buildRootfs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
`fc-orch` is a Firecracker microVM snapshot orchestrator. It creates a single "golden" VM, snapshots it at a stable boot state, then rapidly restores arbitrarily many clones from that snapshot. Clones share the golden memory file via Linux kernel MAP_PRIVATE copy-on-write, so the incremental cost of each clone is only its dirty pages and a private copy of the VM state file.
All commands require root privileges (`sudo`).
---
## Table of Contents
- [Global Configuration](#global-configuration)
- [Directory Layout](#directory-layout)
- [Network Topology](#network-topology)
- [`init`](#init)
- [`golden`](#golden)
- [`spawn`](#spawn)
- [`status`](#status)
- [`kill`](#kill)
- [`cleanup`](#cleanup)
---
## Global Configuration
All tunables are set via environment variables. Every variable has a default; none are required unless you want non-default behavior.
| Variable | Default | Description |
|---|---|---|
| `FC_BIN` | `firecracker` | Path or name of the Firecracker binary (resolved via `$PATH`) |
| `FC_BASE_DIR` | `/tmp/fc-orch` | Root working directory for all state |
| `FC_KERNEL` | `$FC_BASE_DIR/vmlinux` | Path to the kernel image |
| `FC_KERNEL_URL` | Pinned Firecracker CI build (vmlinux-6.1.166) | URL to download the kernel if `FC_KERNEL` is missing |
| `FC_ROOTFS` | `$FC_BASE_DIR/rootfs.ext4` | Path to the base ext4 rootfs image |
| `FC_VCPUS` | `1` | Number of vCPUs per VM |
| `FC_MEM_MIB` | `128` | Memory per VM in MiB |
| `FC_BRIDGE` | `fcbr0` | Host bridge name. Set to `none` to disable all networking |
| `FC_BRIDGE_CIDR` | `172.30.0.1/24` | IP address and prefix assigned to the host bridge |
| `FC_GUEST_PREFIX` | `172.30.0` | IP prefix for guest address allocation |
After running all commands, `$FC_BASE_DIR` (`/tmp/fc-orch` by default) contains:
```
/tmp/fc-orch/
├── vmlinux # kernel image (shared, immutable)
├── rootfs.ext4 # base Alpine rootfs (shared, immutable)
├── golden/
│ ├── api.sock # Firecracker API socket (golden VM, transient)
│ ├── rootfs.ext4 # COW copy of base rootfs used by golden VM
│ ├── mem # memory snapshot (read by all clones, never written)
│ └── vmstate # VM state snapshot (golden reference)
├── clones/
│ ├── 1/
│ │ ├── api.sock # Firecracker API socket (clone 1)
│ │ ├── rootfs.ext4 # private COW copy of golden rootfs
│ │ └── vmstate # private copy of golden vmstate
│ ├── 2/
│ │ └── ...
│ └── N/
│ └── ...
└── pids/
├── golden.pid # PID of golden Firecracker process (transient)
├── clone-1.pid
├── clone-2.pid
└── ...
```
---
## Network Topology
When `FC_BRIDGE` is not `none` (the default), a Linux bridge and per-VM TAP devices are used:
```
Internet
│
Host NIC (e.g. eth0)
│
iptables NAT MASQUERADE
│
Bridge: fcbr0 (172.30.0.1/24)
├── fctap0 (golden VM — exists only during golden command)
├── fctap1 (clone 1)
├── fctap2 (clone 2)
└── fctapN (clone N)
```
Each clone receives a unique TAP device and MAC address (`AA:FC:00:00:XX:XX`). IP assignment inside the guest is the guest OS's responsibility (the rootfs init script only brings `eth0` up; no DHCP server is included).
Set `FC_BRIDGE=none` to skip all network configuration. VMs will boot without a network interface.
---
## `init`
### Purpose
Downloads the Linux kernel image and builds a minimal Alpine Linux ext4 rootfs. This command only needs to run once; both artifacts are reused by all subsequent `golden` invocations. `init` is idempotent — it skips any artifact that already exists on disk.
| `build rootfs: ...` | `dd`, `mkfs.ext4`, `mount`, or `tar` failed | Ensure the tools are installed and you are running as root |
---
## `golden`
### Purpose
Boots a fresh Firecracker VM from the base artifacts produced by `init`, waits 3 seconds for the guest to finish its init sequence, pauses the VM, and takes a snapshot of the entire machine state (memory + VM state). The snapshot artifacts are the input to every `spawn` invocation. The golden VM process is terminated after snapshotting — only the artifacts on disk are kept.
This command always recreates the golden directory from scratch, discarding any previous snapshot.
### Usage
```sh
sudo ./fc-orch golden
```
Optional overrides:
```sh
sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
```
### Prerequisites
- `init` must have been run (kernel and rootfs must exist)
- `firecracker` binary must be in `$PATH` (or set via `FC_BIN`)
- `ip`, `iptables`, `sysctl` must be in `$PATH` (when networking is enabled)
### Step-by-step execution
1. **Verify prerequisites**
Checks that `FC_KERNEL` and `FC_ROOTFS` exist. Exits with an error if either is missing and directs the user to run `init`.
On filesystems that do not support reflinks (e.g. ext4), this falls back to a regular byte-for-byte copy via `io.Copy`. On btrfs or xfs, the reflink is instant and consumes no additional space until the VM writes to the disk.
4. **Network setup** (skipped when `FC_BRIDGE=none`)
a. Create bridge (idempotent — skipped if `fcbr0` already exists):
```sh
ip link add fcbr0 type bridge
ip addr add 172.30.0.1/24 dev fcbr0
ip link set fcbr0 up
```
b. Enable IP forwarding and NAT:
```sh
ip -4 route show default # detect egress interface, e.g. "eth0"
sysctl -qw net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
```
c. Create and attach the golden TAP device:
```sh
ip tuntap add dev fctap0 mode tap
ip link set fctap0 up
ip link set fctap0 master fcbr0
```
5. **Build Firecracker machine configuration** (passed to the SDK in memory):
The SDK then applies the machine configuration via HTTP calls to the Firecracker API socket.
7. **Boot the VM**
```go
m.Start(ctx) // SDK call — PUT /actions {"action_type": "InstanceStart"}
```
The golden VM PID is written to `/tmp/fc-orch/pids/golden.pid`.
8. **Wait for guest init to settle**
```go
time.Sleep(3 * time.Second)
```
This is a fixed delay. The guest's `rcS` script mounts pseudo-filesystems and brings up `eth0`. 3 seconds is conservative enough for the Alpine init sequence to complete.
- `mem`: full dump of guest physical memory (~128 MiB). All clones map this file read-only; the kernel's MAP_PRIVATE gives each clone copy-on-write semantics over it.
- `vmstate`: serialized vCPU and device state (typically a few hundred KiB). Each clone gets its own copy.
Sizes of both files are logged.
11. **Terminate golden VM**
```go
m.StopVMM() // SDK call — PUT /actions {"action_type": "SendCtrlAltDel"}
```
12. **Destroy golden TAP device**
```sh
ip link del fctap0
```
### Outputs
| Path | Description |
|---|---|
| `/tmp/fc-orch/golden/mem` | Full memory snapshot (~`FC_MEM_MIB` MiB) |
| `/tmp/fc-orch/golden/vmstate` | VM state snapshot (vCPU registers, device state) |
| `/tmp/fc-orch/golden/rootfs.ext4` | COW copy of base rootfs (not needed after snapshotting, kept for reference) |
### Error conditions
| Error | Cause | Resolution |
|---|---|---|
| `kernel not found — run init first` | `FC_KERNEL` path does not exist | Run `init` first |
| `rootfs not found — run init first` | `FC_ROOTFS` path does not exist | Run `init` first |
| `firecracker binary not found` | `FC_BIN` not in `$PATH` | Install Firecracker or set `FC_BIN` |
| `create bridge: ...` | `ip link add` failed | Check if another bridge with the same name exists with incompatible config |
| `start golden VM: ...` | Firecracker failed to boot | Check Firecracker logs; verify kernel and rootfs are valid |
| `pause VM: ...` | VM did not reach a pauseable state in 3s | Increase settle time in source or investigate guest crash via serial console |
| `create snapshot: ...` | Snapshot write failed | Check disk space in `FC_BASE_DIR` |
---
## `spawn`
### Purpose
Restores one or more VM clones from the golden snapshot. Each clone is an independent Firecracker process that resumes execution from exactly the paused state captured by `golden`. Clones differ only in their TAP device, MAC address, rootfs COW layer, and vmstate file; they all map the same `golden/mem` file.
Clone IDs are auto-incremented: if clones 1–3 already exist, the next `spawn 2` creates clones 4 and 5. Spawn can be called multiple times to add more clones incrementally.
### Usage
```sh
sudo ./fc-orch spawn # spawn 1 clone (default)
sudo ./fc-orch spawn 10 # spawn 10 clones
```
### Prerequisites
- `golden` must have been run (`golden/vmstate` and `golden/mem` must exist)
- `firecracker` binary must be available
- Sufficient disk space for per-clone rootfs copies (each is a copy of the ~512 MiB golden rootfs, but reflinks are instant on btrfs/xfs)
### Step-by-step execution (per clone)
The following steps are performed once for each requested clone. Let `{id}` be the auto-assigned clone number (e.g. `1`, `2`, ...).
1. **Verify golden artifacts exist**
Checks for both `/tmp/fc-orch/golden/vmstate` and `/tmp/fc-orch/golden/mem`. Exits with an error if either is missing.
2. **Create directories**
```sh
mkdir -p /tmp/fc-orch/clones /tmp/fc-orch/pids
mkdir -p /tmp/fc-orch/clones/{id}
```
3. **Setup bridge** (idempotent, skipped if bridge already exists or `FC_BRIDGE=none`)
Same sequence as step 4 of `golden`. No-op if `fcbr0` is already up.
Falls back to a full copy if reflinks are unsupported.
5. **Shared memory reference** (no copy)
The clone's Firecracker config will point directly at `/tmp/fc-orch/golden/mem`. No file operation is needed here — the kernel's MAP_PRIVATE ensures each clone's writes are private.
Restoration time (from `m.Start` call to return) is measured and logged.
11. **Record PID**
```sh
echo {pid} > /tmp/fc-orch/pids/clone-{id}.pid
```
12. **Register clone in memory**
The running clone is tracked in an in-process map keyed by clone ID, holding the Firecracker SDK handle, context cancel function, and TAP device name. This allows `kill` to cleanly terminate clones started in the same process invocation.
After all clones are spawned, `Status()` is called automatically to display the running clone table.
### Outputs
For each clone `{id}`:
| Path | Description |
|---|---|
| `/tmp/fc-orch/clones/{id}/rootfs.ext4` | Private COW copy of the golden rootfs |
| `/tmp/fc-orch/clones/{id}/vmstate` | Private copy of golden vmstate |
| `/tmp/fc-orch/clones/{id}/api.sock` | Firecracker API socket (live while clone is running) |
| `/tmp/fc-orch/pids/clone-{id}.pid` | PID of this clone's Firecracker process |
| `fctap{id}` | Host TAP network device attached to `fcbr0` |
### Error conditions
| Error | Cause | Resolution |
|---|---|---|
| `golden vmstate not found — run golden first` | `golden` has not been run | Run `golden` first |
| `golden mem not found — run golden first` | Same as above | Run `golden` first |
| `firecracker not found` | Binary missing | Install Firecracker or set `FC_BIN` |
| `copy rootfs: ...` | Disk full or source missing | Check disk space; re-run `golden` |
| `restore clone {id}: ...` | Firecracker failed to load snapshot | Check that `golden/mem` is not corrupted; re-run `golden` |
| `create tap {name}: ...` | TAP device already exists or `ip` failed | Run `kill` to clean up stale TAPs, then retry |
| Individual clone failure | Per-clone errors are logged but do not abort the batch | Check logs; surviving clones continue running |
---
## `status`
### Purpose
Displays a table of all clones that have been spawned, along with their PIDs and liveness. Liveness is determined by checking whether `/proc/{pid}` exists — a process that has exited will no longer appear in `/proc`.
This command does not require the clones to have been started in the current process invocation; it reads PID files written to disk by `spawn`.
### Usage
```sh
sudo ./fc-orch status
```
### Prerequisites
None. Can be run at any time, even with no clones running.
### Step-by-step execution
1. **Read PID directory**
Lists all files in `/tmp/fc-orch/pids/`.
2. **Filter for clone PID files**
Only files whose names start with `clone-` are considered (excludes `golden.pid`).
3. **Check liveness**
For each file:
```sh
# read pid from clone-{id}.pid
test -d /proc/{pid} # alive if directory exists
```
4. **Print table**
```
=== Running clones ===
clone-1 pid=12345 alive
clone-2 pid=12346 alive
clone-3 pid=12347 DEAD
```
### Outputs
Prints to stdout only. No files are modified.
---
## `kill`
### Purpose
Terminates all running Firecracker VM processes and removes all TAP devices. Handles two cases:
- **In-memory clones**: clones started in the same `fc-orch` process invocation, tracked via the in-process clone map.
- **Orphaned clones**: clones from a previous `fc-orch spawn` invocation, whose PIDs are recorded in PID files.
Both cases are always handled, so `kill` works correctly even after a restart or crash of the orchestrator process.
The command does **not** remove snapshot files or the `clones/` directory — use `cleanup` for full teardown.
### Usage
```sh
sudo ./fc-orch kill
```
### Prerequisites
None. Safe to run even if no VMs are running.
### Step-by-step execution
1. **Stop in-memory clones** (clones started in this process invocation)
For each clone tracked in the in-process map:
```go
clone.Machine.StopVMM() // SDK: PUT /actions {"action_type": "SendCtrlAltDel"}
clone.Cancel() // cancels the clone's context
```
```sh
ip link del fctap{id}
```
The clone is removed from the in-memory map.
2. **Kill orphaned processes from PID files**
For each file in `/tmp/fc-orch/pids/`:
```go
// equivalent to:
kill -9 {pid}
```
The PID file is then deleted:
```sh
rm /tmp/fc-orch/pids/{file}
```
3. **Destroy stale TAP devices**
```sh
ip -o link show
```
Each line containing `fctap` is parsed to extract the device name, then:
```sh
ip link del {tapname}
```
This cleans up any TAP devices left over from previous runs that were not removed by steps 1 or 2.
### Outputs
All Firecracker processes are terminated. All `fctap*` network interfaces are removed. PID files are deleted. The `clones/`, `golden/`, and `pids/` directories themselves remain on disk.
### Error conditions
Errors from individual `StopVMM`, `kill`, or `ip link del` calls are logged but do not abort the rest of the kill sequence. The command always attempts to clean up everything it finds.
---
## `cleanup`
### Purpose
Performs a full teardown: kills all VMs (same as `kill`), removes all working directories under `$FC_BASE_DIR`, and tears down the host bridge. After `cleanup`, the system is in the same state as before `golden` was run. The base kernel and rootfs files are **not** removed — only `golden/`, `clones/`, and `pids/` are deleted.
### Usage
```sh
sudo ./fc-orch cleanup
```
### Prerequisites
None. Safe to run at any time.
### Step-by-step execution
1. **Kill all VMs**
Performs the full `kill` sequence (see [`kill`](#kill) above).
2. **Remove working directories**
```sh
rm -rf /tmp/fc-orch/clones
rm -rf /tmp/fc-orch/golden
rm -rf /tmp/fc-orch/pids
```
3. **Tear down bridge** (skipped when `FC_BRIDGE=none`)
```sh
ip link del fcbr0
```
This also implicitly removes all addresses and routes associated with the bridge. The iptables NAT rule added during `golden`/`spawn` is **not** removed automatically — remove it manually if needed:
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Claude Sonnet 4.6