firecracker-orchestrator/docs/commands.md

fc-orch Command Reference

fc-orch is a Firecracker microVM snapshot orchestrator. It creates a single "golden" VM, snapshots it at a stable boot state, then rapidly restores arbitrarily many clones from that snapshot. Clones share the golden memory file via Linux kernel MAP_PRIVATE copy-on-write, so the incremental cost of each clone is only its dirty pages and a private copy of the VM state file.

All commands require root privileges (sudo).

---

Table of Contents

• Global Configuration
• Directory Layout
• Network Topology
• init
• golden
• spawn
• status
• kill
• cleanup

---

Global Configuration

All tunables are set via environment variables. Every variable has a default; none are required unless you want non-default behavior.

Variable	Default	Description
FC_BIN	firecracker	Path or name of the Firecracker binary (resolved via $PATH)
FC_BASE_DIR	/tmp/fc-orch	Root working directory for all state
FC_KERNEL	$FC_BASE_DIR/vmlinux	Path to the kernel image
FC_KERNEL_URL	Pinned Firecracker CI build (vmlinux-6.1.166)	URL to download the kernel if FC_KERNEL is missing
FC_ROOTFS	$FC_BASE_DIR/rootfs.ext4	Path to the base ext4 rootfs image
FC_VCPUS	1	Number of vCPUs per VM
FC_MEM_MIB	128	Memory per VM in MiB
FC_BRIDGE	fcbr0	Host bridge name. Set to none to disable all networking
FC_BRIDGE_CIDR	172.30.0.1/24	IP address and prefix assigned to the host bridge
FC_GUEST_PREFIX	172.30.0	IP prefix for guest address allocation
FC_GUEST_GW	172.30.0.1	Default gateway advertised to guests

Kernel boot arguments are hardcoded and not user-configurable:

console=ttyS0 reboot=k panic=1 pci=off i8042.noaux quiet loglevel=0

---

Directory Layout

After running all commands, $FC_BASE_DIR (/tmp/fc-orch by default) contains:

/tmp/fc-orch/
├── vmlinux                    # kernel image (shared, immutable)
├── rootfs.ext4                # base Alpine rootfs (shared, immutable)
├── golden/
│   ├── api.sock               # Firecracker API socket (golden VM, transient)
│   ├── rootfs.ext4            # COW copy of base rootfs used by golden VM
│   ├── mem                    # memory snapshot (read by all clones, never written)
│   └── vmstate                # VM state snapshot (golden reference)
├── clones/
│   ├── 1/
│   │   ├── api.sock           # Firecracker API socket (clone 1)
│   │   ├── rootfs.ext4        # private COW copy of golden rootfs
│   │   └── vmstate            # private copy of golden vmstate
│   ├── 2/
│   │   └── ...
│   └── N/
│       └── ...
└── pids/
    ├── golden.pid             # PID of golden Firecracker process (transient)
    ├── clone-1.pid
    ├── clone-2.pid
    └── ...

---

Network Topology

When FC_BRIDGE is not none (the default), a Linux bridge and per-VM TAP devices are used:

  Internet
     │
  Host NIC (e.g. eth0)
     │
  iptables NAT MASQUERADE
     │
  Bridge: fcbr0 (172.30.0.1/24)
     ├── fctap0  (golden VM — exists only during golden command)
     ├── fctap1  (clone 1)
     ├── fctap2  (clone 2)
     └── fctapN  (clone N)

Each clone receives a unique TAP device and MAC address (AA:FC:00:00:XX:XX). IP assignment inside the guest is the guest OS's responsibility (the rootfs init script only brings eth0 up; no DHCP server is included).

Set FC_BRIDGE=none to skip all network configuration. VMs will boot without a network interface.

---

init

Purpose

Downloads the Linux kernel image and builds a minimal Alpine Linux ext4 rootfs. This command only needs to run once; both artifacts are reused by all subsequent golden invocations. init is idempotent — it skips any artifact that already exists on disk.

Usage

sudo ./fc-orch init

Optional overrides:

sudo FC_KERNEL_URL=https://example.com/vmlinux FC_BASE_DIR=/data/fc ./fc-orch init

Prerequisites

• dd, mkfs.ext4 (e2fsprogs), mount/umount (util-linux), tar must be in $PATH
• Internet access to download kernel and Alpine tarball
• Root privileges (required for mount)

Step-by-step execution

1. Create base directory

   mkdir -p /tmp/fc-orch
   

2. Download kernel (skipped if /tmp/fc-orch/vmlinux already exists)

   GET https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/20260408-ce2a467895c1-0/x86_64/vmlinux-6.1.166
   → /tmp/fc-orch/vmlinux
   

3. Create empty ext4 image (skipped if /tmp/fc-orch/rootfs.ext4 already exists)

   dd if=/dev/zero of=/tmp/fc-orch/rootfs.ext4 bs=1M count=512 status=none
   

4. Format as ext4

   mkfs.ext4 -qF /tmp/fc-orch/rootfs.ext4
   

5. Mount the image

   mkdir -p /tmp/fc-orch/mnt
   mount -o loop /tmp/fc-orch/rootfs.ext4 /tmp/fc-orch/mnt
   

6. Download Alpine minirootfs tarball

   GET https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/x86_64/alpine-minirootfs-3.20.0-x86_64.tar.gz
   → /tmp/fc-orch/alpine-minirootfs-3.20.0-x86_64.tar.gz
   

7. Extract Alpine into the mounted image

   tar xzf /tmp/fc-orch/alpine-minirootfs-3.20.0-x86_64.tar.gz -C /tmp/fc-orch/mnt
   

8. Write /etc/init.d/rcS inside the mounted image

   #!/bin/sh
   mount -t proc proc /proc
   mount -t sysfs sys /sys
   mount -t devtmpfs devtmpfs /dev
   ip link set eth0 up 2>/dev/null
   

9. Write /etc/inittab inside the mounted image

   ::sysinit:/etc/init.d/rcS
   ttyS0::respawn:/bin/sh
   

   This causes the guest to launch a shell on the serial console (ttyS0) and respawn it if it exits.

10. Unmount the image

    umount /tmp/fc-orch/mnt
    

Outputs

Path	Description
/tmp/fc-orch/vmlinux	Linux kernel image for Firecracker
/tmp/fc-orch/rootfs.ext4	512 MiB Alpine Linux ext4 image

Error conditions

Error	Cause	Resolution
download kernel: ...	Network failure or bad FC_KERNEL_URL	Check connectivity; verify the URL
download alpine: ...	Network failure downloading Alpine tarball	Check connectivity
build rootfs: ...	dd, mkfs.ext4, mount, or tar failed	Ensure the tools are installed and you are running as root

---

golden

Purpose

Boots a fresh Firecracker VM from the base artifacts produced by init, waits 3 seconds for the guest to finish its init sequence, pauses the VM, and takes a snapshot of the entire machine state (memory + VM state). The snapshot artifacts are the input to every spawn invocation. The golden VM process is terminated after snapshotting — only the artifacts on disk are kept.

This command always recreates the golden directory from scratch, discarding any previous snapshot.

Usage

sudo ./fc-orch golden

Optional overrides:

sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden

Prerequisites

• init must have been run (kernel and rootfs must exist)
• firecracker binary must be in $PATH (or set via FC_BIN)
• ip, iptables, sysctl must be in $PATH (when networking is enabled)

Step-by-step execution

1. Verify prerequisites

   Checks that FC_KERNEL and FC_ROOTFS exist. Exits with an error if either is missing and directs the user to run init.

2. Recreate golden directory

   rm -rf /tmp/fc-orch/golden
   mkdir -p /tmp/fc-orch/golden /tmp/fc-orch/pids
   

3. COW copy of base rootfs

   cp --reflink=always /tmp/fc-orch/rootfs.ext4 /tmp/fc-orch/golden/rootfs.ext4
   

   On filesystems that do not support reflinks (e.g. ext4), this falls back to a regular byte-for-byte copy via io.Copy. On btrfs or xfs, the reflink is instant and consumes no additional space until the VM writes to the disk.

4. Network setup (skipped when FC_BRIDGE=none)

   a. Create bridge (idempotent — skipped if fcbr0 already exists):

   ip link add fcbr0 type bridge
   ip addr add 172.30.0.1/24 dev fcbr0
   ip link set fcbr0 up
   

   b. Enable IP forwarding and NAT:

   ip -4 route show default          # detect egress interface, e.g. "eth0"
   sysctl -qw net.ipv4.ip_forward=1
   iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
   

   c. Create and attach the golden TAP device:

   ip tuntap add dev fctap0 mode tap
   ip link set fctap0 up
   ip link set fctap0 master fcbr0
   

5. Build Firecracker machine configuration (passed to the SDK in memory):

   SocketPath:      /tmp/fc-orch/golden/api.sock
   KernelImagePath: /tmp/fc-orch/vmlinux
   KernelArgs:      console=ttyS0 reboot=k panic=1 pci=off i8042.noaux quiet loglevel=0
   MachineCfg:
     VcpuCount:       1
     MemSizeMib:      128
     TrackDirtyPages: true      ← required for snapshot support
   Drives:
     - DriveID:      rootfs
       PathOnHost:   /tmp/fc-orch/golden/rootfs.ext4
       IsRootDevice: true
       IsReadOnly:   false
   NetworkInterfaces:
     - MacAddress:  AA:FC:00:00:00:01
       HostDevName: fctap0
   

6. Launch Firecracker process

   The Firecracker Go SDK spawns:

   firecracker --api-sock /tmp/fc-orch/golden/api.sock
   

   The SDK then applies the machine configuration via HTTP calls to the Firecracker API socket.

7. Boot the VM

   m.Start(ctx)   // SDK call — PUT /actions {"action_type": "InstanceStart"}
   

   The golden VM PID is written to /tmp/fc-orch/pids/golden.pid.

8. Wait for guest init to settle

   time.Sleep(3 * time.Second)
   

   This is a fixed delay. The guest's rcS script mounts pseudo-filesystems and brings up eth0. 3 seconds is conservative enough for the Alpine init sequence to complete.

9. Pause the VM

   m.PauseVM(ctx)   // SDK call — PATCH /vm {"state": "Paused"}
   

   The VM's vCPUs are frozen. No guest code runs after this point.

10. Create snapshot

    m.CreateSnapshot(ctx,
        "/tmp/fc-orch/golden/mem",
        "/tmp/fc-orch/golden/vmstate",
    )
    // SDK call — PUT /snapshot/create
    // {
    //   "mem_file_path":      "/tmp/fc-orch/golden/mem",
    //   "snapshot_path":      "/tmp/fc-orch/golden/vmstate",
    //   "snapshot_type":      "Full"
    // }
    

    • mem: full dump of guest physical memory (~128 MiB). All clones map this file read-only; the kernel's MAP_PRIVATE gives each clone copy-on-write semantics over it.
    • vmstate: serialized vCPU and device state (typically a few hundred KiB). Each clone gets its own copy.

    Sizes of both files are logged.

11. Terminate golden VM

    m.StopVMM()   // SDK call — PUT /actions {"action_type": "SendCtrlAltDel"}
    

12. Destroy golden TAP device

    ip link del fctap0
    

Outputs

Path	Description
/tmp/fc-orch/golden/mem	Full memory snapshot (~FC_MEM_MIB MiB)
/tmp/fc-orch/golden/vmstate	VM state snapshot (vCPU registers, device state)
/tmp/fc-orch/golden/rootfs.ext4	COW copy of base rootfs (not needed after snapshotting, kept for reference)

Error conditions

Error	Cause	Resolution
kernel not found — run init first	FC_KERNEL path does not exist	Run init first
rootfs not found — run init first	FC_ROOTFS path does not exist	Run init first
firecracker binary not found	FC_BIN not in $PATH	Install Firecracker or set FC_BIN
create bridge: ...	ip link add failed	Check if another bridge with the same name exists with incompatible config
start golden VM: ...	Firecracker failed to boot	Check Firecracker logs; verify kernel and rootfs are valid
pause VM: ...	VM did not reach a pauseable state in 3s	Increase settle time in source or investigate guest crash via serial console
create snapshot: ...	Snapshot write failed	Check disk space in FC_BASE_DIR

---

spawn

Purpose

Restores one or more VM clones from the golden snapshot. Each clone is an independent Firecracker process that resumes execution from exactly the paused state captured by golden. Clones differ only in their TAP device, MAC address, rootfs COW layer, and vmstate file; they all map the same golden/mem file.

Clone IDs are auto-incremented: if clones 1–3 already exist, the next spawn 2 creates clones 4 and 5. Spawn can be called multiple times to add more clones incrementally.

Usage

sudo ./fc-orch spawn        # spawn 1 clone (default)
sudo ./fc-orch spawn 10     # spawn 10 clones

Prerequisites

• golden must have been run (golden/vmstate and golden/mem must exist)
• firecracker binary must be available
• Sufficient disk space for per-clone rootfs copies (each is a copy of the ~512 MiB golden rootfs, but reflinks are instant on btrfs/xfs)

Step-by-step execution (per clone)

The following steps are performed once for each requested clone. Let {id} be the auto-assigned clone number (e.g. 1, 2, ...).

1. Verify golden artifacts exist

   Checks for both /tmp/fc-orch/golden/vmstate and /tmp/fc-orch/golden/mem. Exits with an error if either is missing.

2. Create directories

   mkdir -p /tmp/fc-orch/clones /tmp/fc-orch/pids
   mkdir -p /tmp/fc-orch/clones/{id}
   

3. Setup bridge (idempotent, skipped if bridge already exists or FC_BRIDGE=none)

   Same sequence as step 4 of golden. No-op if fcbr0 is already up.

4. COW copy of golden rootfs

   cp --reflink=always /tmp/fc-orch/golden/rootfs.ext4 /tmp/fc-orch/clones/{id}/rootfs.ext4
   

   Falls back to a full copy if reflinks are unsupported.

5. Shared memory reference (no copy)

   The clone's Firecracker config will point directly at /tmp/fc-orch/golden/mem. No file operation is needed here — the kernel's MAP_PRIVATE ensures each clone's writes are private.

6. Copy vmstate

   # implemented as io.Copy in Go
   cp /tmp/fc-orch/golden/vmstate /tmp/fc-orch/clones/{id}/vmstate
   

   The vmstate file is small (typically < 1 MiB), so a full copy is cheap.

7. Create and attach TAP device (skipped when FC_BRIDGE=none)

   ip tuntap add dev fctap{id} mode tap
   ip link set fctap{id} up
   ip link set fctap{id} master fcbr0
   

   MAC address is derived from the clone ID:

   AA:FC:00:00:{id/256 in hex}:{id%256 in hex}
   # e.g. clone 1  → AA:FC:00:00:00:01
   #      clone 2  → AA:FC:00:00:00:02
   #      clone 256 → AA:FC:00:00:01:00
   

8. Build Firecracker snapshot-restore configuration (in memory):

   SocketPath: /tmp/fc-orch/clones/{id}/api.sock
   MachineCfg:
     VcpuCount:  1
     MemSizeMib: 128
   NetworkInterfaces:
     - MacAddress:  AA:FC:00:00:00:{id:02X}
       HostDevName: fctap{id}
   Snapshot:
     MemFilePath:  /tmp/fc-orch/golden/mem      ← shared, read-only mapping
     SnapshotPath: /tmp/fc-orch/clones/{id}/vmstate
     ResumeVM:     true                          ← restore instead of fresh boot
   

   Note: KernelImagePath and Drives are omitted when restoring from a snapshot — Firecracker uses the snapshot state instead.

9. Launch Firecracker process

   firecracker --api-sock /tmp/fc-orch/clones/{id}/api.sock
   

10. Restore and resume VM

    m.Start(ctx)
    // SDK call — POST /snapshot/load
    // {
    //   "mem_file_path":      "/tmp/fc-orch/golden/mem",
    //   "snapshot_path":      "/tmp/fc-orch/clones/{id}/vmstate",
    //   "resume_vm":          true
    // }
    

    Restoration time (from m.Start call to return) is measured and logged.

11. Record PID

    echo {pid} > /tmp/fc-orch/pids/clone-{id}.pid
    

12. Register clone in memory

    The running clone is tracked in an in-process map keyed by clone ID, holding the Firecracker SDK handle, context cancel function, and TAP device name. This allows kill to cleanly terminate clones started in the same process invocation.

After all clones are spawned, Status() is called automatically to display the running clone table.

Outputs

For each clone {id}:

Path	Description
/tmp/fc-orch/clones/{id}/rootfs.ext4	Private COW copy of the golden rootfs
/tmp/fc-orch/clones/{id}/vmstate	Private copy of golden vmstate
/tmp/fc-orch/clones/{id}/api.sock	Firecracker API socket (live while clone is running)
/tmp/fc-orch/pids/clone-{id}.pid	PID of this clone's Firecracker process
fctap{id}	Host TAP network device attached to fcbr0

Error conditions

Error	Cause	Resolution
golden vmstate not found — run golden first	golden has not been run	Run golden first
golden mem not found — run golden first	Same as above	Run golden first
firecracker not found	Binary missing	Install Firecracker or set FC_BIN
copy rootfs: ...	Disk full or source missing	Check disk space; re-run golden
restore clone {id}: ...	Firecracker failed to load snapshot	Check that golden/mem is not corrupted; re-run golden
create tap {name}: ...	TAP device already exists or ip failed	Run kill to clean up stale TAPs, then retry
Individual clone failure	Per-clone errors are logged but do not abort the batch	Check logs; surviving clones continue running

---

status

Purpose

Displays a table of all clones that have been spawned, along with their PIDs and liveness. Liveness is determined by checking whether /proc/{pid} exists — a process that has exited will no longer appear in /proc.

This command does not require the clones to have been started in the current process invocation; it reads PID files written to disk by spawn.

Usage

sudo ./fc-orch status

Prerequisites

None. Can be run at any time, even with no clones running.

Step-by-step execution

1. Read PID directory

   Lists all files in /tmp/fc-orch/pids/.

2. Filter for clone PID files

   Only files whose names start with clone- are considered (excludes golden.pid).

3. Check liveness

   For each file:

   # read pid from clone-{id}.pid
   test -d /proc/{pid}   # alive if directory exists
   

4. Print table

   === Running clones ===
     clone-1       pid=12345     alive
     clone-2       pid=12346     alive
     clone-3       pid=12347     DEAD
   

Outputs

Prints to stdout only. No files are modified.

---

kill

Purpose

Terminates all running Firecracker VM processes and removes all TAP devices. Handles two cases:

• In-memory clones: clones started in the same fc-orch process invocation, tracked via the in-process clone map.
• Orphaned clones: clones from a previous fc-orch spawn invocation, whose PIDs are recorded in PID files.

Both cases are always handled, so kill works correctly even after a restart or crash of the orchestrator process.

The command does not remove snapshot files or the clones/ directory — use cleanup for full teardown.

Usage

sudo ./fc-orch kill

Prerequisites

None. Safe to run even if no VMs are running.

Step-by-step execution

1. Stop in-memory clones (clones started in this process invocation)

   For each clone tracked in the in-process map:

   clone.Machine.StopVMM()   // SDK: PUT /actions {"action_type": "SendCtrlAltDel"}
   clone.Cancel()            // cancels the clone's context
   

   ip link del fctap{id}
   

   The clone is removed from the in-memory map.

2. Kill orphaned processes from PID files

   For each file in /tmp/fc-orch/pids/:

   // equivalent to:
   kill -9 {pid}
   

   The PID file is then deleted:

   rm /tmp/fc-orch/pids/{file}
   

3. Destroy stale TAP devices

   ip -o link show
   

   Each line containing fctap is parsed to extract the device name, then:

   ip link del {tapname}
   

   This cleans up any TAP devices left over from previous runs that were not removed by steps 1 or 2.

Outputs

All Firecracker processes are terminated. All fctap* network interfaces are removed. PID files are deleted. The clones/, golden/, and pids/ directories themselves remain on disk.

Error conditions

Errors from individual StopVMM, kill, or ip link del calls are logged but do not abort the rest of the kill sequence. The command always attempts to clean up everything it finds.

---

cleanup

Purpose

Performs a full teardown: kills all VMs (same as kill), removes all working directories under $FC_BASE_DIR, and tears down the host bridge. After cleanup, the system is in the same state as before golden was run. The base kernel and rootfs files are not removed — only golden/, clones/, and pids/ are deleted.

Usage

sudo ./fc-orch cleanup

Prerequisites

None. Safe to run at any time.

Step-by-step execution

1. Kill all VMs

   Performs the full kill sequence (see kill above).

2. Remove working directories

   rm -rf /tmp/fc-orch/clones
   rm -rf /tmp/fc-orch/golden
   rm -rf /tmp/fc-orch/pids
   

3. Tear down bridge (skipped when FC_BRIDGE=none)

   ip link del fcbr0
   

   This also implicitly removes all addresses and routes associated with the bridge. The iptables NAT rule added during golden/spawn is not removed automatically — remove it manually if needed:

   iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
   

Outputs

After cleanup:

• All Firecracker processes are terminated
• /tmp/fc-orch/clones/, /tmp/fc-orch/golden/, /tmp/fc-orch/pids/ are deleted
• fcbr0 bridge and all associated fctap* devices are removed
• /tmp/fc-orch/vmlinux and /tmp/fc-orch/rootfs.ext4 remain intact

To also remove the kernel and rootfs:

sudo ./fc-orch cleanup
rm -f /tmp/fc-orch/vmlinux /tmp/fc-orch/rootfs.ext4

Error conditions

Errors from rm -rf or ip link del are ignored (logged only). Cleanup always completes all steps.

---

Typical Workflow

# One-time setup: download kernel and build rootfs
sudo ./fc-orch init

# Create golden snapshot (re-run any time you want a fresh baseline)
sudo ./fc-orch golden

# Spawn clones
sudo ./fc-orch spawn 10

# Check what's running
sudo ./fc-orch status

# Add more clones to the existing set
sudo ./fc-orch spawn 5

# Tear down all clones (keeps golden snapshot)
sudo ./fc-orch kill

# Full reset (keeps kernel and rootfs)
sudo ./fc-orch cleanup