firecracker-orchestrator/docs/commands.md

# fc-orch Command Reference

`fc-orch` is a Firecracker microVM snapshot orchestrator. It creates a single "golden" VM, snapshots it at a stable boot state, then rapidly restores arbitrarily many clones from that snapshot. Clones share the golden memory file via Linux kernel MAP_PRIVATE copy-on-write, so the incremental cost of each clone is only its dirty pages and a private copy of the VM state file.

All commands require root privileges (`sudo`).

---

## Table of Contents

- [Global Configuration](#global-configuration)
- [Directory Layout](#directory-layout)
- [Network Topology](#network-topology)
- [`init`](#init)
- [`golden`](#golden)
- [`spawn`](#spawn)
- [`status`](#status)
- [`kill`](#kill)
- [`cleanup`](#cleanup)

---

## Global Configuration

All tunables are set via environment variables. Every variable has a default; none are required unless you want non-default behavior.

| Variable | Default | Description |
|---|---|---|
| `FC_BIN` | `firecracker` | Path or name of the Firecracker binary (resolved via `$PATH`) |
| `FC_BASE_DIR` | `/tmp/fc-orch` | Root working directory for all state |
| `FC_KERNEL` | `$FC_BASE_DIR/vmlinux` | Path to the kernel image |
| `FC_KERNEL_URL` | Pinned Firecracker CI build (vmlinux-6.1.166) | URL to download the kernel if `FC_KERNEL` is missing |
| `FC_ROOTFS` | `$FC_BASE_DIR/rootfs.ext4` | Path to the base ext4 rootfs image |
| `FC_VCPUS` | `1` | Number of vCPUs per VM |
| `FC_MEM_MIB` | `128` | Memory per VM in MiB |
| `FC_BRIDGE` | `fcbr0` | Host bridge name. Set to `none` to disable all networking |
| `FC_BRIDGE_CIDR` | `172.30.0.1/24` | IP address and prefix assigned to the host bridge |
| `FC_GUEST_PREFIX` | `172.30.0` | IP prefix for guest address allocation (used with `FC_AUTO_NET_CONFIG`) |
| `FC_GUEST_GW` | `172.30.0.1` | Default gateway advertised to guests (used with `FC_AUTO_NET_CONFIG`) |
| `FC_AUTO_NET_CONFIG` | _(unset)_ | Set to `1` to automatically assign guest IPs via MMDS on clone start |

Kernel boot arguments are hardcoded and not user-configurable:

```
console=ttyS0 reboot=k panic=1 pci=off i8042.noaux quiet loglevel=0
```

---

## Directory Layout

After running all commands, `$FC_BASE_DIR` (`/tmp/fc-orch` by default) contains:

```
/tmp/fc-orch/
├── vmlinux                    # kernel image (shared, immutable)
├── rootfs.ext4                # base Alpine rootfs (shared, immutable)
├── golden/
│   ├── default/               # "default" tag directory
│   │   ├── api.sock           # Firecracker API socket (golden VM, transient)
│   │   ├── rootfs.ext4        # COW copy of base rootfs used by golden VM
│   │   ├── mem                # memory snapshot (read by all clones, never written)
│   │   └── vmstate            # VM state snapshot (golden reference)
│   └── <tag>/                 # other tagged snapshots
├── clones/
│   ├── 1/
│   │   ├── api.sock           # Firecracker API socket (clone 1)
│   │   ├── rootfs.ext4        # private COW copy of golden rootfs
│   │   └── vmstate            # private copy of golden vmstate
│   ├── 2/
│   │   └── ...
│   └── N/
│       └── ...
└── pids/
    ├── golden.pid             # PID of golden Firecracker process (transient)
    ├── clone-1.pid
    ├── clone-2.pid
    └── ...
```

---

## Network Topology

When `FC_BRIDGE` is not `none` (the default), a Linux bridge and per-VM TAP devices are used:

```
  Internet
     │
  Host NIC (e.g. eth0)
     │
  iptables NAT MASQUERADE
     │
  Bridge: fcbr0 (172.30.0.1/24)
     ├── fctap0  (golden VM — exists only during golden command)
     ├── fctap1  (clone 1)
     ├── fctap2  (clone 2)
     └── fctapN  (clone N)
```

Each clone receives a unique TAP device and MAC address (`AA:FC:00:00:XX:XX`). The host-side bridge has NAT masquerading enabled so guests can reach the internet through the host's default route.

Set `FC_BRIDGE=none` to skip all network configuration. VMs will boot without a network interface.

### Guest IP assignment

The rootfs init script brings `eth0` up at the link layer only. Guests have no IP address by default. There are two ways to configure networking inside a VM:

#### Manual configuration (inside the VM console)

```sh
# Pick an unused IP in the bridge subnet — e.g. .11 for clone 1, .12 for clone 2
ip addr add 172.30.0.11/24 dev eth0
ip route add default via 172.30.0.1
echo "nameserver 1.1.1.1" > /etc/resolv.conf
ping 1.1.1.1   # verify
```

Manual config is ephemeral — it is lost when the clone is stopped. Use the automatic option below for persistent configuration.

#### Automatic configuration via MMDS (`FC_AUTO_NET_CONFIG=1`)

When `FC_AUTO_NET_CONFIG=1` is set, the orchestrator uses the Firecracker **Microvm Metadata Service (MMDS)** to inject per-clone network config immediately after the VM starts. A small background daemon embedded in the rootfs (`/sbin/fc-net-init`) polls `169.254.169.254` and applies the config automatically — no manual steps needed.

IPs are assigned deterministically from `FC_GUEST_PREFIX`:

```
clone 1  → 172.30.0.11/24
clone 2  → 172.30.0.12/24
…
clone N  → 172.30.0.(10+N)/24
```

Usage:

```sh
sudo FC_AUTO_NET_CONFIG=1 ./fc-orch start
```

Within ~1–2 seconds of clone start, `eth0` inside the VM will have the assigned IP, default route, and DNS (`1.1.1.1`) configured.

> **Note:** `FC_AUTO_NET_CONFIG` requires `fc-orch init` and `fc-orch golden` to have been run (or re-run) after this feature was added, so that the `fc-net-init` daemon is present in the golden snapshot.

---

## `init`

### Purpose

Downloads the Linux kernel image and builds a minimal filesystem (Alpine, Debian, or Ubuntu). This command only needs to run once per distro; both artifacts are reused by `golden` invocations. `init` is idempotent — it skips any artifact that already exists on disk.

### Usage

```sh
sudo ./fc-orch init [distro]
```
Where `[distro]` can be `alpine` (default), `debian`, or `ubuntu`.

Optional overrides:

```sh
sudo FC_KERNEL_URL=https://example.com/vmlinux FC_BASE_DIR=/data/fc ./fc-orch init
```

### Prerequisites

- `dd`, `mkfs.ext4` (e2fsprogs), `mount`/`umount` (util-linux), `tar` must be in `$PATH`
- Internet access to download kernel and Alpine tarball
- Root privileges (required for `mount`)

### Step-by-step execution

1. **Create base directory**

   ```sh
   mkdir -p /tmp/fc-orch
   ```

2. **Download kernel** (skipped if `/tmp/fc-orch/vmlinux` already exists)

   ```
   GET https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/20260408-ce2a467895c1-0/x86_64/vmlinux-6.1.166
   → /tmp/fc-orch/vmlinux
   ```

3. **Create empty ext4 image** (skipped if `/tmp/fc-orch/rootfs.ext4` already exists)

   ```sh
   dd if=/dev/zero of=/tmp/fc-orch/rootfs.ext4 bs=1M count=512 status=none
   ```

4. **Format as ext4**

   ```sh
   mkfs.ext4 -qF /tmp/fc-orch/rootfs.ext4
   ```

5. **Mount the image**

   ```sh
   mkdir -p /tmp/fc-orch/mnt
   mount -o loop /tmp/fc-orch/rootfs.ext4 /tmp/fc-orch/mnt
   ```

6. **Download Alpine minirootfs tarball**

   ```
   GET https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/x86_64/alpine-minirootfs-3.20.0-x86_64.tar.gz
   → /tmp/fc-orch/alpine-minirootfs-3.20.0-x86_64.tar.gz
   ```

7. **Extract Alpine into the mounted image**

   ```sh
   tar xzf /tmp/fc-orch/alpine-minirootfs-3.20.0-x86_64.tar.gz -C /tmp/fc-orch/mnt
   ```

8. **Write `/etc/init.d/rcS`** inside the mounted image

   ```sh
   #!/bin/sh
   mount -t proc proc /proc
   mount -t sysfs sys /sys
   mount -t devtmpfs devtmpfs /dev
   ip link set eth0 up 2>/dev/null
   ```

9. **Write `/etc/inittab`** inside the mounted image

   ```
   ::sysinit:/etc/init.d/rcS
   ttyS0::respawn:/bin/sh
   ```

   This causes the guest to launch a shell on the serial console (`ttyS0`) and respawn it if it exits.

10. **Unmount the image**

    ```sh
    umount /tmp/fc-orch/mnt
    ```

### Outputs

| Path | Description |
|---|---|
| `/tmp/fc-orch/vmlinux` | Linux kernel image for Firecracker |
| `/tmp/fc-orch/rootfs.ext4` | 512 MiB Alpine Linux ext4 image |

### Error conditions

| Error | Cause | Resolution |
|---|---|---|
| `download kernel: ...` | Network failure or bad `FC_KERNEL_URL` | Check connectivity; verify the URL |
| `download alpine: ...` | Network failure downloading Alpine tarball | Check connectivity |
| `build rootfs: ...` | `dd`, `mkfs.ext4`, `mount`, or `tar` failed | Ensure the tools are installed and you are running as root |

---

## `golden`

### Purpose

Boots a fresh Firecracker VM from the base artifacts produced by `init`, waits 3 seconds for the guest to finish its init sequence, pauses the VM, and takes a snapshot of the entire machine state (memory + VM state). The snapshot artifacts are the input to every `spawn` invocation. The golden VM process is terminated after snapshotting — only the artifacts on disk are kept.

This command always recreates the golden directory from scratch, discarding any previous snapshot.

### Usage

```sh
sudo ./fc-orch golden [tag] [distro]
```

Where `[tag]` identifies the snapshot baseline name (default `default`), and `[distro]` dictates the source `.ext4` image to use (default: `alpine`).

Optional overrides:

```sh
sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden ubuntu
```

### Prerequisites

- `init` must have been run (kernel and rootfs must exist)
- `firecracker` binary must be in `$PATH` (or set via `FC_BIN`)
- `ip`, `iptables`, `sysctl` must be in `$PATH` (when networking is enabled)

### Step-by-step execution

1. **Verify prerequisites**

   Checks that `FC_KERNEL` and `FC_ROOTFS` exist. Exits with an error if either is missing and directs the user to run `init`.

2. **Recreate golden directory**

   ```sh
   rm -rf /tmp/fc-orch/golden/<tag>
   mkdir -p /tmp/fc-orch/golden/<tag> /tmp/fc-orch/pids
   ```

3. **COW copy of base rootfs**

   ```sh
   cp --reflink=always /tmp/fc-orch/rootfs.ext4 /tmp/fc-orch/golden/<tag>/rootfs.ext4
   ```

   On filesystems that do not support reflinks (e.g. ext4), this falls back to a regular byte-for-byte copy via `io.Copy`. On btrfs or xfs, the reflink is instant and consumes no additional space until the VM writes to the disk.

4. **Network setup** (skipped when `FC_BRIDGE=none`)

   a. Create bridge (idempotent — skipped if `fcbr0` already exists):

   ```sh
   ip link add fcbr0 type bridge
   ip addr add 172.30.0.1/24 dev fcbr0
   ip link set fcbr0 up
   ```

   b. Enable IP forwarding and NAT:

   ```sh
   ip -4 route show default          # detect egress interface, e.g. "eth0"
   sysctl -qw net.ipv4.ip_forward=1
   iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
   ```

   c. Create and attach the golden TAP device:

   ```sh
   ip tuntap add dev fctap0 mode tap
   ip link set fctap0 up
   ip link set fctap0 master fcbr0
   ```

5. **Build Firecracker machine configuration** (passed to the SDK in memory):

   ```
   SocketPath:      /tmp/fc-orch/golden/<tag>/api.sock
   KernelImagePath: /tmp/fc-orch/vmlinux
   KernelArgs:      console=ttyS0 reboot=k panic=1 pci=off i8042.noaux quiet loglevel=0
   MachineCfg:
     VcpuCount:       1
     MemSizeMib:      128
     TrackDirtyPages: true      ← required for snapshot support
   Drives:
     - DriveID:      rootfs
       PathOnHost:   /tmp/fc-orch/golden/<tag>/rootfs.ext4
       IsRootDevice: true
       IsReadOnly:   false
   NetworkInterfaces:
     - MacAddress:  AA:FC:00:00:00:01
       HostDevName: fctap0
   ```

6. **Launch Firecracker process**

   The Firecracker Go SDK spawns:

   ```sh
   firecracker --api-sock /tmp/fc-orch/golden/<tag>/api.sock
   ```

   The SDK then applies the machine configuration via HTTP calls to the Firecracker API socket.

7. **Boot the VM**

   ```go
   m.Start(ctx)   // SDK call — PUT /actions {"action_type": "InstanceStart"}
   ```

   The golden VM PID is written to `/tmp/fc-orch/pids/golden.pid`.

8. **Wait for guest init to settle**

   ```go
   time.Sleep(3 * time.Second)
   ```

   This is a fixed delay. The guest's `rcS` script mounts pseudo-filesystems and brings up `eth0`. 3 seconds is conservative enough for the Alpine init sequence to complete.

9. **Pause the VM**

   ```go
   m.PauseVM(ctx)   // SDK call — PATCH /vm {"state": "Paused"}
   ```

   The VM's vCPUs are frozen. No guest code runs after this point.

10. **Create snapshot**

    ```go
    m.CreateSnapshot(ctx,
        "/tmp/fc-orch/golden/<tag>/mem",
        "/tmp/fc-orch/golden/<tag>/vmstate",
    )
    // SDK call — PUT /snapshot/create
    // {
    //   "mem_file_path":      "/tmp/fc-orch/golden/<tag>/mem",
    //   "snapshot_path":      "/tmp/fc-orch/golden/<tag>/vmstate",
    //   "snapshot_type":      "Full"
    // }
    ```

    - `mem`: full dump of guest physical memory (~128 MiB). All clones map this file read-only; the kernel's MAP_PRIVATE gives each clone copy-on-write semantics over it.
    - `vmstate`: serialized vCPU and device state (typically a few hundred KiB). Each clone gets its own copy.

    Sizes of both files are logged.

11. **Terminate golden VM**

    ```go
    m.StopVMM()   // SDK call — PUT /actions {"action_type": "SendCtrlAltDel"}
    ```

12. **Destroy golden TAP device**

    ```sh
    ip link del fctap0
    ```

### Outputs

| Path | Description |
|---|---|
| `/tmp/fc-orch/golden/<tag>/mem` | Full memory snapshot (~`FC_MEM_MIB` MiB) |
| `/tmp/fc-orch/golden/<tag>/vmstate` | VM state snapshot (vCPU registers, device state) |
| `/tmp/fc-orch/golden/<tag>/rootfs.ext4` | COW copy of base rootfs (not needed after snapshotting, kept for reference) |

### Error conditions

| Error | Cause | Resolution |
|---|---|---|
| `kernel not found — run init first` | `FC_KERNEL` path does not exist | Run `init` first |
| `rootfs not found — run init first` | Ext4 file does not exist | Run `init [distro]` first |
| `firecracker binary not found` | `FC_BIN` not in `$PATH` | Install Firecracker or set `FC_BIN` |
| `create bridge: ...` | `ip link add` failed | Check if another bridge with the same name exists with incompatible config |
| `start golden VM: ...` | Firecracker failed to boot | Check Firecracker logs; verify kernel and rootfs are valid |
| `pause VM: ...` | VM did not reach a pauseable state in 3s | Increase settle time in source or investigate guest crash via serial console |
| `create snapshot: ...` | Snapshot write failed | Check disk space in `FC_BASE_DIR` |

---

## `spawn`

### Purpose

Restores one or more VM clones from the golden snapshot. Each clone is an independent Firecracker process that resumes execution from exactly the paused state captured by `golden`. Clones differ only in their TAP device, MAC address, rootfs COW layer, and vmstate file; they all map the same `golden/mem` file.

Clone IDs are auto-incremented: if clones 1–3 already exist, the next `spawn 2` creates clones 4 and 5. Spawn can be called multiple times to add more clones incrementally.

### Usage

```sh
sudo ./fc-orch spawn               # spawn 1 clone from the "default" golden snapshot
sudo ./fc-orch spawn ubuntu 10     # spawn 10 clones from the "ubuntu" golden snapshot
```

### Prerequisites

- `golden` must have been run (`golden/vmstate` and `golden/mem` must exist)
- `firecracker` binary must be available
- Sufficient disk space for per-clone rootfs copies (each is a copy of the ~512 MiB golden rootfs, but reflinks are instant on btrfs/xfs)

### Step-by-step execution (per clone)

The following steps are performed once for each requested clone. Let `{id}` be the auto-assigned clone number (e.g. `1`, `2`, ...).

1. **Verify golden artifacts exist**

   Checks for both `/tmp/fc-orch/golden/<tag>/vmstate` and `/tmp/fc-orch/golden/<tag>/mem`. Exits with an error if either is missing.

2. **Create directories**

   ```sh
   mkdir -p /tmp/fc-orch/clones /tmp/fc-orch/pids
   mkdir -p /tmp/fc-orch/clones/{id}
   ```

3. **Setup bridge** (idempotent, skipped if bridge already exists or `FC_BRIDGE=none`)

   Same sequence as step 4 of `golden`. No-op if `fcbr0` is already up.

4. **COW copy of golden rootfs**

   ```sh
   cp --reflink=always /tmp/fc-orch/golden/<tag>/rootfs.ext4 /tmp/fc-orch/clones/{id}/rootfs.ext4
   ```

   Falls back to a full copy if reflinks are unsupported.

5. **Shared memory reference** (no copy)

   The clone's Firecracker config will point directly at `/tmp/fc-orch/golden/<tag>/mem`. No file operation is needed here — the kernel's MAP_PRIVATE ensures each clone's writes are private.

6. **Copy vmstate**

   ```sh
   # implemented as io.Copy in Go
   cp /tmp/fc-orch/golden/<tag>/vmstate /tmp/fc-orch/clones/{id}/vmstate
   ```

   The vmstate file is small (typically < 1 MiB), so a full copy is cheap.

7. **Create and attach TAP device** (skipped when `FC_BRIDGE=none`)

   ```sh
   ip tuntap add dev fctap{id} mode tap
   ip link set fctap{id} up
   ip link set fctap{id} master fcbr0
   ```

   MAC address is derived from the clone ID:

   ```
   AA:FC:00:00:{id/256 in hex}:{id%256 in hex}
   # e.g. clone 1  → AA:FC:00:00:00:01
   #      clone 2  → AA:FC:00:00:00:02
   #      clone 256 → AA:FC:00:00:01:00
   ```

8. **Build Firecracker snapshot-restore configuration** (in memory):

   ```
   SocketPath: /tmp/fc-orch/clones/{id}/api.sock
   MachineCfg:
     VcpuCount:  1
     MemSizeMib: 128
   NetworkInterfaces:
     - MacAddress:  AA:FC:00:00:00:{id:02X}
       HostDevName: fctap{id}
   Snapshot:
     MemFilePath:  /tmp/fc-orch/golden/<tag>/mem      ← shared, read-only mapping
     SnapshotPath: /tmp/fc-orch/clones/{id}/vmstate
     ResumeVM:     true                          ← restore instead of fresh boot
   ```

   Note: `KernelImagePath` and `Drives` are omitted when restoring from a snapshot — Firecracker uses the snapshot state instead.

9. **Launch Firecracker process**

   ```sh
   firecracker --api-sock /tmp/fc-orch/clones/{id}/api.sock
   ```

10. **Restore and resume VM**

    ```go
    m.Start(ctx)
    // SDK call — POST /snapshot/load
    // {
    //   "mem_file_path":      "/tmp/fc-orch/golden/<tag>/mem",
    //   "snapshot_path":      "/tmp/fc-orch/clones/{id}/vmstate",
    //   "resume_vm":          true
    // }
    ```

    Restoration time (from `m.Start` call to return) is measured and logged.

11. **Inject network config via MMDS** (only when `FC_AUTO_NET_CONFIG=1` and networking is enabled)

    Immediately after the snapshot is restored, the orchestrator configures the MMDS for this clone via two API calls to the clone's Firecracker socket:

    ```
    PUT /mmds/config
    {"version": "V1", "network_interfaces": ["1"]}

    PUT /mmds
    {"ip": "172.30.0.{10+id}/24", "gw": "172.30.0.1", "dns": "1.1.1.1"}
    ```

    The `fc-net-init` daemon already running inside the guest (started during golden VM boot, captured in the snapshot) polls `169.254.169.254` via a link-local route and applies the config to `eth0` within ~1 second of clone resume.

12. **Record PID**

    ```sh
    echo {pid} > /tmp/fc-orch/pids/clone-{id}.pid
    ```

13. **Register clone in memory**

    The running clone is tracked in an in-process map keyed by clone ID, holding the Firecracker SDK handle, context cancel function, and TAP device name. This allows `kill` to cleanly terminate clones started in the same process invocation.

After all clones are spawned, `Status()` is called automatically to display the running clone table.

### Outputs

For each clone `{id}`:

| Path | Description |
|---|---|
| `/tmp/fc-orch/clones/{id}/rootfs.ext4` | Private COW copy of the golden rootfs |
| `/tmp/fc-orch/clones/{id}/vmstate` | Private copy of golden vmstate |
| `/tmp/fc-orch/clones/{id}/api.sock` | Firecracker API socket (live while clone is running) |
| `/tmp/fc-orch/pids/clone-{id}.pid` | PID of this clone's Firecracker process |
| `fctap{id}` | Host TAP network device attached to `fcbr0` |

### Error conditions

| Error | Cause | Resolution |
|---|---|---|
| `golden vmstate not found — run golden first` | `golden` has not been run | Run `golden` first |
| `golden mem not found — run golden first` | Same as above | Run `golden` first |
| `firecracker not found` | Binary missing | Install Firecracker or set `FC_BIN` |
| `copy rootfs: ...` | Disk full or source missing | Check disk space; re-run `golden` |
| `restore clone {id}: ...` | Firecracker failed to load snapshot | Check that `golden/mem` is not corrupted; re-run `golden` |
| `create tap {name}: ...` | TAP device already exists or `ip` failed | Run `kill` to clean up stale TAPs, then retry |
| Individual clone failure | Per-clone errors are logged but do not abort the batch | Check logs; surviving clones continue running |

---

## `status`

### Purpose

Displays a table of all clones that have been spawned, along with their PIDs and liveness. Liveness is determined by checking whether `/proc/{pid}` exists — a process that has exited will no longer appear in `/proc`.

This command does not require the clones to have been started in the current process invocation; it reads PID files written to disk by `spawn`.

### Usage

```sh
sudo ./fc-orch status
```

### Prerequisites

None. Can be run at any time, even with no clones running.

### Step-by-step execution

1. **Read PID directory**

   Lists all files in `/tmp/fc-orch/pids/`.

2. **Filter for clone PID files**

   Only files whose names start with `clone-` are considered (excludes `golden.pid`).

3. **Check liveness**

   For each file:

   ```sh
   # read pid from clone-{id}.pid
   test -d /proc/{pid}   # alive if directory exists
   ```

4. **Print table**

   ```
   === Running clones ===
     clone-1       pid=12345     alive
     clone-2       pid=12346     alive
     clone-3       pid=12347     DEAD
   ```

### Outputs

Prints to stdout only. No files are modified.

---

## `kill`

### Purpose

Terminates all running Firecracker VM processes and removes all TAP devices. Handles two cases:

- **In-memory clones**: clones started in the same `fc-orch` process invocation, tracked via the in-process clone map.
- **Orphaned clones**: clones from a previous `fc-orch spawn` invocation, whose PIDs are recorded in PID files.

Both cases are always handled, so `kill` works correctly even after a restart or crash of the orchestrator process.

The command does **not** remove snapshot files or the `clones/` directory — use `cleanup` for full teardown.

### Usage

```sh
sudo ./fc-orch kill
```

### Prerequisites

None. Safe to run even if no VMs are running.

### Step-by-step execution

1. **Stop in-memory clones** (clones started in this process invocation)

   For each clone tracked in the in-process map:

   ```go
   clone.Machine.StopVMM()   // SDK: PUT /actions {"action_type": "SendCtrlAltDel"}
   clone.Cancel()            // cancels the clone's context
   ```

   ```sh
   ip link del fctap{id}
   ```

   The clone is removed from the in-memory map.

2. **Kill orphaned processes from PID files**

   For each file in `/tmp/fc-orch/pids/`:

   ```go
   // equivalent to:
   kill -9 {pid}
   ```

   The PID file is then deleted:

   ```sh
   rm /tmp/fc-orch/pids/{file}
   ```

3. **Destroy stale TAP devices**

   ```sh
   ip -o link show
   ```

   Each line containing `fctap` is parsed to extract the device name, then:

   ```sh
   ip link del {tapname}
   ```

   This cleans up any TAP devices left over from previous runs that were not removed by steps 1 or 2.

### Outputs

All Firecracker processes are terminated. All `fctap*` network interfaces are removed. PID files are deleted. The `clones/`, `golden/`, and `pids/` directories themselves remain on disk.

### Error conditions

Errors from individual `StopVMM`, `kill`, or `ip link del` calls are logged but do not abort the rest of the kill sequence. The command always attempts to clean up everything it finds.

---

## `cleanup`

### Purpose

Performs a full teardown: kills all VMs (same as `kill`), removes all working directories under `$FC_BASE_DIR`, and tears down the host bridge. After `cleanup`, the system is in the same state as before `golden` was run. The base kernel and rootfs files are **not** removed — only `golden/`, `clones/`, and `pids/` are deleted.

### Usage

```sh
sudo ./fc-orch cleanup
```

### Prerequisites

None. Safe to run at any time.

### Step-by-step execution

1. **Kill all VMs**

   Performs the full `kill` sequence (see [`kill`](#kill) above).

2. **Remove working directories**

   ```sh
   rm -rf /tmp/fc-orch/clones
   rm -rf /tmp/fc-orch/golden
   rm -rf /tmp/fc-orch/pids
   ```

3. **Tear down bridge** (skipped when `FC_BRIDGE=none`)

   ```sh
   ip link del fcbr0
   ```

   This also implicitly removes all addresses and routes associated with the bridge. The iptables NAT rule added during `golden`/`spawn` is **not** removed automatically — remove it manually if needed:

   ```sh
   iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
   ```

### Outputs

After `cleanup`:

- All Firecracker processes are terminated
- `/tmp/fc-orch/clones/`, `/tmp/fc-orch/golden/`, `/tmp/fc-orch/pids/` are deleted
- `fcbr0` bridge and all associated `fctap*` devices are removed
- `/tmp/fc-orch/vmlinux` and `/tmp/fc-orch/rootfs.ext4` remain intact

To also remove the kernel and rootfs:

```sh
sudo ./fc-orch cleanup
rm -f /tmp/fc-orch/vmlinux /tmp/fc-orch/rootfs.ext4
```

### Error conditions

Errors from `rm -rf` or `ip link del` are ignored (logged only). Cleanup always completes all steps.

---

## Typical Workflow

```sh
# One-time setup: download kernel and build rootfs
sudo ./fc-orch init

# Create golden snapshot (re-run any time you want a fresh baseline)
sudo ./fc-orch golden

# Spawn clones
sudo ./fc-orch spawn 10

# Check what's running
sudo ./fc-orch status

# Add more clones to the existing set
sudo ./fc-orch spawn 5

# Tear down all clones (keeps golden snapshot)
sudo ./fc-orch kill

# Full reset (keeps kernel and rootfs)
sudo ./fc-orch cleanup
```