feat: show golden VM tag in clone list, add console logging, fix ubuntu boot

- Persist golden VM tag to clones/{id}/tag at spawn time
- GET /clones now returns [{id, tag}] objects instead of plain IDs
- Web UI renders tag as a dim label next to each clone entry (clone 3 · default)
- Pre-existing fixes included in this commit:
  - console: tee all PTY output to clones/{id}/console.log for boot capture
  - network: destroy stale tap before recreating to avoid EBUSY errors
  - orchestrator: fix ubuntu systemd boot (custom fc-console.service, fstab,
    mask serial-getty udev dep, longer settle time, correct package list)
  - config: remove quiet/loglevel=0 from default boot args

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs/commands.md

@@ -35,8 +35,9 @@ All tunables are set via environment variables. Every variable has a default; no
 | `FC_MEM_MIB` | `128` | Memory per VM in MiB |
 | `FC_BRIDGE` | `fcbr0` | Host bridge name. Set to `none` to disable all networking |
 | `FC_BRIDGE_CIDR` | `172.30.0.1/24` | IP address and prefix assigned to the host bridge |
-| `FC_GUEST_PREFIX` | `172.30.0` | IP prefix for guest address allocation |
+| `FC_GUEST_PREFIX` | `172.30.0` | IP prefix for guest address allocation (used with `FC_AUTO_NET_CONFIG`) |
-| `FC_GUEST_GW` | `172.30.0.1` | Default gateway advertised to guests |
+| `FC_GUEST_GW` | `172.30.0.1` | Default gateway advertised to guests (used with `FC_AUTO_NET_CONFIG`) |
+| `FC_AUTO_NET_CONFIG` | _(unset)_ | Set to `1` to automatically assign guest IPs via MMDS on clone start |
 
 Kernel boot arguments are hardcoded and not user-configurable:
 
@@ -55,10 +56,12 @@ After running all commands, `$FC_BASE_DIR` (`/tmp/fc-orch` by default) contains:
 ├── vmlinux                    # kernel image (shared, immutable)
 ├── rootfs.ext4                # base Alpine rootfs (shared, immutable)
 ├── golden/
-│   ├── api.sock               # Firecracker API socket (golden VM, transient)
+│   ├── default/               # "default" tag directory
-│   ├── rootfs.ext4            # COW copy of base rootfs used by golden VM
+│   │   ├── api.sock           # Firecracker API socket (golden VM, transient)
-│   ├── mem                    # memory snapshot (read by all clones, never written)
+│   │   ├── rootfs.ext4        # COW copy of base rootfs used by golden VM
-│   └── vmstate                # VM state snapshot (golden reference)
+│   │   ├── mem                # memory snapshot (read by all clones, never written)
+│   │   └── vmstate            # VM state snapshot (golden reference)
+│   └── <tag>/                 # other tagged snapshots
 ├── clones/
 │   ├── 1/
 │   │   ├── api.sock           # Firecracker API socket (clone 1)
@@ -95,23 +98,63 @@ When `FC_BRIDGE` is not `none` (the default), a Linux bridge and per-VM TAP devi
      └── fctapN  (clone N)
 ```
 
-Each clone receives a unique TAP device and MAC address (`AA:FC:00:00:XX:XX`). IP assignment inside the guest is the guest OS's responsibility (the rootfs init script only brings `eth0` up; no DHCP server is included).
+Each clone receives a unique TAP device and MAC address (`AA:FC:00:00:XX:XX`). The host-side bridge has NAT masquerading enabled so guests can reach the internet through the host's default route.
 
 Set `FC_BRIDGE=none` to skip all network configuration. VMs will boot without a network interface.
 
+### Guest IP assignment
+
+The rootfs init script brings `eth0` up at the link layer only. Guests have no IP address by default. There are two ways to configure networking inside a VM:
+
+#### Manual configuration (inside the VM console)
+
+```sh
+# Pick an unused IP in the bridge subnet — e.g. .11 for clone 1, .12 for clone 2
+ip addr add 172.30.0.11/24 dev eth0
+ip route add default via 172.30.0.1
+echo "nameserver 1.1.1.1" > /etc/resolv.conf
+ping 1.1.1.1   # verify
+```
+
+Manual config is ephemeral — it is lost when the clone is stopped. Use the automatic option below for persistent configuration.
+
+#### Automatic configuration via MMDS (`FC_AUTO_NET_CONFIG=1`)
+
+When `FC_AUTO_NET_CONFIG=1` is set, the orchestrator uses the Firecracker **Microvm Metadata Service (MMDS)** to inject per-clone network config immediately after the VM starts. A small background daemon embedded in the rootfs (`/sbin/fc-net-init`) polls `169.254.169.254` and applies the config automatically — no manual steps needed.
+
+IPs are assigned deterministically from `FC_GUEST_PREFIX`:
+
+```
+clone 1  → 172.30.0.11/24
+clone 2  → 172.30.0.12/24
+…
+clone N  → 172.30.0.(10+N)/24
+```
+
+Usage:
+
+```sh
+sudo FC_AUTO_NET_CONFIG=1 ./fc-orch start
+```
+
+Within ~1–2 seconds of clone start, `eth0` inside the VM will have the assigned IP, default route, and DNS (`1.1.1.1`) configured.
+
+> **Note:** `FC_AUTO_NET_CONFIG` requires `fc-orch init` and `fc-orch golden` to have been run (or re-run) after this feature was added, so that the `fc-net-init` daemon is present in the golden snapshot.
+
 ---
 
 ## `init`
 
 ### Purpose
 
-Downloads the Linux kernel image and builds a minimal Alpine Linux ext4 rootfs. This command only needs to run once; both artifacts are reused by all subsequent `golden` invocations. `init` is idempotent — it skips any artifact that already exists on disk.
+Downloads the Linux kernel image and builds a minimal filesystem (Alpine, Debian, or Ubuntu). This command only needs to run once per distro; both artifacts are reused by `golden` invocations. `init` is idempotent — it skips any artifact that already exists on disk.
 
 ### Usage
 
 ```sh
-sudo ./fc-orch init
+sudo ./fc-orch init [distro]
 ```
+Where `[distro]` can be `alpine` (default), `debian`, or `ubuntu`.
 
 Optional overrides:
 
@@ -225,13 +268,15 @@ This command always recreates the golden directory from scratch, discarding any
 ### Usage
 
 ```sh
-sudo ./fc-orch golden
+sudo ./fc-orch golden [tag] [distro]
 ```
 
+Where `[tag]` identifies the snapshot baseline name (default `default`), and `[distro]` dictates the source `.ext4` image to use (default: `alpine`).
+
 Optional overrides:
 
 ```sh
-sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
+sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden ubuntu
 ```
 
 ### Prerequisites
@@ -249,14 +294,14 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
 2. **Recreate golden directory**
 
    ```sh
-   rm -rf /tmp/fc-orch/golden
+   rm -rf /tmp/fc-orch/golden/<tag>
-   mkdir -p /tmp/fc-orch/golden /tmp/fc-orch/pids
+   mkdir -p /tmp/fc-orch/golden/<tag> /tmp/fc-orch/pids
    ```
 
 3. **COW copy of base rootfs**
 
    ```sh
-   cp --reflink=always /tmp/fc-orch/rootfs.ext4 /tmp/fc-orch/golden/rootfs.ext4
+   cp --reflink=always /tmp/fc-orch/rootfs.ext4 /tmp/fc-orch/golden/<tag>/rootfs.ext4
    ```
 
    On filesystems that do not support reflinks (e.g. ext4), this falls back to a regular byte-for-byte copy via `io.Copy`. On btrfs or xfs, the reflink is instant and consumes no additional space until the VM writes to the disk.
@@ -290,7 +335,7 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
 5. **Build Firecracker machine configuration** (passed to the SDK in memory):
 
    ```
-   SocketPath:      /tmp/fc-orch/golden/api.sock
+   SocketPath:      /tmp/fc-orch/golden/<tag>/api.sock
    KernelImagePath: /tmp/fc-orch/vmlinux
    KernelArgs:      console=ttyS0 reboot=k panic=1 pci=off i8042.noaux quiet loglevel=0
    MachineCfg:
@@ -299,7 +344,7 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
      TrackDirtyPages: true      ← required for snapshot support
    Drives:
      - DriveID:      rootfs
-       PathOnHost:   /tmp/fc-orch/golden/rootfs.ext4
+       PathOnHost:   /tmp/fc-orch/golden/<tag>/rootfs.ext4
        IsRootDevice: true
        IsReadOnly:   false
    NetworkInterfaces:
@@ -312,7 +357,7 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
    The Firecracker Go SDK spawns:
 
    ```sh
-   firecracker --api-sock /tmp/fc-orch/golden/api.sock
+   firecracker --api-sock /tmp/fc-orch/golden/<tag>/api.sock
    ```
 
    The SDK then applies the machine configuration via HTTP calls to the Firecracker API socket.
@@ -345,13 +390,13 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
 
     ```go
     m.CreateSnapshot(ctx,
-        "/tmp/fc-orch/golden/mem",
+        "/tmp/fc-orch/golden/<tag>/mem",
-        "/tmp/fc-orch/golden/vmstate",
+        "/tmp/fc-orch/golden/<tag>/vmstate",
     )
     // SDK call — PUT /snapshot/create
     // {
-    //   "mem_file_path":      "/tmp/fc-orch/golden/mem",
+    //   "mem_file_path":      "/tmp/fc-orch/golden/<tag>/mem",
-    //   "snapshot_path":      "/tmp/fc-orch/golden/vmstate",
+    //   "snapshot_path":      "/tmp/fc-orch/golden/<tag>/vmstate",
     //   "snapshot_type":      "Full"
     // }
     ```
@@ -377,16 +422,16 @@ sudo FC_MEM_MIB=256 FC_VCPUS=2 ./fc-orch golden
 
 | Path | Description |
 |---|---|
-| `/tmp/fc-orch/golden/mem` | Full memory snapshot (~`FC_MEM_MIB` MiB) |
+| `/tmp/fc-orch/golden/<tag>/mem` | Full memory snapshot (~`FC_MEM_MIB` MiB) |
-| `/tmp/fc-orch/golden/vmstate` | VM state snapshot (vCPU registers, device state) |
+| `/tmp/fc-orch/golden/<tag>/vmstate` | VM state snapshot (vCPU registers, device state) |
-| `/tmp/fc-orch/golden/rootfs.ext4` | COW copy of base rootfs (not needed after snapshotting, kept for reference) |
+| `/tmp/fc-orch/golden/<tag>/rootfs.ext4` | COW copy of base rootfs (not needed after snapshotting, kept for reference) |
 
 ### Error conditions
 
 | Error | Cause | Resolution |
 |---|---|---|
 | `kernel not found — run init first` | `FC_KERNEL` path does not exist | Run `init` first |
-| `rootfs not found — run init first` | `FC_ROOTFS` path does not exist | Run `init` first |
+| `rootfs not found — run init first` | Ext4 file does not exist | Run `init [distro]` first |
 | `firecracker binary not found` | `FC_BIN` not in `$PATH` | Install Firecracker or set `FC_BIN` |
 | `create bridge: ...` | `ip link add` failed | Check if another bridge with the same name exists with incompatible config |
 | `start golden VM: ...` | Firecracker failed to boot | Check Firecracker logs; verify kernel and rootfs are valid |
@@ -406,8 +451,8 @@ Clone IDs are auto-incremented: if clones 1–3 already exist, the next `spawn 2
 ### Usage
 
 ```sh
-sudo ./fc-orch spawn        # spawn 1 clone (default)
+sudo ./fc-orch spawn               # spawn 1 clone from the "default" golden snapshot
-sudo ./fc-orch spawn 10     # spawn 10 clones
+sudo ./fc-orch spawn ubuntu 10     # spawn 10 clones from the "ubuntu" golden snapshot
 ```
 
 ### Prerequisites
@@ -422,7 +467,7 @@ The following steps are performed once for each requested clone. Let `{id}` be t
 
 1. **Verify golden artifacts exist**
 
-   Checks for both `/tmp/fc-orch/golden/vmstate` and `/tmp/fc-orch/golden/mem`. Exits with an error if either is missing.
+   Checks for both `/tmp/fc-orch/golden/<tag>/vmstate` and `/tmp/fc-orch/golden/<tag>/mem`. Exits with an error if either is missing.
 
 2. **Create directories**
 
@@ -438,20 +483,20 @@ The following steps are performed once for each requested clone. Let `{id}` be t
 4. **COW copy of golden rootfs**
 
    ```sh
-   cp --reflink=always /tmp/fc-orch/golden/rootfs.ext4 /tmp/fc-orch/clones/{id}/rootfs.ext4
+   cp --reflink=always /tmp/fc-orch/golden/<tag>/rootfs.ext4 /tmp/fc-orch/clones/{id}/rootfs.ext4
    ```
 
    Falls back to a full copy if reflinks are unsupported.
 
 5. **Shared memory reference** (no copy)
 
-   The clone's Firecracker config will point directly at `/tmp/fc-orch/golden/mem`. No file operation is needed here — the kernel's MAP_PRIVATE ensures each clone's writes are private.
+   The clone's Firecracker config will point directly at `/tmp/fc-orch/golden/<tag>/mem`. No file operation is needed here — the kernel's MAP_PRIVATE ensures each clone's writes are private.
 
 6. **Copy vmstate**
 
    ```sh
    # implemented as io.Copy in Go
-   cp /tmp/fc-orch/golden/vmstate /tmp/fc-orch/clones/{id}/vmstate
+   cp /tmp/fc-orch/golden/<tag>/vmstate /tmp/fc-orch/clones/{id}/vmstate
    ```
 
    The vmstate file is small (typically < 1 MiB), so a full copy is cheap.
@@ -484,7 +529,7 @@ The following steps are performed once for each requested clone. Let `{id}` be t
      - MacAddress:  AA:FC:00:00:00:{id:02X}
        HostDevName: fctap{id}
    Snapshot:
-     MemFilePath:  /tmp/fc-orch/golden/mem      ← shared, read-only mapping
+     MemFilePath:  /tmp/fc-orch/golden/<tag>/mem      ← shared, read-only mapping
      SnapshotPath: /tmp/fc-orch/clones/{id}/vmstate
      ResumeVM:     true                          ← restore instead of fresh boot
    ```
@@ -503,7 +548,7 @@ The following steps are performed once for each requested clone. Let `{id}` be t
     m.Start(ctx)
     // SDK call — POST /snapshot/load
     // {
-    //   "mem_file_path":      "/tmp/fc-orch/golden/mem",
+    //   "mem_file_path":      "/tmp/fc-orch/golden/<tag>/mem",
     //   "snapshot_path":      "/tmp/fc-orch/clones/{id}/vmstate",
     //   "resume_vm":          true
     // }
@@ -511,13 +556,27 @@ The following steps are performed once for each requested clone. Let `{id}` be t
 
     Restoration time (from `m.Start` call to return) is measured and logged.
 
-11. **Record PID**
+11. **Inject network config via MMDS** (only when `FC_AUTO_NET_CONFIG=1` and networking is enabled)
+
+    Immediately after the snapshot is restored, the orchestrator configures the MMDS for this clone via two API calls to the clone's Firecracker socket:
+
+    ```
+    PUT /mmds/config
+    {"version": "V1", "network_interfaces": ["1"]}
+
+    PUT /mmds
+    {"ip": "172.30.0.{10+id}/24", "gw": "172.30.0.1", "dns": "1.1.1.1"}
+    ```
+
+    The `fc-net-init` daemon already running inside the guest (started during golden VM boot, captured in the snapshot) polls `169.254.169.254` via a link-local route and applies the config to `eth0` within ~1 second of clone resume.
+
+12. **Record PID**
 
     ```sh
     echo {pid} > /tmp/fc-orch/pids/clone-{id}.pid
     ```
 
-12. **Register clone in memory**
+13. **Register clone in memory**
 
     The running clone is tracked in an in-process map keyed by clone ID, holding the Firecracker SDK handle, context cancel function, and TAP device name. This allows `kill` to cleanly terminate clones started in the same process invocation.
 

docs/create_golden_image.md

@@ -0,0 +1,166 @@
+# Guide: Creating Custom Golden Images
+
+This guide outlines exactly how to create new, customized golden images (e.g. pivoting from Alpine to an Ubuntu or Node.js environment) and seamlessly integrate them into the `fc-orch` tagging system. 
+
+By default, executing `./fc-orch init` gives you a basic Alpine Linux image, but you can also generate built-in Ubuntu and Debian environments trivially via `./fc-orch init ubuntu` or `./fc-orch init debian`. The real power of `fc-orch` lies in maintaining multiple customized snapshot bases (golden tags).
+
+---
+
+## 1. Acquiring Custom Assets
+
+To build a fresh golden image, at minimum you must provide a new filesystem:
+- **Custom Root Filesystem**: An uncompressed `ext4` filesystem image that contains your system and libraries.
+- **Custom Kernel** *(Optional)*: An uncompressed Linux kernel binary (`vmlinux`). If not provided, the default Firecracker CI VM kernel will continue to be utilized flawlessly.
+
+### Recommendations for Custom Distros:
+If you are generating a completely custom/unsupported base, you may use tools like `docker export` compiled via `mkfs.ext4`, or utilize `debootstrap` to provision your image. 
+
+Ensure that your custom root filesystem contains an appropriate bootstrap sequence inside `/etc/init.d/rcS` (or systemd if configured) to natively mount essential directories (`/proc`, `/sys`, `/dev`) and configure the `eth0` link interface, as Firecracker expects the guest OS to prepare these primitives natively. Our native `init` tool handles this automatically for `alpine`, `ubuntu`, and `debian` distributions.
+
+## 2. Using Environment Overrides
+
+Rather than replacing the default `/tmp/fc-orch/rootfs.ext4`, `fc-orch` implements powerful environment variables you can override prior to capturing the golden snapshot. 
+
+The essential variables to override are:
+- `FC_ROOTFS`: Path to your custom `.ext4` image (e.g., `/home/user/ubuntu.ext4`).
+- `FC_MEM_MIB`: Amount of initial memory the golden VM receives. Heavier OS's like Ubuntu typically require more than the 128 MiB default (e.g., `512`).
+- `FC_VCPUS`: Processing allocation to start the VM. Default is `1`.
+
+## 3. Capturing the Custom Golden Snapshot
+
+Let's assume we want to provision a standard Ubuntu environment. First, create the rootfs (this automatically downloads and sets up the rootfs on your host):
+```bash
+sudo ./fc-orch init ubuntu
+```
+
+Then capture the baseline using the `ubuntu` tag and the `ubuntu` distro target. Note the increased resources `FC_MEM_MIB` allocating 512MB RAM for tighter operations.
+```bash
+sudo FC_MEM_MIB=512 FC_VCPUS=2 ./fc-orch golden ubuntu ubuntu
+```
+
+### What happens in the background?
+1. The orchestrator prepares a brand new directory for this baseline exclusively at: `/tmp/fc-orch/golden/ubuntu/`.
+2. It takes your customized `/home/user/ubuntu.ext4` and utilizes it as the root block device for the orchestrator environment.
+3. Firecracker boots the VM. It waits exactly 3 seconds for the OS initialization logic to naturally settle.
+4. The VM instance is aggressively paused. A serialized register state checkpoint (`vmstate`) and raw memory projection (`mem`) are exported permanently into the `/tmp/fc-orch/golden/ubuntu/` directory.
+
+> **Note**: Firecracker terminates the internal process after finalizing the artifacts. The custom snapshot baseline completely persists!
+
+## 4. Spawning Scalable Clones
+
+Since your image is now indexed inside the `ubuntu` tag boundary, it can be cloned independently using Copy-on-Write (COW).
+
+Simply address the target tag along with the desired replica count during the `spawn` command:
+
+```bash
+sudo ./fc-orch spawn ubuntu 10
+```
+
+This immediately duplicates the exact hardware footprint, generating 10 concurrent active Firecracker VMs resolving locally to your custom OS without disturbing your previous generic Alpine builds. Multiple base OS architectures can run collectively side-by-side using this methodology!
+
+---
+
+## How Ubuntu VM Configuration Works
+
+### Build-time: chroot package installation
+
+`ubuntu-base` is a deliberately bare tarball — it ships no shell beyond `/bin/sh` (dash), no network tools, and no package cache.  When `fc-orch init ubuntu` runs, after extracting the tarball the orchestrator performs a chroot install step:
+
+1. **Virtual filesystems are bind-mounted** into the image (`/proc`, `/sys`, `/dev`, `/dev/pts`) so that `apt-get` can function correctly inside the chroot.
+2. **`/etc/resolv.conf` is copied** from the host so DNS works during the install.
+3. **`apt-get` installs the following packages** with `--no-install-recommends` to keep the image lean:
+
+   | Package | Purpose |
+   |---|---|
+   | `bash` | Interactive shell |
+   | `curl` | General-purpose HTTP client |
+   | `iproute2` | Provides the `ip` command (required by `fc-net-init`) |
+   | `wget` | Used by `fc-net-init` to poll the MMDS metadata endpoint |
+   | `ca-certificates` | Trusted CA bundle so HTTPS works out of the box |
+
+4. **`apt` cache is purged** (`apt-get clean` + `rm -rf /var/lib/apt/lists/*`) before unmounting, keeping the final image around 200 MB on disk rather than 2 GB.
+5. All bind mounts are removed before the function returns, whether or not the install succeeded.
+
+The resulting ext4 image is **512 MB** (vs. 2 GB for a stock Ubuntu cloud image), comfortably fitting the installed packages with room for runtime state.
+
+### Boot-time: guest network autoconfiguration via MMDS
+
+Every Ubuntu image gets `/sbin/fc-net-init` embedded at build time.  On Ubuntu this script is wired into systemd as `fc-net-init.service` (enabled in `multi-user.target`).
+
+When a clone VM resumes from its golden snapshot the service runs the following sequence:
+
+```
+1. ip addr add 169.254.169.2/32 dev eth0
+   — Adds a link-local address so the guest can reach the Firecracker MMDS
+     gateway at 169.254.169.254 without any prior routing state.
+
+2. Poll GET http://169.254.169.254/ip  (1-second timeout, retry every 1 s)
+   — Loops until the host has injected the per-clone IP config via
+     PUT /mmds on the Firecracker API socket.
+
+3. Once /ip responds, fetch /gw and /dns from the same endpoint.
+
+4. ip addr flush dev eth0
+   ip addr add  <ip>   dev eth0
+   ip route add default via <gw> dev eth0
+   echo "nameserver <dns>" > /etc/resolv.conf
+   — Applies the config atomically and exits.
+```
+
+The host side (see `orchestrator/network.go`) injects the three keys (`ip`, `gw`, `dns`) via the Firecracker MMDS API **after** the snapshot is loaded but **before** the VM is resumed, so the guest sees the data on its very first poll.
+
+This design means the golden snapshot captures the polling loop already running.  Clones that are spawned without `FC_AUTO_NET_CONFIG=1` will still run the loop — it simply never exits, which is harmless and consumes negligible CPU.
+
+### Serial console
+
+`serial-getty@ttyS0.service` is enabled at build time via a symlink in `getty.target.wants`.  The root password is cleared so the console auto-logs-in without a password prompt.  Connect with:
+
+```bash
+sudo ./fc-orch console <clone-id>
+```
+
+---
+
+## Appendix: Practical Examples
+
+### Creating Multiple Golden Images with Different Specs
+
+You can manage a rich registry of different tagged images, provisioning them with varying specifications.
+
+**1. Standard Alpine (Default, 128 MiB RAM, 1 vCPU)**
+```bash
+sudo ./fc-orch golden alpine alpine
+```
+
+**2. Ubuntu Web Server (1024 MiB RAM, 2 vCPUs)**
+```bash
+# assuming init ubuntu was already run
+sudo FC_MEM_MIB=1024 FC_VCPUS=2 ./fc-orch golden my-ubuntu-server ubuntu
+```
+
+**3. Debian Database Node (4096 MiB RAM, 4 vCPUs)**
+```bash
+# assuming init debian was already run
+sudo FC_MEM_MIB=4096 FC_VCPUS=4 ./fc-orch golden my-debian-db debian
+```
+
+**4. External Custom Image (E.g. CentOS via Manual Provision)**
+```bash
+sudo FC_ROOTFS=/images/centos.ext4 FC_MEM_MIB=4096 FC_VCPUS=4 ./fc-orch golden tag-centos
+```
+
+### Inspecting Your Hypervisor State
+
+To easily visualize what your orchestrator has stored and where, you can run the following hypervisor commands:
+
+**View the structured layout of all golden image namespaces:**
+```bash
+tree -a /tmp/fc-orch/golden
+```
+*(If `tree` is not installed, you can use `ls -R /tmp/fc-orch/golden`)*
+
+**View the exact disk usage and file sizes for a specific image artifact (like ubuntu):**
+```bash
+ls -lh /tmp/fc-orch/golden/ubuntu/
+```
+Output will similarly demonstrate that `mem` represents your full allocated RAM (e.g., 1024M), while `vmstate` is essentially negligible.

go.mod

@@ -1,6 +1,6 @@
 module github.com/kacerr/fc-orchestrator
 
-go 1.23
+go 1.25.0
 
 require (
 	github.com/firecracker-microvm/firecracker-go-sdk v1.0.0
@@ -14,6 +14,7 @@ require (
 	github.com/containerd/fifo v1.0.0 // indirect
 	github.com/containernetworking/cni v1.0.1 // indirect
 	github.com/containernetworking/plugins v1.0.1 // indirect
+	github.com/creack/pty v1.1.24 // indirect
 	github.com/go-openapi/analysis v0.21.2 // indirect
 	github.com/go-openapi/errors v0.20.2 // indirect
 	github.com/go-openapi/jsonpointer v0.19.5 // indirect
@@ -26,6 +27,7 @@ require (
 	github.com/go-openapi/validate v0.22.0 // indirect
 	github.com/go-stack/stack v1.8.1 // indirect
 	github.com/google/uuid v1.3.0 // indirect
+	github.com/gorilla/websocket v1.5.3 // indirect
 	github.com/hashicorp/errwrap v1.0.0 // indirect
 	github.com/hashicorp/go-multierror v1.1.1 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
@@ -38,7 +40,8 @@ require (
 	github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f // indirect
 	go.mongodb.org/mongo-driver v1.8.3 // indirect
 	golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd // indirect
-	golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 // indirect
+	golang.org/x/sys v0.43.0 // indirect
+	golang.org/x/term v0.42.0 // indirect
 	golang.org/x/text v0.3.7 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
 )

go.sum

@@ -208,6 +208,8 @@ github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:ma
 github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
 github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=
 github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
+github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
+github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
 github.com/cyphar/filepath-securejoin v0.2.2/go.mod h1:FpkQEhXnPnOthhzymB7CGsFk2G9VLXONKD9G7QGMM+4=
 github.com/d2g/dhcp4 v0.0.0-20170904100407-a1d1b6c41b1c/go.mod h1:Ct2BUK8SB0YC1SMSibvLzxjeJLnrYEVLULFNiHY9YfQ=
 github.com/d2g/dhcp4client v1.0.0/go.mod h1:j0hNfjhrt2SxUOw55nL0ATM/z4Yt3t2Kd1mW34z5W5s=
@@ -406,6 +408,8 @@ github.com/gorilla/mux v1.7.2/go.mod h1:1lud6UwP+6orDFRuTfBEV8e9/aOM/c4fVVCaMa2z
 github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
 github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
 github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
+github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
+github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
 github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
 github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
 github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
@@ -444,6 +448,7 @@ github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1
 github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
 github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
 github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
+github.com/kacerrmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7JulP+udvsHwJoVG1YGAP6VLg4y9I5dyZdqmA=
 github.com/karrick/godirwalk v1.8.0/go.mod h1:H5KPZjojv4lE+QYImBI8xVtrBRgYrIVsaRPx4tDPEn4=
 github.com/karrick/godirwalk v1.10.3/go.mod h1:RoGL9dQei4vP9ilrpETWE8CLOZ1kiN0LhBygSwrAsHA=
 github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
@@ -686,7 +691,7 @@ github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415/go.mod h1:
 github.com/xeipuuv/gojsonschema v0.0.0-20180618132009-1d523034197f/go.mod h1:5yf86TLmAcydyeJq5YvxkGPE2fm/u4myDekKRoLuqhs=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
 github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
-github.com/kacerrmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7JulP+udvsHwJoVG1YGAP6VLg4y9I5dyZdqmA=
+github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7JulP+udvsHwJoVG1YGAP6VLg4y9I5dyZdqmA=
 github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
 github.com/yvasiyarov/go-metrics v0.0.0-20140926110328-57bccd1ccd43/go.mod h1:aX5oPXxHm3bOH+xeAttToC8pqch2ScQN/JoXYupl6xs=
@@ -889,9 +894,13 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20220204135822-1c1b9b1eba6a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8 h1:0A+M6Uqn+Eje4kHMK80dtF3JCXC4ykBgQG4Fe06QRhQ=
 golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI=
+golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
+golang.org/x/term v0.42.0 h1:UiKe+zDFmJobeJ5ggPwOshJIVt6/Ft0rcfrXZDLWAWY=
+golang.org/x/term v0.42.0/go.mod h1:Dq/D+snpsbazcBG5+F9Q1n2rXV8Ma+71xEjTRufARgY=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=

main.go

@@ -16,13 +16,40 @@
 package main
 
 import (
+	"flag"
 	"fmt"
 	"os"
+	"path/filepath"
+	"runtime"
+	"strconv"
+
+	log "github.com/sirupsen/logrus"
 
 	"github.com/kacerr/fc-orchestrator/orchestrator"
 )
 
 func main() {
+	// strip --dev flag before subcommand routing
+	dev := false
+	filtered := os.Args[:1]
+	for _, a := range os.Args[1:] {
+		if a == "--dev" {
+			dev = true
+		} else {
+			filtered = append(filtered, a)
+		}
+	}
+	os.Args = filtered
+
+	if dev {
+		log.SetReportCaller(true)
+		log.SetFormatter(&log.TextFormatter{
+			CallerPrettyfier: func(f *runtime.Frame) (string, string) {
+				return "", fmt.Sprintf("%s:%d", filepath.Base(f.File), f.Line)
+			},
+		})
+	}
+
 	// figure out if we are running as root
 	if os.Geteuid() == 0 {
 		fmt.Println("Running with root/sudo privileges!")
@@ -39,21 +66,66 @@ func main() {
 
 	switch os.Args[1] {
 	case "init":
-		fatal(orch.Init())
+		distro := "alpine"
+		if len(os.Args) > 2 {
+			distro = os.Args[2]
+		}
+		fatal(orch.Init(distro))
 	case "golden":
-		fatal(orch.Golden())
+		tag := "default"
+		distro := "alpine"
+		if len(os.Args) > 2 {
+			tag = os.Args[2]
+		}
+		if len(os.Args) > 3 {
+			distro = os.Args[3]
+		}
+		fatal(orch.Golden(tag, distro))
 	case "spawn":
 		n := 1
-		if len(os.Args) > 2 {
+		tag := "default"
-			fmt.Sscanf(os.Args[2], "%d", &n)
+		for _, arg := range os.Args[2:] {
+			if parsed, err := strconv.Atoi(arg); err == nil {
+				n = parsed
+			} else {
+				tag = arg
+			}
 		}
-		fatal(orch.Spawn(n))
+		fatal(orch.Spawn(n, tag))
 	case "status":
 		orch.Status()
 	case "kill":
 		fatal(orch.Kill())
 	case "cleanup":
 		fatal(orch.Cleanup())
+	case "serve":
+		addr := ":8080"
+		if len(os.Args) > 2 {
+			addr = os.Args[2]
+		}
+		fatal(orchestrator.Serve(orch, addr))
+	case "console":
+		if len(os.Args) < 3 {
+			fmt.Fprintf(os.Stderr, "usage: %s console <id>\n", os.Args[0])
+			os.Exit(1)
+		}
+		var id int
+		fmt.Sscanf(os.Args[2], "%d", &id)
+		fatal(orchestrator.ConnectConsole(orchestrator.DefaultConfig(), id))
+	case "_console-proxy":
+		// Internal subcommand: started by spawnOne, runs as a background daemon.
+		fs := flag.NewFlagSet("_console-proxy", flag.ContinueOnError)
+		var id int
+		var tag string
+		var tap string
+		fs.IntVar(&id, "id", 0, "clone ID")
+		fs.StringVar(&tag, "tag", "default", "Golden VM tag")
+		fs.StringVar(&tap, "tap", "", "TAP device name")
+		if err := fs.Parse(os.Args[2:]); err != nil {
+			fmt.Fprintf(os.Stderr, "console-proxy: %v\n", err)
+			os.Exit(1)
+		}
+		fatal(orchestrator.RunConsoleProxy(orchestrator.DefaultConfig(), id, tap, tag))
 	default:
 		usage()
 		os.Exit(1)
@@ -61,12 +133,17 @@ func main() {
 }
 
 func usage() {
-	fmt.Fprintf(os.Stderr, `Usage: %s <command> [args]
+	fmt.Fprintf(os.Stderr, `Usage: %s [--dev] <command> [args]
+
+Flags:
+  --dev         log format with source file:line (e.g. file="orchestrator.go:123")
 
 Commands:
-  init          Download kernel + create Alpine rootfs
+  init [distro] Download kernel + create distro rootfs (default: alpine, options: alpine, debian, ubuntu)
-  golden        Boot golden VM → pause → snapshot
+  golden [tag] [distro] Boot golden VM → pause → snapshot (default tag: default, default distro: alpine)
-  spawn [N]     Restore N clones from golden snapshot (default: 1)
+  spawn [tag] [N] Restore N clones from golden snapshot (default tag: default, default N: 1)
+  serve [addr]  Start terminal web UI (default: :8080)
+  console <id>  Attach to the serial console of a running clone (Ctrl+] to detach)
   status        Show running clones
   kill          Kill all running VMs
   cleanup       Kill VMs + remove all state

orchestrator/config.go

@@ -2,6 +2,7 @@ package orchestrator
 
 import (
 	"os"
+	"path/filepath"
 	"strconv"
 )
 
@@ -11,14 +12,15 @@ type Config struct {
 	BaseDir    string // working directory for all state
 	Kernel     string // path to vmlinux
 	KernelURL  string // URL to download vmlinux if Kernel file is missing
-	Rootfs     string // path to base rootfs.ext4
+	CustomRootfs string // Custom path to rootfs if FC_ROOTFS is set
 	VCPUs      int64
 	MemMiB     int64
-	Bridge     string // host bridge name, or "none" to skip networking
+	Bridge       string // host bridge name, or "none" to skip networking
-	BridgeCIDR string // e.g. "172.30.0.1/24"
+	BridgeCIDR   string // e.g. "172.30.0.1/24"
-	GuestPrefix string // e.g. "172.30.0" — clones get .10, .11, ...
+	GuestPrefix  string // e.g. "172.30.0" — clones get .11, .12, ...
-	GuestGW    string
+	GuestGW      string // default gateway for guest VMs
-	BootArgs   string
+	AutoNetConfig bool   // inject guest IP/GW/DNS via MMDS on clone start
+	BootArgs     string
 }
 
 func DefaultConfig() Config {
@@ -29,17 +31,26 @@ func DefaultConfig() Config {
 		MemMiB:      envOrInt("FC_MEM_MIB", 128),
 		Bridge:      envOr("FC_BRIDGE", "fcbr0"),
 		BridgeCIDR:  envOr("FC_BRIDGE_CIDR", "172.30.0.1/24"),
-		GuestPrefix: envOr("FC_GUEST_PREFIX", "172.30.0"),
+		GuestPrefix:  envOr("FC_GUEST_PREFIX", "172.30.0"),
-		GuestGW:     envOr("FC_GUEST_GW", "172.30.0.1"),
+		GuestGW:      envOr("FC_GUEST_GW", "172.30.0.1"),
-		BootArgs:    "console=ttyS0 reboot=k panic=1 pci=off i8042.noaux quiet loglevel=0",
+		AutoNetConfig: envOr("FC_AUTO_NET_CONFIG", "") == "1",
+		BootArgs:     "console=ttyS0 reboot=k panic=1 pci=off i8042.noaux",
 	}
 	c.Kernel = envOr("FC_KERNEL", c.BaseDir+"/vmlinux")
 	c.KernelURL = envOr("FC_KERNEL_URL",
 		"https://s3.amazonaws.com/spec.ccfc.min/firecracker-ci/20260408-ce2a467895c1-0/x86_64/vmlinux-6.1.166")
-	c.Rootfs = envOr("FC_ROOTFS", c.BaseDir+"/rootfs.ext4")
+	c.CustomRootfs = os.Getenv("FC_ROOTFS")
 	return c
 }
 
+// RootfsPath returns the path to the root filesystem depending on the requested distribution.
+func (c Config) RootfsPath(distro string) string {
+	if c.CustomRootfs != "" {
+		return c.CustomRootfs
+	}
+	return filepath.Join(c.BaseDir, "rootfs-"+distro+".ext4")
+}
+
 func envOr(key, fallback string) string {
 	if v := os.Getenv(key); v != "" {
 		return v

orchestrator/console.go

@@ -0,0 +1,353 @@
+package orchestrator
+
+import (
+	"bufio"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"sync"
+	"syscall"
+	"time"
+
+	"github.com/creack/pty"
+	firecracker "github.com/firecracker-microvm/firecracker-go-sdk"
+	"github.com/firecracker-microvm/firecracker-go-sdk/client/models"
+	log "github.com/sirupsen/logrus"
+)
+
+// RunConsoleProxy is the entrypoint for the "_console-proxy" internal subcommand.
+// It restores a Firecracker clone from the golden snapshot, connecting its serial
+// console (ttyS0) to a PTY, then serves the PTY master on a Unix socket at
+// {cloneDir}/console.sock for the lifetime of the VM.
+func RunConsoleProxy(cfg Config, id int, tapName, tag string) error {
+	logger := log.WithField("component", fmt.Sprintf("console-proxy[%d]", id))
+
+	cloneDir := filepath.Join(cfg.BaseDir, "clones", strconv.Itoa(id))
+	goldenDir := filepath.Join(cfg.BaseDir, "golden", tag)
+	sockPath := filepath.Join(cloneDir, "api.sock")
+	consoleSockPath := filepath.Join(cloneDir, "console.sock")
+	sharedMem := filepath.Join(goldenDir, "mem")
+	cloneVmstate := filepath.Join(cloneDir, "vmstate")
+
+	// --- Create PTY ---
+	// ptm = master (we hold and proxy), pts = slave (firecracker's stdio)
+	ptm, pts, err := pty.Open()
+	if err != nil {
+		return fmt.Errorf("open pty: %w", err)
+	}
+	pty.Setsize(ptm, &pty.Winsize{Rows: 24, Cols: 80}) //nolint:errcheck
+
+	fcBin, err := exec.LookPath(cfg.FCBin)
+	if err != nil {
+		pts.Close()
+		ptm.Close()
+		return fmt.Errorf("firecracker not found: %w", err)
+	}
+
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	// --- Build firecracker command with slave PTY as stdin/stdout/stderr ---
+	// Setsid makes the slave PTY the controlling terminal for firecracker,
+	// which is required for job control (Ctrl+C, SIGWINCH, etc.) to work.
+	cmd := firecracker.VMCommandBuilder{}.
+		WithBin(fcBin).
+		WithSocketPath(sockPath).
+		Build(ctx)
+	cmd.Stdin = pts
+	cmd.Stdout = pts
+	cmd.Stderr = pts
+	cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
+
+	// --- Build Machine config (mirrors spawnOne) ---
+	vcpus := cfg.VCPUs
+	mem := cfg.MemMiB
+
+	fcCfg := firecracker.Config{
+		SocketPath: sockPath,
+		MachineCfg: models.MachineConfiguration{
+			VcpuCount:  &vcpus,
+			MemSizeMib: &mem,
+		},
+		LogPath:       sockPath + ".log",
+		LogLevel:      "Debug",
+		FifoLogWriter: logger.Writer(),
+	}
+
+	if cfg.Bridge != "none" && tapName != "" {
+		mac := fmt.Sprintf("AA:FC:00:00:%02X:%02X", id/256, id%256)
+		fcCfg.NetworkInterfaces = firecracker.NetworkInterfaces{
+			{
+				StaticConfiguration: &firecracker.StaticNetworkConfiguration{
+					MacAddress:  mac,
+					HostDevName: tapName,
+				},
+				AllowMMDS: true,
+			},
+		}
+	}
+
+	m, err := firecracker.NewMachine(ctx, fcCfg,
+		firecracker.WithProcessRunner(cmd),
+		firecracker.WithLogger(logger),
+		firecracker.WithSnapshot(sharedMem, cloneVmstate, func(sc *firecracker.SnapshotConfig) {
+			sc.ResumeVM = true
+		}),
+	)
+	if err != nil {
+		pts.Close()
+		ptm.Close()
+		return fmt.Errorf("new machine: %w", err)
+	}
+
+	// Swap in network-override snapshot handler (same as spawnOne)
+	if cfg.Bridge != "none" && tapName != "" {
+		m.Handlers.FcInit = m.Handlers.FcInit.Swap(firecracker.Handler{
+			Name: firecracker.LoadSnapshotHandlerName,
+			Fn: func(ctx context.Context, m *firecracker.Machine) error {
+				return loadSnapshotWithNetworkOverride(
+					ctx, sockPath, sharedMem, cloneVmstate, tapName,
+				)
+			},
+		})
+	}
+
+	// --- Start VM (blocks until snapshot is loaded and VM is PAUSED) ---
+	start := time.Now()
+	logger.Infof("restoring clone %d from snapshot ...", id)
+	if err := m.Start(ctx); err != nil {
+		pts.Close()
+		ptm.Close()
+		return fmt.Errorf("restore clone %d: %w", id, err)
+	}
+
+	// Inject per-clone IP config via MMDS so the fc-net-init guest daemon
+	// can configure eth0 without any manual steps inside the VM.
+	// This must happen while the VM is PAUSED (ResumeVM: false in snapshot load).
+	if cfg.AutoNetConfig && cfg.Bridge != "none" {
+		guestIP := fmt.Sprintf("%s.%d/24", cfg.GuestPrefix, 10+id)
+		if err := configureMmds(ctx, sockPath, guestIP, cfg.GuestGW, "1.1.1.1"); err != nil {
+			logger.Warnf("MMDS config failed (guest network will be unconfigured): %v", err)
+		} else {
+			logger.Infof("MMDS: assigned %s gw %s to clone %d", guestIP, cfg.GuestGW, id)
+		}
+	}
+
+	// Now RESUME the VM to start execution!
+	if err := m.ResumeVM(ctx); err != nil {
+		pts.Close()
+		ptm.Close()
+		return fmt.Errorf("resume clone %d: %w", id, err)
+	}
+	elapsed := time.Since(start)
+
+	// Release our copy of the slave — firecracker holds its own fd now.
+	// Closing here ensures we get EOF on ptm when firecracker exits.
+	pts.Close()
+
+	// --- Write PID file ---
+	pidsDir := filepath.Join(cfg.BaseDir, "pids")
+	os.MkdirAll(pidsDir, 0o755) //nolint:errcheck
+	if cmd.Process != nil {
+		os.WriteFile( //nolint:errcheck
+			filepath.Join(pidsDir, fmt.Sprintf("clone-%d.pid", id)),
+			[]byte(strconv.Itoa(cmd.Process.Pid)),
+			0o644,
+		)
+	}
+	logger.Infof("clone %d: restored in %s (pid=%d, tap=%s)",
+		id, elapsed.Round(time.Millisecond), cmd.Process.Pid, tapName)
+
+	// --- Open console log (captures all serial output from boot) ---
+	consoleLogPath := filepath.Join(cloneDir, "console.log")
+	consoleLog, err := os.OpenFile(consoleLogPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644)
+	if err != nil {
+		logger.Warnf("could not open console log: %v", err)
+		consoleLog = nil
+	}
+	if consoleLog != nil {
+		defer consoleLog.Close()
+	}
+
+	// --- Create console socket ---
+	os.Remove(consoleSockPath) //nolint:errcheck
+	listener, err := net.Listen("unix", consoleSockPath)
+	if err != nil {
+		ptm.Close()
+		return fmt.Errorf("listen on console socket: %w", err)
+	}
+
+	// --- Create resize sideband socket ---
+	resizeSockPath := filepath.Join(cloneDir, "console-resize.sock")
+	os.Remove(resizeSockPath) //nolint:errcheck
+	resizeListener, err := net.Listen("unix", resizeSockPath)
+	if err != nil {
+		logger.Warnf("could not create resize socket, terminal resize will be unavailable: %v", err)
+		resizeListener = nil
+	}
+
+	// --- Serve until VM exits ---
+	vmDone := make(chan struct{})
+	go func() {
+		m.Wait(ctx) //nolint:errcheck
+		close(vmDone)
+	}()
+
+	if resizeListener != nil {
+		go serveResize(resizeListener, ptm, vmDone, logger)
+	}
+	serveConsole(listener, ptm, consoleLog, vmDone, logger)
+
+	listener.Close()
+	if resizeListener != nil {
+		resizeListener.Close()
+	}
+	ptm.Close()
+	os.Remove(consoleSockPath) //nolint:errcheck
+	os.Remove(resizeSockPath)  //nolint:errcheck
+
+	if tapName != "" {
+		destroyTap(tapName)
+	}
+
+	logger.Infof("clone %d: exiting", id)
+	return nil
+}
+
+// resizeMsg is the JSON payload sent over the resize sideband socket.
+type resizeMsg struct {
+	Rows uint16 `json:"rows"`
+	Cols uint16 `json:"cols"`
+}
+
+// serveResize accepts connections on the resize sideband listener and applies
+// PTY window size changes as JSON resize messages arrive.
+func serveResize(listener net.Listener, ptm *os.File, vmDone <-chan struct{}, logger *log.Entry) {
+	go func() {
+		<-vmDone
+		listener.Close()
+	}()
+	for {
+		conn, err := listener.Accept()
+		if err != nil {
+			return
+		}
+		go handleResize(conn, ptm, logger)
+	}
+}
+
+// handleResize reads newline-delimited JSON resize messages from conn and
+// applies each one to the PTY master.
+func handleResize(conn net.Conn, ptm *os.File, logger *log.Entry) {
+	defer conn.Close()
+	scanner := bufio.NewScanner(conn)
+	for scanner.Scan() {
+		var msg resizeMsg
+		if err := json.Unmarshal(scanner.Bytes(), &msg); err != nil {
+			logger.Warnf("resize: bad message: %v", err)
+			continue
+		}
+		if msg.Rows == 0 || msg.Cols == 0 {
+			continue
+		}
+		if err := pty.Setsize(ptm, &pty.Winsize{Rows: msg.Rows, Cols: msg.Cols}); err != nil {
+			logger.Warnf("resize pty: %v", err)
+		}
+	}
+}
+
+// atomicWriter wraps an io.Writer behind a read/write lock so it can be
+// swapped between connections without holding the lock during writes.
+type atomicWriter struct {
+	mu sync.RWMutex
+	w  io.Writer
+}
+
+func (a *atomicWriter) set(w io.Writer) {
+	a.mu.Lock()
+	defer a.mu.Unlock()
+	a.w = w
+}
+
+func (a *atomicWriter) Write(p []byte) (int, error) {
+	a.mu.RLock()
+	defer a.mu.RUnlock()
+	return a.w.Write(p)
+}
+
+// serveConsole accepts connections on listener and proxies console I/O via ptm.
+// A background goroutine reads from the PTY master continuously (discarding
+// output when no client is connected so the VM never blocks on a full buffer).
+// Only one client is served at a time; sessions are serialised.
+func serveConsole(listener net.Listener, ptm *os.File, logFile *os.File, vmDone <-chan struct{}, logger *log.Entry) {
+	aw := &atomicWriter{w: io.Discard}
+
+	// Background PTY reader — runs for the full VM lifetime.
+	// All output is tee'd to logFile (if set) so boot messages are never lost.
+	go func() {
+		buf := make([]byte, 4096)
+		for {
+			n, err := ptm.Read(buf)
+			if n > 0 {
+				aw.Write(buf[:n]) //nolint:errcheck
+				if logFile != nil {
+					logFile.Write(buf[:n]) //nolint:errcheck
+				}
+			}
+			if err != nil {
+				return // PTY closed (VM exited)
+			}
+		}
+	}()
+
+	// Unblock Accept() when the VM exits.
+	go func() {
+		<-vmDone
+		listener.Close()
+	}()
+
+	for {
+		conn, err := listener.Accept()
+		if err != nil {
+			return // listener closed
+		}
+
+		logger.Info("console client connected")
+
+		// Route PTY output to this connection.
+		aw.set(conn)
+
+		// Forward client input to the PTY master.
+		clientGone := make(chan struct{})
+		go func() {
+			defer close(clientGone)
+			buf := make([]byte, 256)
+			for {
+				n, err := conn.Read(buf)
+				if n > 0 {
+					ptm.Write(buf[:n]) //nolint:errcheck
+				}
+				if err != nil {
+					return
+				}
+			}
+		}()
+
+		// Block until client disconnects or VM dies.
+		select {
+		case <-clientGone:
+			logger.Info("console client disconnected")
+		case <-vmDone:
+			logger.Info("VM exited while console client was connected")
+		}
+
+		aw.set(io.Discard)
+		conn.Close()
+	}
+}

orchestrator/console_client.go

@@ -0,0 +1,104 @@
+package orchestrator
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"net"
+	"os"
+	"os/signal"
+	"path/filepath"
+	"strconv"
+	"sync"
+	"syscall"
+
+	"golang.org/x/term"
+)
+
+// ConnectConsole attaches the calling terminal to the serial console of the
+// clone with the given ID. The connection is made over the Unix socket at
+// {cloneDir}/console.sock served by the console-proxy process.
+//
+// Escape sequence: Ctrl+] (byte 0x1D) detaches without stopping the VM.
+func ConnectConsole(cfg Config, id int) error {
+	sockPath := filepath.Join(cfg.BaseDir, "clones", strconv.Itoa(id), "console.sock")
+	if _, err := os.Stat(sockPath); err != nil {
+		return fmt.Errorf("clone %d has no console socket — is it running?", id)
+	}
+
+	conn, err := net.Dial("unix", sockPath)
+	if err != nil {
+		return fmt.Errorf("connect to console: %w", err)
+	}
+	defer conn.Close()
+
+	if !term.IsTerminal(int(os.Stdin.Fd())) {
+		return fmt.Errorf("console requires an interactive terminal")
+	}
+
+	oldState, err := term.MakeRaw(int(os.Stdin.Fd()))
+	if err != nil {
+		return fmt.Errorf("set raw terminal: %w", err)
+	}
+	defer term.Restore(int(os.Stdin.Fd()), oldState) //nolint:errcheck
+
+	// Ensure the terminal is restored even on SIGTERM / SIGHUP.
+	sigCh := make(chan os.Signal, 1)
+	signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGHUP)
+	go func() {
+		<-sigCh
+		term.Restore(int(os.Stdin.Fd()), oldState) //nolint:errcheck
+		os.Exit(0)
+	}()
+
+	fmt.Fprintf(os.Stderr, "Connected to clone %d console. Escape: Ctrl+]\r\n", id)
+
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	var wg sync.WaitGroup
+	wg.Add(2)
+
+	// stdin → VM (with Ctrl+] escape detection)
+	go func() {
+		defer wg.Done()
+		io.Copy(conn, &escapeReader{r: os.Stdin, cancel: cancel}) //nolint:errcheck
+		// Half-close so the proxy knows we're done sending.
+		if uc, ok := conn.(*net.UnixConn); ok {
+			uc.CloseWrite() //nolint:errcheck
+		}
+	}()
+
+	// VM → stdout
+	go func() {
+		defer wg.Done()
+		io.Copy(os.Stdout, conn) //nolint:errcheck
+		cancel()                 // VM exited or proxy closed the connection
+	}()
+
+	<-ctx.Done()
+	conn.Close() // unblock both goroutines
+	wg.Wait()
+
+	fmt.Fprintf(os.Stderr, "\r\nDetached from clone %d.\r\n", id)
+	return nil
+}
+
+// escapeReader wraps an io.Reader and intercepts Ctrl+] (0x1D), calling cancel
+// and returning io.EOF when the escape byte is seen.
+type escapeReader struct {
+	r      io.Reader
+	cancel context.CancelFunc
+}
+
+func (e *escapeReader) Read(p []byte) (int, error) {
+	n, err := e.r.Read(p)
+	for i := 0; i < n; i++ {
+		if p[i] == 0x1D { // Ctrl+]
+			e.cancel()
+			// Return only the bytes before the escape so nothing leaks to the VM.
+			return i, io.EOF
+		}
+	}
+	return n, err
+}

orchestrator/network.go

@@ -1,7 +1,13 @@
 package orchestrator
 
 import (
+	"bytes"
+	"context"
+	"encoding/json"
 	"fmt"
+	"io"
+	"net"
+	"net/http"
 	"os/exec"
 	"strings"
 )
@@ -48,6 +54,8 @@ func (o *Orchestrator) setupBridge() error {
 
 // createTap creates a tap device and attaches it to the bridge.
 func (o *Orchestrator) createTap(name string) error {
+	// Destroy any stale tap with this name before (re)creating it.
+	_ = run("ip", "link", "del", name)
 	if err := run("ip", "tuntap", "add", "dev", name, "mode", "tap"); err != nil {
 		return fmt.Errorf("create tap %s: %w", name, err)
 	}
@@ -66,6 +74,57 @@ func destroyTap(name string) {
 	_ = run("ip", "link", "del", name)
 }
 
+// configureMmds writes per-clone IP config to the Firecracker MMDS so that
+// the fc-net-init daemon running inside the guest can read and apply it.
+// It makes two API calls to the Firecracker Unix socket:
+//
+//  1. PUT /mmds/config — associates MMDS with the guest's first NIC ("1")
+//  2. PUT /mmds        — stores ip/gw/dns values the guest daemon will read
+func configureMmds(ctx context.Context, sockPath, ip, gw, dns string) error {
+	httpClient := &http.Client{
+		Transport: &http.Transport{
+			DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+				return net.Dial("unix", sockPath)
+			},
+		},
+	}
+
+	doJSON := func(method, path string, body any) error {
+		data, err := json.Marshal(body)
+		if err != nil {
+			return fmt.Errorf("marshal %s: %w", path, err)
+		}
+		req, err := http.NewRequestWithContext(ctx, method,
+			"http://localhost"+path, bytes.NewReader(data))
+		if err != nil {
+			return fmt.Errorf("build request %s: %w", path, err)
+		}
+		req.Header.Set("Content-Type", "application/json")
+		resp, err := httpClient.Do(req)
+		if err != nil {
+			return fmt.Errorf("%s %s: %w", method, path, err)
+		}
+		defer resp.Body.Close()
+		if resp.StatusCode != http.StatusNoContent {
+			b, _ := io.ReadAll(resp.Body)
+			return fmt.Errorf("%s %s failed (%d): %s", method, path, resp.StatusCode, b)
+		}
+		return nil
+	}
+
+	// 1. MMDS configuration (version, network_interfaces binding, etc.) is
+	// persisted in the golden snapshot, so we don't need to configure it here.
+	// In fact, Firecracker will reject PUT /mmds/config with a 400 error
+	// on a restored VM, which previously caused this function to abort early.
+
+	// 2. Store the network config the guest daemon will poll for.
+	return doJSON(http.MethodPut, "/mmds", map[string]string{
+		"ip":  ip,
+		"gw":  gw,
+		"dns": dns,
+	})
+}
+
 // run executes a command, returning an error if it fails.
 func run(name string, args ...string) error {
 	return exec.Command(name, args...).Run()

orchestrator/orchestrator.go

@@ -11,6 +11,7 @@ import (
 	"strconv"
 	"strings"
 	"sync"
+	"syscall"
 	"time"
 
 	firecracker "github.com/firecracker-microvm/firecracker-go-sdk"
@@ -42,13 +43,13 @@ func New(cfg Config) *Orchestrator {
 	}
 }
 
-func (o *Orchestrator) goldenDir() string { return filepath.Join(o.cfg.BaseDir, "golden") }
+func (o *Orchestrator) goldenDir(tag string) string { return filepath.Join(o.cfg.BaseDir, "golden", tag) }
 func (o *Orchestrator) clonesDir() string { return filepath.Join(o.cfg.BaseDir, "clones") }
 func (o *Orchestrator) pidsDir() string   { return filepath.Join(o.cfg.BaseDir, "pids") }
 
 // ——— Init ————————————————————————————————————————————————————————————————
 
-func (o *Orchestrator) Init() error {
+func (o *Orchestrator) Init(distro string) error {
 	if err := os.MkdirAll(o.cfg.BaseDir, 0o755); err != nil {
 		return err
 	}
@@ -64,92 +65,237 @@ func (o *Orchestrator) Init() error {
 	}
 
 	// Build rootfs if missing
-	if _, err := os.Stat(o.cfg.Rootfs); os.IsNotExist(err) {
+	rootfsPath := o.cfg.RootfsPath(distro)
-		o.log.Info("building minimal Alpine rootfs ...")
+	if _, err := os.Stat(rootfsPath); os.IsNotExist(err) {
-		if err := o.buildRootfs(); err != nil {
+		o.log.Infof("building minimal %s rootfs ...", distro)
+		if err := o.buildRootfs(distro, rootfsPath); err != nil {
 			return fmt.Errorf("build rootfs: %w", err)
 		}
-		o.log.Infof("rootfs saved to %s", o.cfg.Rootfs)
+		o.log.Infof("rootfs saved to %s", rootfsPath)
 	}
 
 	o.log.Info("init complete")
 	return nil
 }
 
-func (o *Orchestrator) buildRootfs() error {
+func (o *Orchestrator) buildRootfs(distro, rootfsPath string) error {
 	sizeMB := 512
+	if distro == "debian" || distro == "ubuntu" {
+		sizeMB = 2048
+	}
 	mnt := filepath.Join(o.cfg.BaseDir, "mnt")
 
 	// create empty ext4 image
-	o.log.Infof("running: dd if=/dev/zero of=%s bs=1M count=%d status=none", o.cfg.Rootfs, sizeMB)
+	o.log.Infof("running: dd if=/dev/zero of=%s bs=1M count=%d status=none", rootfsPath, sizeMB)
-	if err := run("dd", "if=/dev/zero", "of="+o.cfg.Rootfs,
+	if err := run("dd", "if=/dev/zero", "of="+rootfsPath,
 		"bs=1M", fmt.Sprintf("count=%d", sizeMB), "status=none"); err != nil {
 		return err
 	}
-	o.log.Infof("running: mkfs.ext4 -qF %s", o.cfg.Rootfs)
+	o.log.Infof("running: mkfs.ext4 -qF %s", rootfsPath)
-	if err := run("mkfs.ext4", "-qF", o.cfg.Rootfs); err != nil {
+	if err := run("mkfs.ext4", "-qF", rootfsPath); err != nil {
 		return err
 	}
 
 	os.MkdirAll(mnt, 0o755)
-	o.log.Infof("running: mount -o loop %s %s", o.cfg.Rootfs, mnt)
+	o.log.Infof("running: mount -o loop %s %s", rootfsPath, mnt)
-	if err := run("mount", "-o", "loop", o.cfg.Rootfs, mnt); err != nil {
+	if err := run("mount", "-o", "loop", rootfsPath, mnt); err != nil {
 		return err
 	}
-	defer run("umount", mnt)
+	defer func() {
+		o.log.Infof("running: umount %s", mnt)
+		run("umount", mnt)
+	}()
 
-	// download and extract Alpine minirootfs
+	// download and extract minirootfs
-	alpineVer := "3.20"
+	switch distro {
-	arch := "x86_64"
+	case "alpine":
-	tarball := fmt.Sprintf("alpine-minirootfs-%s.0-%s.tar.gz", alpineVer, arch)
+		alpineVer := "3.20"
-	url := fmt.Sprintf("https://dl-cdn.alpinelinux.org/alpine/v%s/releases/%s/%s",
+		arch := "x86_64"
-		alpineVer, arch, tarball)
+		tarball := fmt.Sprintf("alpine-minirootfs-%s.0-%s.tar.gz", alpineVer, arch)
-
+		url := fmt.Sprintf("https://dl-cdn.alpinelinux.org/alpine/v%s/releases/%s/%s",
-	tarPath := filepath.Join(o.cfg.BaseDir, tarball)
+			alpineVer, arch, tarball)
-	if err := downloadFile(url, tarPath); err != nil {
+		tarPath := filepath.Join(o.cfg.BaseDir, tarball)
-		return fmt.Errorf("download alpine: %w", err)
+		o.log.Infof("downloading http request: GET %s to %s", url, tarPath)
+		if err := downloadFile(url, tarPath); err != nil {
+			return fmt.Errorf("download alpine: %w", err)
+		}
+		o.log.Infof("running: tar xzf %s -C %s", tarPath, mnt)
+		if err := run("tar", "xzf", tarPath, "-C", mnt); err != nil {
+			return err
+		}
+	case "debian":
+		tarball := "debian-12-nocloud-amd64.tar.xz"
+		url := "https://cloud.debian.org/images/cloud/bookworm/latest/" + tarball
+		tarPath := filepath.Join(o.cfg.BaseDir, tarball)
+		o.log.Infof("downloading http request: GET %s to %s", url, tarPath)
+		if err := downloadFile(url, tarPath); err != nil {
+			return fmt.Errorf("download debian: %w", err)
+		}
+		o.log.Infof("running: tar xJf %s -C %s", tarPath, mnt)
+		if err := run("tar", "xJf", tarPath, "-C", mnt); err != nil {
+			return err
+		}
+	case "ubuntu":
+		tarball := "ubuntu-base-24.04.4-base-amd64.tar.gz"
+		url := "https://cdimage.ubuntu.com/ubuntu-base/releases/24.04/release/" + tarball
+		tarPath := filepath.Join(o.cfg.BaseDir, tarball)
+		o.log.Infof("downloading http request: GET %s to %s", url, tarPath)
+		if err := downloadFile(url, tarPath); err != nil {
+			return fmt.Errorf("download ubuntu: %w", err)
+		}
+		o.log.Infof("running: tar xzf %s -C %s", tarPath, mnt)
+		if err := run("tar", "xzf", tarPath, "-C", mnt); err != nil {
+			return err
+		}
+		o.log.Info("installing essential packages in ubuntu chroot ...")
+		if err := installUbuntuPackages(mnt, o.log); err != nil {
+			return fmt.Errorf("install ubuntu packages: %w", err)
+		}
+	default:
+		return fmt.Errorf("unsupported distro: %s", distro)
 	}
-	o.log.Infof("running: tar xzf %s -C %s", tarPath, mnt)
+
-	if err := run("tar", "xzf", tarPath, "-C", mnt); err != nil {
+	// write fc-net-init daemon: polls MMDS for IP config and applies it.
+	// Always embedded — harmless if MMDS is never populated (sleeps 1 s/loop).
+	// Captured in the golden snapshot so it runs on every clone resume too.
+	netInitScript := `#!/bin/sh
+# Poll Firecracker MMDS for network config, apply it, then exit.
+# Runs in background; loops until MMDS responds (survives snapshot resume).
+ip link set eth0 up 2>/dev/null
+ip route add 169.254.169.254 dev eth0 2>/dev/null
+ip addr add 169.254.169.2/32 dev eth0 2>/dev/null
+while true; do
+  ip=$(wget -q -T1 -O- http://169.254.169.254/ip 2>/dev/null | tr -d '"')
+  [ -n "$ip" ] || { sleep 1; continue; }
+  gw=$(wget -q -T1 -O- http://169.254.169.254/gw 2>/dev/null | tr -d '"')
+  dns=$(wget -q -T1 -O- http://169.254.169.254/dns 2>/dev/null | tr -d '"')
+  ip addr flush dev eth0 2>/dev/null
+  ip addr add "$ip" dev eth0 2>/dev/null
+  ip route add default via "$gw" dev eth0 2>/dev/null
+  printf "nameserver %s\n" "$dns" > /etc/resolv.conf
+  break
+done
+`
+	os.MkdirAll(filepath.Join(mnt, "sbin"), 0o755)
+	if err := os.WriteFile(filepath.Join(mnt, "sbin", "fc-net-init"), []byte(netInitScript), 0o755); err != nil {
 		return err
 	}
 
-	// write init script
+	if distro == "alpine" {
-	initScript := `#!/bin/sh
+		// write init script
+		initScript := `#!/bin/sh
 mount -t proc proc /proc
 mount -t sysfs sys /sys
 mount -t devtmpfs devtmpfs /dev
 ip link set eth0 up 2>/dev/null
+ip route add 169.254.169.254 dev eth0 2>/dev/null
+/sbin/fc-net-init &
 `
-	initPath := filepath.Join(mnt, "etc", "init.d", "rcS")
+		initPath := filepath.Join(mnt, "etc", "init.d", "rcS")
-	os.MkdirAll(filepath.Dir(initPath), 0o755)
+		os.MkdirAll(filepath.Dir(initPath), 0o755)
-	if err := os.WriteFile(initPath, []byte(initScript), 0o755); err != nil {
+		if err := os.WriteFile(initPath, []byte(initScript), 0o755); err != nil {
-		return err
+			return err
-	}
+		}
 
-	// write inittab
+		// write inittab
-	inittab := "::sysinit:/etc/init.d/rcS\nttyS0::respawn:/bin/sh\n"
+		inittab := "::sysinit:/etc/init.d/rcS\nttyS0::respawn:/bin/sh\n"
-	return os.WriteFile(filepath.Join(mnt, "etc", "inittab"), []byte(inittab), 0o644)
+		return os.WriteFile(filepath.Join(mnt, "etc", "inittab"), []byte(inittab), 0o644)
+	} else {
+		// systemd-based distributions (Debian, Ubuntu)
+		svc := `[Unit]
+Description=Firecracker Network Init
+After=basic.target
+
+[Service]
+Type=simple
+ExecStart=/sbin/fc-net-init
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
+`
+		svcPath := filepath.Join(mnt, "etc", "systemd", "system", "fc-net-init.service")
+		os.MkdirAll(filepath.Dir(svcPath), 0o755)
+		if err := os.WriteFile(svcPath, []byte(svc), 0o644); err != nil {
+			return err
+		}
+
+		// Enable service dynamically
+		wantsDir := filepath.Join(mnt, "etc", "systemd", "system", "multi-user.target.wants")
+		os.MkdirAll(wantsDir, 0o755)
+		os.Symlink("/etc/systemd/system/fc-net-init.service", filepath.Join(wantsDir, "fc-net-init.service")) //nolint:errcheck
+
+		// Mask serial-getty@ttyS0.service and the udev device unit it depends on.
+		// In Firecracker, udev never runs so dev-ttyS0.device never activates,
+		// causing a 90-second systemd timeout. We replace it entirely with a
+		// custom service that uses ConditionPathExists (filesystem check) instead.
+		systemdDir := filepath.Join(mnt, "etc", "systemd", "system")
+		os.Symlink("/dev/null", filepath.Join(systemdDir, "serial-getty@ttyS0.service")) //nolint:errcheck
+		os.Symlink("/dev/null", filepath.Join(systemdDir, "dev-ttyS0.device"))          //nolint:errcheck
+
+		// Custom console service: no udev dependency, autologin as root.
+		consoleSvc := `[Unit]
+Description=Serial Console (ttyS0)
+After=basic.target
+ConditionPathExists=/dev/ttyS0
+
+[Service]
+ExecStart=/sbin/agetty --autologin root --noclear ttyS0 vt220
+Restart=always
+RestartSec=1
+
+[Install]
+WantedBy=multi-user.target
+`
+		consoleSvcPath := filepath.Join(systemdDir, "fc-console.service")
+		os.WriteFile(consoleSvcPath, []byte(consoleSvc), 0o644) //nolint:errcheck
+		wantsDir2 := filepath.Join(systemdDir, "multi-user.target.wants")
+		os.MkdirAll(wantsDir2, 0o755)
+		os.Symlink("/etc/systemd/system/fc-console.service", filepath.Join(wantsDir2, "fc-console.service")) //nolint:errcheck
+
+		// Clear root password for auto-login on console
+		shadowPath := filepath.Join(mnt, "etc", "shadow")
+		if shadowBytes, err := os.ReadFile(shadowPath); err == nil {
+			lines := strings.Split(string(shadowBytes), "\n")
+			for i, line := range lines {
+				if strings.HasPrefix(line, "root:") {
+					parts := strings.Split(line, ":")
+					if len(parts) > 1 {
+						parts[1] = ""
+						lines[i] = strings.Join(parts, ":")
+					}
+				}
+			}
+			os.WriteFile(shadowPath, []byte(strings.Join(lines, "\n")), 0o640) //nolint:errcheck
+		}
+
+		// Write fstab so systemd mounts virtual filesystems at boot.
+		// Minimal tarball rootfs has no fstab; without it /proc, /sys, /dev are not mounted.
+		fstab := "proc\t/proc\tproc\tdefaults\t0 0\nsysfs\t/sys\tsysfs\tdefaults\t0 0\ndevtmpfs\t/dev\tdevtmpfs\tdefaults\t0 0\n"
+		os.WriteFile(filepath.Join(mnt, "etc", "fstab"), []byte(fstab), 0o644) //nolint:errcheck
+	}
+	return nil
 }
 
 // ——— Golden VM ——————————————————————————————————————————————————————————
 
-func (o *Orchestrator) Golden() error {
+func (o *Orchestrator) Golden(tag string, distro string) error {
 	if _, err := os.Stat(o.cfg.Kernel); err != nil {
 		return fmt.Errorf("kernel not found — run init first: %w", err)
 	}
-	if _, err := os.Stat(o.cfg.Rootfs); err != nil {
+	rootfsPath := o.cfg.RootfsPath(distro)
+	if _, err := os.Stat(rootfsPath); err != nil {
 		return fmt.Errorf("rootfs not found — run init first: %w", err)
 	}
 
-	goldenDir := o.goldenDir()
+	goldenDir := o.goldenDir(tag)
 	os.RemoveAll(goldenDir)
 	os.MkdirAll(goldenDir, 0o755)
 	os.MkdirAll(o.pidsDir(), 0o755)
 
 	// COW copy of rootfs for golden VM
 	goldenRootfs := filepath.Join(goldenDir, "rootfs.ext4")
-	if err := reflinkCopy(o.cfg.Rootfs, goldenRootfs); err != nil {
+	if err := reflinkCopy(rootfsPath, goldenRootfs); err != nil {
 		return fmt.Errorf("copy rootfs: %w", err)
 	}
 
@@ -174,6 +320,7 @@ func (o *Orchestrator) Golden() error {
 					MacAddress:  "AA:FC:00:00:00:01",
 					HostDevName: tap,
 				},
+				AllowMMDS: true,
 			},
 		}
 	}
@@ -203,6 +350,9 @@ func (o *Orchestrator) Golden() error {
 			},
 		},
 		NetworkInterfaces: netIfaces,
+		LogPath:           sockPath + ".log",
+		LogLevel:          "Debug",
+		FifoLogWriter:     o.log.Writer(),
 	}
 
 	ctx, cancel := context.WithCancel(context.Background())
@@ -218,7 +368,10 @@ func (o *Orchestrator) Golden() error {
 		WithSocketPath(sockPath).
 		Build(ctx)
 
-	m, err := firecracker.NewMachine(ctx, fcCfg, firecracker.WithProcessRunner(cmd))
+	m, err := firecracker.NewMachine(ctx, fcCfg,
+		firecracker.WithProcessRunner(cmd),
+		firecracker.WithLogger(o.log),
+	)
 	if err != nil {
 		return fmt.Errorf("new machine: %w", err)
 	}
@@ -234,8 +387,15 @@ func (o *Orchestrator) Golden() error {
 			[]byte(fmt.Sprintf("%d", cmd.Process.Pid)), 0o644)
 	}
 
-	o.log.Info("golden VM booted, letting it settle ...")
+	settleTime := 3 * time.Second
-	time.Sleep(3 * time.Second)
+	if distro == "debian" || distro == "ubuntu" {
+		// systemd takes significantly longer to reach multi-user.target than
+		// Alpine's busybox init. Snapshot too early and serial-getty@ttyS0
+		// won't have started yet, leaving the console unresponsive on resume.
+		settleTime = 20 * time.Second
+	}
+	o.log.Infof("golden VM booted, letting it settle (%s) ...", settleTime)
+	time.Sleep(settleTime)
 
 	// pause
 	o.log.Info("pausing golden VM ...")
@@ -267,13 +427,29 @@ func (o *Orchestrator) Golden() error {
 	return nil
 }
 
+// GoldenTags returns a list of all existing golden VM tags.
+func (o *Orchestrator) GoldenTags() []string {
+	goldenDir := filepath.Join(o.cfg.BaseDir, "golden")
+	entries, err := os.ReadDir(goldenDir)
+	if err != nil {
+		return nil
+	}
+	var tags []string
+	for _, e := range entries {
+		if e.IsDir() {
+			tags = append(tags, e.Name())
+		}
+	}
+	return tags
+}
+
 // ——— Spawn clones ——————————————————————————————————————————————————————
 
-func (o *Orchestrator) Spawn(count int) error {
+func (o *Orchestrator) Spawn(count int, tag string) error {
-	goldenDir := o.goldenDir()
+	goldenDir := o.goldenDir(tag)
 	for _, f := range []string{"vmstate", "mem"} {
 		if _, err := os.Stat(filepath.Join(goldenDir, f)); err != nil {
-			return fmt.Errorf("golden %s not found — run golden first", f)
+			return fmt.Errorf("golden %s not found for tag %s — run golden first", f, tag)
 		}
 	}
 
@@ -287,7 +463,7 @@ func (o *Orchestrator) Spawn(count int) error {
 
 	for i := 0; i < count; i++ {
 		id := o.nextCloneID()
-		if err := o.spawnOne(id); err != nil {
+		if err := o.spawnOne(id, o.cfg.AutoNetConfig, tag); err != nil {
 			o.log.Errorf("clone %d failed: %v", id, err)
 			continue
 		}
@@ -298,24 +474,69 @@ func (o *Orchestrator) Spawn(count int) error {
 	return nil
 }
 
-func (o *Orchestrator) spawnOne(id int) error {
+// SpawnSingle spawns exactly one new clone and returns its ID.
-	goldenDir := o.goldenDir()
+// It is safe to call from multiple goroutines (nextCloneID is serialised by the
+// filesystem scan, and each clone gets its own directory/tap).
+// SpawnSingle spawns one clone. net controls whether the guest receives
+// automatic IP configuration via MMDS (overrides FC_AUTO_NET_CONFIG for this
+// clone). Pass cfg.AutoNetConfig to preserve the global default.
+func (o *Orchestrator) SpawnSingle(net bool, tag string) (int, error) {
+	goldenDir := o.goldenDir(tag)
+	for _, f := range []string{"vmstate", "mem"} {
+		if _, err := os.Stat(filepath.Join(goldenDir, f)); err != nil {
+			return 0, fmt.Errorf("golden %s not found for tag %s — run golden first", f, tag)
+		}
+	}
+	os.MkdirAll(o.clonesDir(), 0o755)
+	os.MkdirAll(o.pidsDir(), 0o755)
+	if o.cfg.Bridge != "none" {
+		if err := o.setupBridge(); err != nil {
+			return 0, err
+		}
+	}
+	id := o.nextCloneID()
+	if err := o.spawnOne(id, net, tag); err != nil {
+		return 0, err
+	}
+	return id, nil
+}
+
+// KillClone kills a single clone by ID: terminates its proxy process,
+// destroys its tap device, and removes its working directory.
+func (o *Orchestrator) KillClone(id int) error {
+	pidFile := filepath.Join(o.pidsDir(), fmt.Sprintf("clone-%d.proxy.pid", id))
+	if data, err := os.ReadFile(pidFile); err == nil {
+		if pid, err := strconv.Atoi(strings.TrimSpace(string(data))); err == nil {
+			if p, err := os.FindProcess(pid); err == nil {
+				_ = p.Kill()
+				o.log.Infof("clone %d: killed proxy pid %d", id, pid)
+			}
+		}
+		os.Remove(pidFile) //nolint:errcheck
+	}
+	tapName := fmt.Sprintf("fctap%d", id)
+	destroyTap(tapName)
+	os.RemoveAll(filepath.Join(o.clonesDir(), strconv.Itoa(id))) //nolint:errcheck
+	o.log.Infof("clone %d: destroyed", id)
+	return nil
+}
+
+func (o *Orchestrator) spawnOne(id int, net bool, tag string) error {
+	goldenDir := o.goldenDir(tag)
 	cloneDir := filepath.Join(o.clonesDir(), strconv.Itoa(id))
 	os.MkdirAll(cloneDir, 0o755)
+	os.WriteFile(filepath.Join(cloneDir, "tag"), []byte(tag), 0o644) //nolint:errcheck
 
 	sockPath := filepath.Join(cloneDir, "api.sock")
 	os.Remove(sockPath)
 
 	// --- COW rootfs ---
 	cloneRootfs := filepath.Join(cloneDir, "rootfs.ext4")
+	o.log.Infof("clone %d: running: cp --reflink=always %s %s", id, filepath.Join(goldenDir, "rootfs.ext4"), cloneRootfs)
 	if err := reflinkCopy(filepath.Join(goldenDir, "rootfs.ext4"), cloneRootfs); err != nil {
 		return fmt.Errorf("copy rootfs: %w", err)
 	}
 
-	// --- Memory: point at the shared golden mem file ---
-	// Firecracker uses MAP_PRIVATE → kernel COW. No copy needed.
-	sharedMem := filepath.Join(goldenDir, "mem")
-
 	// --- vmstate: small, cheap copy ---
 	cloneVmstate := filepath.Join(cloneDir, "vmstate")
 	if err := copyFile(filepath.Join(goldenDir, "vmstate"), cloneVmstate); err != nil {
@@ -324,88 +545,76 @@ func (o *Orchestrator) spawnOne(id int) error {
 
 	// --- Networking ---
 	tapName := fmt.Sprintf("fctap%d", id)
-	var netIfaces firecracker.NetworkInterfaces
 	if o.cfg.Bridge != "none" {
+		o.log.Infof("clone %d: running: ip tuntap add dev %s mode tap", id, tapName)
+		o.log.Infof("clone %d: running: ip link set %s up", id, tapName)
+		o.log.Infof("clone %d: running: ip link set %s master %s", id, tapName, o.cfg.Bridge)
 		if err := o.createTap(tapName); err != nil {
 			return err
 		}
-		mac := fmt.Sprintf("AA:FC:00:00:%02X:%02X", id/256, id%256)
+	}
-		netIfaces = firecracker.NetworkInterfaces{
+
-			firecracker.NetworkInterface{
+	// --- Launch console proxy (detached daemon) ---
-				StaticConfiguration: &firecracker.StaticNetworkConfiguration{
+	// The proxy owns the full VM lifecycle: it starts firecracker with a PTY,
-					MacAddress:  mac,
+	// loads the snapshot, and serves cloneDir/console.sock until the VM exits.
-					HostDevName: tapName,
+	selfExe, err := os.Executable()
-				},
+	if err != nil {
-			},
+		return fmt.Errorf("resolve self path: %w", err)
+	}
+
+	proxyArgs := []string{"_console-proxy", "--id", strconv.Itoa(id), "--tag", tag}
+	if o.cfg.Bridge != "none" {
+		proxyArgs = append(proxyArgs, "--tap", tapName)
+	}
+	proxyCmd := exec.Command(selfExe, proxyArgs...)
+	// New session: proxy is detached from our terminal and survives our exit.
+	proxyCmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
+	proxyCmd.Stdin = nil
+	proxyCmd.Stdout = nil
+	proxyCmd.Stderr = nil
+	// Build proxy env: inherit parent env, then force FC_AUTO_NET_CONFIG to
+	// match the per-clone net flag so the proxy picks it up via DefaultConfig().
+	proxyEnv := make([]string, 0, len(os.Environ())+1)
+	for _, kv := range os.Environ() {
+		if !strings.HasPrefix(kv, "FC_AUTO_NET_CONFIG=") {
+			proxyEnv = append(proxyEnv, kv)
 		}
 	}
+	if net {
+		proxyEnv = append(proxyEnv, "FC_AUTO_NET_CONFIG=1")
+	}
+	proxyCmd.Env = proxyEnv
 
-	// --- Restore from snapshot ---
+	if err := proxyCmd.Start(); err != nil {
-	ctx, cancel := context.WithCancel(context.Background())
+		return fmt.Errorf("start console proxy: %w", err)
+	}
+	os.WriteFile(filepath.Join(o.pidsDir(), fmt.Sprintf("clone-%d.proxy.pid", id)),
+		[]byte(strconv.Itoa(proxyCmd.Process.Pid)), 0o644) //nolint:errcheck
 
-	fcBin, err := exec.LookPath(o.cfg.FCBin)
+	// Wait for the console socket to appear — it is created by the proxy once
-	if err != nil {
+	// the VM is running, so this also gates on successful snapshot restore.
-		cancel()
+	consoleSockPath := filepath.Join(cloneDir, "console.sock")
-		return fmt.Errorf("firecracker not found: %w", err)
+	if err := waitForSocket(consoleSockPath, 15*time.Second); err != nil {
+		return fmt.Errorf("clone %d: %w", id, err)
 	}
 
-	cmd := firecracker.VMCommandBuilder{}.
+	o.log.Infof("clone %d: ready (proxy pid=%d, tap=%s, console=%s)",
-		WithBin(fcBin).
+		id, proxyCmd.Process.Pid, tapName, consoleSockPath)
-		WithSocketPath(sockPath).
-		Build(ctx)
-
-	vcpus := o.cfg.VCPUs
-	mem := o.cfg.MemMiB
-
-	fcCfg := firecracker.Config{
-		SocketPath: sockPath,
-		MachineCfg: models.MachineConfiguration{
-			VcpuCount:  &vcpus,
-			MemSizeMib: &mem,
-		},
-		NetworkInterfaces: netIfaces,
-		// Snapshot config: tells the SDK to restore instead of fresh boot.
-		Snapshot: firecracker.SnapshotConfig{
-			MemFilePath:  sharedMem,
-			SnapshotPath: cloneVmstate,
-			ResumeVM:     true,
-		},
-	}
-
-	m, err := firecracker.NewMachine(ctx, fcCfg, firecracker.WithProcessRunner(cmd))
-	if err != nil {
-		cancel()
-		return fmt.Errorf("new machine: %w", err)
-	}
-
-	start := time.Now()
-	if err := m.Start(ctx); err != nil {
-		cancel()
-		return fmt.Errorf("restore clone %d: %w", id, err)
-	}
-	elapsed := time.Since(start)
-
-	// store PID
-	if cmd.Process != nil {
-		os.WriteFile(filepath.Join(o.pidsDir(), fmt.Sprintf("clone-%d.pid", id)),
-			[]byte(strconv.Itoa(cmd.Process.Pid)), 0o644)
-	}
-
-	o.mu.Lock()
-	o.clones[id] = &cloneInfo{
-		ID:      id,
-		Machine: m,
-		Cancel:  cancel,
-		Tap:     tapName,
-	}
-	o.mu.Unlock()
-
-	o.log.Infof("clone %d: restored in %s (pid=%d, tap=%s)",
-		id, elapsed.Round(time.Millisecond), cmd.Process.Pid, tapName)
-
 	return nil
 }
 
+// waitForSocket polls path every 50 ms until it appears or timeout elapses.
+func waitForSocket(path string, timeout time.Duration) error {
+	deadline := time.Now().Add(timeout)
+	for time.Now().Before(deadline) {
+		if _, err := os.Stat(path); err == nil {
+			return nil
+		}
+		time.Sleep(50 * time.Millisecond)
+	}
+	return fmt.Errorf("timed out waiting for %s", path)
+}
+
 // ——— Status ————————————————————————————————————————————————————————————
 
 func (o *Orchestrator) Status() {
@@ -478,7 +687,7 @@ func (o *Orchestrator) Kill() error {
 func (o *Orchestrator) Cleanup() error {
 	o.Kill()
 	os.RemoveAll(o.clonesDir())
-	os.RemoveAll(o.goldenDir())
+	os.RemoveAll(filepath.Join(o.cfg.BaseDir, "golden"))
 	os.RemoveAll(o.pidsDir())
 
 	if o.cfg.Bridge != "none" {
@@ -492,6 +701,56 @@ func (o *Orchestrator) Cleanup() error {
 
 // ——— Helpers ——————————————————————————————————————————————————————————
 
+// installUbuntuPackages bind-mounts the virtual filesystems into mnt, then
+// runs apt-get inside the chroot to install the minimal toolset required for
+// network operation and general use.  Bind mounts are always cleaned up on
+// return regardless of whether apt-get succeeds.
+func installUbuntuPackages(mnt string, logger *log.Entry) error {
+	type bm struct{ fstype, src, dst string }
+	mounts := []bm{
+		{"proc", "proc", "proc"},
+		{"sysfs", "sysfs", "sys"},
+		{"devtmpfs", "devtmpfs", "dev"},
+		{"devpts", "devpts", "dev/pts"},
+	}
+
+	// mount in order; on any failure unmount whatever succeeded and return.
+	for i, m := range mounts {
+		dst := filepath.Join(mnt, m.dst)
+		os.MkdirAll(dst, 0o755)
+		logger.Infof("running: mount -t %s %s %s", m.fstype, m.src, dst)
+		if err := run("mount", "-t", m.fstype, m.src, dst); err != nil {
+			for j := i - 1; j >= 0; j-- {
+				logger.Infof("running: umount %s", filepath.Join(mnt, mounts[j].dst))
+				run("umount", filepath.Join(mnt, mounts[j].dst)) //nolint:errcheck
+			}
+			return fmt.Errorf("mount %s: %w", m.dst, err)
+		}
+	}
+	defer func() {
+		for i := len(mounts) - 1; i >= 0; i-- {
+			logger.Infof("running: umount %s", filepath.Join(mnt, mounts[i].dst))
+			run("umount", filepath.Join(mnt, mounts[i].dst)) //nolint:errcheck
+		}
+	}()
+
+	// Provide DNS resolution inside the chroot so apt-get can reach the network.
+	if data, err := os.ReadFile("/etc/resolv.conf"); err == nil {
+		os.WriteFile(filepath.Join(mnt, "etc/resolv.conf"), data, 0o644) //nolint:errcheck
+	}
+
+	pkgs := "bash curl iproute2 wget ca-certificates systemd systemd-sysv util-linux"
+	script := "DEBIAN_FRONTEND=noninteractive apt-get update -q && " +
+		"DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends " + pkgs + " && " +
+		"apt-get clean && rm -rf /var/lib/apt/lists/*"
+
+	logger.Infof("running: chroot %s /bin/sh -c %q", mnt, script)
+	cmd := exec.Command("chroot", mnt, "/bin/sh", "-c", script)
+	cmd.Stdout = logger.Writer()
+	cmd.Stderr = logger.Writer()
+	return cmd.Run()
+}
+
 func (o *Orchestrator) nextCloneID() int {
 	max := 0
 	entries, _ := os.ReadDir(o.clonesDir())

orchestrator/serve.go

@@ -0,0 +1,255 @@
+package orchestrator
+
+import (
+	"encoding/json"
+	"fmt"
+	_ "embed"
+	"net"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+
+	"github.com/gorilla/websocket"
+	log "github.com/sirupsen/logrus"
+)
+
+//go:embed web/terminal.html
+var terminalHTML []byte
+
+var upgrader = websocket.Upgrader{
+	// Allow all origins for local/dev use. Add an origin check for production.
+	CheckOrigin: func(r *http.Request) bool { return true },
+}
+
+// Serve starts an HTTP server that exposes:
+//
+//	GET    /              — terminal UI (xterm.js); ?id=N selects a clone
+//	GET    /clones        — JSON list of running clone IDs
+//	POST   /clones        — spawn a new clone; returns {"id": N}
+//	DELETE /clones/{id}   — destroy clone {id}
+//	GET    /ws/{id}       — WebSocket console for clone {id}
+//
+// Binary WebSocket frames carry raw terminal bytes (stdin/stdout).
+// Text WebSocket frames carry JSON resize commands: {"rows":N,"cols":M}.
+func Serve(orch *Orchestrator, addr string) error {
+	mux := http.NewServeMux()
+
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path != "/" {
+			http.NotFound(w, r)
+			return
+		}
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		w.Write(terminalHTML) //nolint:errcheck
+	})
+
+	// /clones — list (GET) or spawn (POST)
+	mux.HandleFunc("/clones", func(w http.ResponseWriter, r *http.Request) {
+		switch r.Method {
+		case http.MethodGet, "":
+			clones := runningClones(orch.cfg)
+			w.Header().Set("Content-Type", "application/json")
+			json.NewEncoder(w).Encode(clones) //nolint:errcheck
+		case http.MethodPost:
+			// Optional JSON body: {"net": bool, "tag": string}
+			// Defaults to the server's FC_AUTO_NET_CONFIG setting.
+			var req struct {
+				Net *bool   `json:"net"`
+				Tag *string `json:"tag"`
+			}
+			if r.ContentLength > 0 {
+				json.NewDecoder(r.Body).Decode(&req) //nolint:errcheck
+			}
+			net := orch.cfg.AutoNetConfig
+			if req.Net != nil {
+				net = *req.Net
+			}
+			tag := "default"
+			if req.Tag != nil && *req.Tag != "" {
+				tag = *req.Tag
+			}
+			id, err := orch.SpawnSingle(net, tag)
+			if err != nil {
+				http.Error(w, err.Error(), http.StatusInternalServerError)
+				return
+			}
+			w.Header().Set("Content-Type", "application/json")
+			w.WriteHeader(http.StatusCreated)
+			json.NewEncoder(w).Encode(map[string]int{"id": id}) //nolint:errcheck
+		default:
+			http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+		}
+	})
+	// /tags — list all available golden VM tags
+	mux.HandleFunc("/tags", func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != http.MethodGet && r.Method != "" {
+			http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+			return
+		}
+		tags := orch.GoldenTags()
+		w.Header().Set("Content-Type", "application/json")
+		json.NewEncoder(w).Encode(tags) //nolint:errcheck
+	})
+
+	// /clones/{id} — destroy (DELETE)
+	mux.HandleFunc("/clones/", func(w http.ResponseWriter, r *http.Request) {
+		idStr := strings.TrimPrefix(r.URL.Path, "/clones/")
+		if idStr == "" {
+			// redirect bare /clones/ to /clones
+			http.Redirect(w, r, "/clones", http.StatusMovedPermanently)
+			return
+		}
+		if r.Method != http.MethodDelete {
+			http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+			return
+		}
+		id, err := strconv.Atoi(idStr)
+		if err != nil {
+			http.Error(w, "invalid clone id", http.StatusBadRequest)
+			return
+		}
+		if err := orch.KillClone(id); err != nil {
+			http.Error(w, err.Error(), http.StatusInternalServerError)
+			return
+		}
+		w.WriteHeader(http.StatusNoContent)
+	})
+
+	mux.HandleFunc("/ws/", func(w http.ResponseWriter, r *http.Request) {
+		idStr := strings.TrimPrefix(r.URL.Path, "/ws/")
+		id, err := strconv.Atoi(idStr)
+		if err != nil {
+			http.Error(w, "invalid clone id", http.StatusBadRequest)
+			return
+		}
+		handleConsoleWS(orch.cfg, id, w, r)
+	})
+
+	log.Infof("terminal UI: http://%s", addr)
+	return http.ListenAndServe(addr, mux)
+}
+
+// handleConsoleWS upgrades the request to a WebSocket and bridges it to the
+// clone's console.sock (I/O) and console-resize.sock (terminal resize).
+func handleConsoleWS(cfg Config, id int, w http.ResponseWriter, r *http.Request) {
+	cloneDir := filepath.Join(cfg.BaseDir, "clones", strconv.Itoa(id))
+	consoleSock := filepath.Join(cloneDir, "console.sock")
+	resizeSock := filepath.Join(cloneDir, "console-resize.sock")
+
+	if _, err := os.Stat(consoleSock); err != nil {
+		http.Error(w, fmt.Sprintf("clone %d is not running", id), http.StatusNotFound)
+		return
+	}
+
+	ws, err := upgrader.Upgrade(w, r, nil)
+	if err != nil {
+		return
+	}
+	defer ws.Close()
+
+	consoleConn, err := net.Dial("unix", consoleSock)
+	if err != nil {
+		writeWSError(ws, fmt.Sprintf("console unavailable: %v", err))
+		return
+	}
+	defer consoleConn.Close()
+
+	resizeConn, err := net.Dial("unix", resizeSock)
+	if err != nil {
+		log.Warnf("clone %d: could not connect to resize socket, resize disabled: %v", id, err)
+		resizeConn = nil
+	}
+	if resizeConn != nil {
+		defer resizeConn.Close()
+	}
+
+	log.Infof("ws: clone %d: client connected from %s", id, r.RemoteAddr)
+	bridgeWS(ws, consoleConn, resizeConn)
+	log.Infof("ws: clone %d: client disconnected", id)
+}
+
+// bridgeWS proxies between a WebSocket connection and a console Unix socket.
+//
+//   - Binary WS frames  → consoleConn (terminal input)
+//   - Text WS frames    → resizeConn as a JSON line (resize command)
+//   - consoleConn reads → binary WS frames (terminal output)
+func bridgeWS(ws *websocket.Conn, consoleConn net.Conn, resizeConn net.Conn) {
+	// console → WebSocket
+	sockDone := make(chan struct{})
+	go func() {
+		defer close(sockDone)
+		buf := make([]byte, 4096)
+		for {
+			n, err := consoleConn.Read(buf)
+			if n > 0 {
+				if werr := ws.WriteMessage(websocket.BinaryMessage, buf[:n]); werr != nil {
+					return
+				}
+			}
+			if err != nil {
+				return
+			}
+		}
+	}()
+
+	// WebSocket → console / resize
+	for {
+		msgType, data, err := ws.ReadMessage()
+		if err != nil {
+			break
+		}
+		switch msgType {
+		case websocket.BinaryMessage:
+			consoleConn.Write(data) //nolint:errcheck
+		case websocket.TextMessage:
+			if resizeConn != nil {
+				// Append newline so the scanner in handleResize sees a complete line.
+				resizeConn.Write(append(data, '\n')) //nolint:errcheck
+			}
+		}
+	}
+
+	consoleConn.Close()
+	<-sockDone
+}
+
+type cloneEntry struct {
+	ID  int    `json:"id"`
+	Tag string `json:"tag"`
+}
+
+// runningClones returns entries for clones that have a live console socket.
+func runningClones(cfg Config) []cloneEntry {
+	clonesDir := filepath.Join(cfg.BaseDir, "clones")
+	entries, err := os.ReadDir(clonesDir)
+	if err != nil {
+		return nil
+	}
+	var clones []cloneEntry
+	for _, e := range entries {
+		if !e.IsDir() {
+			continue
+		}
+		id, err := strconv.Atoi(e.Name())
+		if err != nil {
+			continue
+		}
+		sock := filepath.Join(clonesDir, e.Name(), "console.sock")
+		if _, err := os.Stat(sock); err != nil {
+			continue
+		}
+		tag := "unknown"
+		if raw, err := os.ReadFile(filepath.Join(clonesDir, e.Name(), "tag")); err == nil {
+			tag = strings.TrimSpace(string(raw))
+		}
+		clones = append(clones, cloneEntry{ID: id, Tag: tag})
+	}
+	return clones
+}
+
+func writeWSError(ws *websocket.Conn, msg string) {
+	ws.WriteMessage(websocket.TextMessage, //nolint:errcheck
+		[]byte("\r\n\x1b[31m["+msg+"]\x1b[0m\r\n"))
+}

orchestrator/snapshot.go

@@ -0,0 +1,71 @@
+package orchestrator
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+)
+
+type networkOverride struct {
+	IfaceID     string `json:"iface_id"`
+	HostDevName string `json:"host_dev_name"`
+}
+
+type snapshotLoadRequest struct {
+	MemFilePath      string            `json:"mem_file_path"`
+	SnapshotPath     string            `json:"snapshot_path"`
+	ResumeVM         bool              `json:"resume_vm,omitempty"`
+	NetworkOverrides []networkOverride `json:"network_overrides,omitempty"`
+}
+
+// loadSnapshotWithNetworkOverride calls PUT /snapshot/load on the Firecracker
+// Unix socket, remapping the first network interface to tapName.
+// This bypasses the SDK's LoadSnapshotHandler which doesn't expose
+// network_overrides (added in Firecracker v1.15, SDK v1.0.0 omits it).
+func loadSnapshotWithNetworkOverride(ctx context.Context, sockPath, memPath, vmstatePath, tapName string) error {
+	payload := snapshotLoadRequest{
+		MemFilePath:  memPath,
+		SnapshotPath: vmstatePath,
+		ResumeVM:     false, // Changed: We pause here so MMDS can be configured BEFORE Resume.
+		NetworkOverrides: []networkOverride{
+			{IfaceID: "1", HostDevName: tapName},
+		},
+	}
+
+	data, err := json.Marshal(payload)
+	if err != nil {
+		return fmt.Errorf("marshal snapshot load params: %w", err)
+	}
+
+	httpClient := &http.Client{
+		Transport: &http.Transport{
+			DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+				return net.Dial("unix", sockPath)
+			},
+		},
+	}
+
+	req, err := http.NewRequestWithContext(ctx, http.MethodPut,
+		"http://localhost/snapshot/load", bytes.NewReader(data))
+	if err != nil {
+		return fmt.Errorf("build snapshot load request: %w", err)
+	}
+	req.Header.Set("Content-Type", "application/json")
+	req.Header.Set("Accept", "application/json")
+
+	resp, err := httpClient.Do(req)
+	if err != nil {
+		return fmt.Errorf("snapshot load request: %w", err)
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusNoContent {
+		body, _ := io.ReadAll(resp.Body)
+		return fmt.Errorf("snapshot load failed (%d): %s", resp.StatusCode, body)
+	}
+	return nil
+}

orchestrator/web/terminal.html

@@ -0,0 +1,363 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <title>fc-orch console</title>
+  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/xterm@5/css/xterm.css"/>
+  <script src="https://cdn.jsdelivr.net/npm/xterm@5/lib/xterm.js"></script>
+  <script src="https://cdn.jsdelivr.net/npm/xterm-addon-fit@0.8/lib/xterm-addon-fit.js"></script>
+  <style>
+    *, *::before, *::after { box-sizing: border-box; }
+    html, body {
+      margin: 0; padding: 0;
+      width: 100%; height: 100%;
+      background: #0d0d0d;
+      font-family: monospace;
+      color: #ccc;
+    }
+
+    /* ── index / clone picker ── */
+    #index {
+      display: flex;
+      flex-direction: column;
+      align-items: center;
+      justify-content: center;
+      height: 100%;
+      gap: 1rem;
+    }
+    #index h1 { margin: 0; font-size: 1.4rem; color: #8be; }
+
+    #clone-list {
+      list-style: none;
+      padding: 0; margin: 0;
+      display: flex;
+      flex-wrap: wrap;
+      gap: .5rem;
+    }
+    .clone-entry {
+      display: flex;
+      align-items: stretch;
+      border: 1px solid #444;
+      border-radius: 4px;
+      overflow: hidden;
+    }
+    .clone-entry a {
+      display: block;
+      padding: .4rem 1rem;
+      color: #8be;
+      text-decoration: none;
+      transition: background .15s;
+    }
+    .clone-entry a:hover { background: #1e2e3e; }
+    .clone-entry button.destroy {
+      padding: .4rem .6rem;
+      background: transparent;
+      border: none;
+      border-left: 1px solid #444;
+      color: #c44;
+      cursor: pointer;
+      font-size: .8rem;
+      transition: background .15s;
+    }
+    .clone-entry button.destroy:hover { background: #2a1a1a; }
+    .clone-entry button.destroy:disabled { color: #555; cursor: default; }
+    .clone-tag {
+      font-size: .72rem;
+      color: #666;
+      margin-left: .4rem;
+    }
+
+    #index .none { color: #666; font-size: .9rem; }
+
+    #spawn-btn {
+      padding: .45rem 1.2rem;
+      background: #1a2e1a;
+      border: 1px solid #4c4;
+      border-radius: 4px;
+      color: #4c4;
+      cursor: pointer;
+      font-family: monospace;
+      font-size: .9rem;
+      transition: background .15s, opacity .15s;
+    }
+    #spawn-btn:hover:not(:disabled) { background: #243e24; }
+    #spawn-btn:disabled { opacity: .5; cursor: default; }
+
+    #spawn-controls {
+      display: flex;
+      gap: 0.5rem;
+      align-items: center;
+    }
+
+    #tag-select {
+      background: #1a1a1a;
+      border: 1px solid #444;
+      border-radius: 4px;
+      color: #8be;
+      padding: 0.4rem 0.8rem;
+      font-family: monospace;
+      font-size: 0.9rem;
+      outline: none;
+      transition: border-color .15s, background .15s;
+      cursor: pointer;
+    }
+    #tag-select:hover { background: #222; }
+    #tag-select:focus { border-color: #8be; }
+    #tag-select:disabled { opacity: .5; cursor: default; }
+
+    #error-msg {
+      color: #c44;
+      font-size: .85rem;
+      display: none;
+    }
+
+    /* ── terminal view ── */
+    #terminal-wrap {
+      display: none;
+      flex-direction: column;
+      height: 100%;
+    }
+    #topbar {
+      display: flex;
+      align-items: center;
+      gap: .75rem;
+      padding: .35rem .75rem;
+      background: #111;
+      border-bottom: 1px solid #222;
+      flex-shrink: 0;
+    }
+    #topbar .title { font-size: .85rem; color: #8be; }
+    #status {
+      font-size: .75rem;
+      padding: .15rem .5rem;
+      border-radius: 3px;
+      background: #1a2a1a;
+      color: #4c4;
+    }
+    #status.disconnected { background: #2a1a1a; color: #c44; }
+    #terminal-container { flex: 1; overflow: hidden; padding: 4px; }
+  </style>
+</head>
+<body>
+
+<!-- clone picker (shown when no ?id= param) -->
+<div id="index">
+  <h1>fc-orch console</h1>
+  <ul id="clone-list"></ul>
+  <p class="none" id="no-clones" style="display:none">No running clones.</p>
+  <div id="spawn-controls">
+    <select id="tag-select"></select>
+    <button id="spawn-btn">+ Spawn clone</button>
+  </div>
+  <p id="error-msg"></p>
+</div>
+
+<!-- terminal (shown when ?id=N) -->
+<div id="terminal-wrap">
+  <div id="topbar">
+    <span class="title" id="topbar-title"></span>
+    <span id="status">connecting…</span>
+    <a href="/" style="margin-left:auto;font-size:.75rem;color:#666;text-decoration:none">← all clones</a>
+  </div>
+  <div id="terminal-container"></div>
+</div>
+
+<script>
+(function () {
+  const params = new URLSearchParams(location.search);
+  const id = params.get('id');
+
+  // ── index view ──────────────────────────────────────────────
+  if (!id) {
+    const ul      = document.getElementById('clone-list');
+    const noneEl  = document.getElementById('no-clones');
+    const spawnBtn = document.getElementById('spawn-btn');
+    const tagSelect = document.getElementById('tag-select');
+    const errEl   = document.getElementById('error-msg');
+
+    function showError(msg) {
+      errEl.textContent = msg;
+      errEl.style.display = '';
+    }
+    function clearError() {
+      errEl.style.display = 'none';
+    }
+
+    function addCloneEntry(c) {
+      noneEl.style.display = 'none';
+      const li = document.createElement('li');
+      li.className = 'clone-entry';
+      li.dataset.id = c.id;
+      li.innerHTML =
+        `<a href="/?id=${c.id}">clone ${c.id}<span class="clone-tag">${c.tag}</span></a>` +
+        `<button class="destroy" title="Destroy clone ${c.id}">✕</button>`;
+      li.querySelector('.destroy').addEventListener('click', () => destroyClone(c.id, li));
+      ul.appendChild(li);
+    }
+
+    function refreshList() {
+      fetch('/clones')
+        .then(r => r.json())
+        .then(clones => {
+          ul.innerHTML = '';
+          if (!clones || clones.length === 0) {
+            noneEl.style.display = '';
+            return;
+          }
+          noneEl.style.display = 'none';
+          clones.forEach(addCloneEntry);
+        })
+        .catch(() => { noneEl.style.display = ''; });
+    }
+
+    function destroyClone(cid, li) {
+      const btn = li.querySelector('.destroy');
+      btn.disabled = true;
+      clearError();
+      fetch(`/clones/${cid}`, { method: 'DELETE' })
+        .then(r => {
+          if (!r.ok) return r.text().then(t => { throw new Error(t); });
+          li.remove();
+          if (ul.children.length === 0) noneEl.style.display = '';
+        })
+        .catch(e => {
+          btn.disabled = false;
+          showError(`destroy failed: ${e.message}`);
+        });
+    }
+
+    spawnBtn.addEventListener('click', () => {
+      spawnBtn.disabled = true;
+      spawnBtn.textContent = 'Spawning…';
+      clearError();
+      fetch('/clones', {
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json' },
+        body: JSON.stringify({ net: true, tag: tagSelect.value }),
+      })
+        .then(r => {
+          if (!r.ok) return r.text().then(t => { throw new Error(t); });
+          return r.json();
+        })
+        .then(data => {
+          location.href = `/?id=${data.id}`;
+        })
+        .catch(e => {
+          spawnBtn.disabled = false;
+          spawnBtn.textContent = '+ Spawn clone';
+          showError(`spawn failed: ${e.message}`);
+        });
+    });
+
+    function refreshTags() {
+      fetch('/tags')
+        .then(r => r.json())
+        .then(tags => {
+          tagSelect.innerHTML = '';
+          if (!tags || tags.length === 0) {
+            const opt = document.createElement('option');
+            opt.value = '';
+            opt.textContent = 'No golden VMs';
+            tagSelect.appendChild(opt);
+            tagSelect.disabled = true;
+            spawnBtn.disabled = true;
+            return;
+          }
+          tagSelect.disabled = false;
+          spawnBtn.disabled = false;
+          tags.forEach(t => {
+            const opt = document.createElement('option');
+            opt.value = t;
+            opt.textContent = t;
+            if (t === 'default' || t === 'alpine') opt.selected = true;
+            tagSelect.appendChild(opt);
+          });
+        })
+        .catch(e => {
+          console.error("fetch tags failed:", e);
+        });
+    }
+
+    refreshList();
+    refreshTags();
+    return;
+  }
+
+  // ── terminal view ────────────────────────────────────────────
+  document.getElementById('index').style.display = 'none';
+  const wrap = document.getElementById('terminal-wrap');
+  wrap.style.display = 'flex';
+  document.getElementById('topbar-title').textContent = `clone ${id}`;
+  document.title = `clone ${id} — fc-orch`;
+
+  const term = new Terminal({
+    cursorBlink: true,
+    scrollback: 5000,
+    theme: {
+      background:   '#0d0d0d',
+      foreground:   '#d0d0d0',
+      cursor:       '#8be',
+      selectionBackground: '#2a4a6a',
+    },
+  });
+  const fitAddon = new FitAddon.FitAddon();
+  term.loadAddon(fitAddon);
+  term.open(document.getElementById('terminal-container'));
+  fitAddon.fit();
+
+  const statusEl = document.getElementById('status');
+  function setStatus(text, ok) {
+    statusEl.textContent = text;
+    statusEl.className = ok ? '' : 'disconnected';
+  }
+
+  const proto = location.protocol === 'https:' ? 'wss:' : 'ws:';
+  const ws = new WebSocket(`${proto}//${location.host}/ws/${id}`);
+  ws.binaryType = 'arraybuffer';
+
+  ws.onopen = () => {
+    setStatus('connected', true);
+    sendResize();
+    term.focus();
+  };
+
+  ws.onmessage = e => {
+    if (e.data instanceof ArrayBuffer) {
+      term.write(new Uint8Array(e.data));
+    }
+  };
+
+  ws.onclose = () => {
+    setStatus('disconnected', false);
+    term.write('\r\n\x1b[31m[connection closed]\x1b[0m\r\n');
+  };
+
+  ws.onerror = () => setStatus('error', false);
+
+  // Keystrokes → VM (binary frame so the server can distinguish from resize)
+  term.onData(data => {
+    if (ws.readyState === WebSocket.OPEN) {
+      const bytes = new TextEncoder().encode(data);
+      ws.send(bytes.buffer);
+    }
+  });
+
+  // Resize handling
+  function sendResize() {
+    if (ws.readyState === WebSocket.OPEN) {
+      ws.send(JSON.stringify({ rows: term.rows, cols: term.cols }));
+    }
+  }
+
+  term.onResize(() => sendResize());
+
+  let resizeTimer;
+  window.addEventListener('resize', () => {
+    clearTimeout(resizeTimer);
+    resizeTimer = setTimeout(() => { fitAddon.fit(); }, 50);
+  });
+}());
+</script>
+</body>
+</html>

test_mmds.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+sock="/tmp/fctest.sock"
+rm -f "$sock"
+firecracker --api-sock "$sock" &
+FCPID=$!
+sleep 1
+
+# Configure MMDS backend
+curl --unix-socket "$sock" -i -X PUT "http://localhost/mmds/config" \
+  -H "Accept: application/json" -H "Content-Type: application/json" \
+  -d '{"version": "V1", "network_interfaces": ["1"], "ipv4_address": "169.254.169.254"}'
+
+# Put data
+curl --unix-socket "$sock" -i -X PUT "http://localhost/mmds" \
+  -H "Accept: application/json" -H "Content-Type: application/json" \
+  -d '{"ip": "10.0.0.2", "gw": "10.0.0.1", "dns": "1.1.1.1"}'
+
+# Read data
+curl --unix-socket "$sock" -i -X GET "http://localhost/mmds"
+
+kill $FCPID

test_mmds_restore.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+sock="/tmp/fctest2.sock"
+rm -f "$sock"
+firecracker --api-sock "$sock" >/dev/null 2>&1 &
+FCPID=$!
+sleep 1
+
+# Start a VM basically
+curl --unix-socket "$sock" -s -X PUT "http://localhost/machine-config" -d '{"vcpu_count": 1, "mem_size_mib": 128}'
+curl --unix-socket "$sock" -s -X PUT "http://localhost/network-interfaces/1" -d '{"iface_id": "1", "guest_mac": "AA:FC:00:00:00:01", "host_dev_name": "lo"}'
+curl --unix-socket "$sock" -s -X PUT "http://localhost/mmds/config" -d '{"version": "V1", "network_interfaces": ["1"], "ipv4_address": "169.254.169.254"}'
+curl --unix-socket "$sock" -s -X PUT "http://localhost/boot-source" -d '{"kernel_image_path": "/tmp/fc-orch/vmlinux", "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"}'
+curl --unix-socket "$sock" -s -X PUT "http://localhost/actions" -d '{"action_type": "InstanceStart"}'
+
+# Pause
+curl --unix-socket "$sock" -s -X PATCH "http://localhost/vm" -d '{"state": "Paused"}'
+
+# TRY TO CONFIGURE MMDS
+curl --unix-socket "$sock" -i -X PUT "http://localhost/mmds/config" -d '{"version": "V1", "network_interfaces": ["1"], "ipv4_address": "169.254.169.254"}'
+
+kill $FCPID