Point the manager config and sample at the beelink KVM host

The default manager config gains a kvm-home provider reaching beelink
over Tailscale, the provider sample names the same host, and the
manager kustomization uses the locally built kvm-dev image.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -1,5 +1,55 @@
 # Changelog
 
+## 2026-09-10 22:58 UTC — Code-review fixes on the KVM branch: deletion, validation, lazy SSH, guest agent
+
+Five findings from a review of `feat/kvm-provider`, all fixed on the branch.
+
+- **A Proxy can always finish deleting.** The delete path used to drop the finalizer
+  only when `Get` answered not-found, so a permanent error there retried forever —
+  a KVM host removed from the config was enough. It now calls `Delete` anyway, since
+  that is idempotent and can finish where `Get` cannot, and
+  `crawl.example.com/skip-cleanup: "true"` abandons the instance for whatever stays
+  stuck. In the KVM provider, deleting over an unconfigured host is a no-op success;
+  `Get` still fails permanently there so a typo cannot look like a missing VM.
+- **The guest agent ships as a vendor-data script**, superseding the cloud-config
+  merge in the entry below. Merging could never be made safe: cloud-init drops
+  vendor-data keys the user-data defines, and the strategy cannot be overridden from
+  either document. The Proxy's cloud-init now reaches the seed byte for byte in any
+  format — Jinja, MIME, gzip, scripts — where the round-trip previously re-typed
+  unquoted scalars (`version: 1.10` arrived as `1.1`), dropped comments and reordered
+  keys. New `installGuestAgent` setting, default on.
+- **A missing `kvm-ssh` Secret no longer crash-loops the manager.** The SSH material
+  is read on first use and cached, so only KVM Proxies fail and they recover when the
+  Secret appears. A missing file is transient, an unparseable key permanent, and
+  startup still logs a warning.
+- **Host names and image keys are validated.** Either can end up in a providerID or a
+  volume file name, where a slash made the VM unaddressable for good.
+- Verified end to end: unit suite, the integration test against a real hypervisor
+  (22 s, agent reporting the IP), both in-cluster Proxies Ready in 2 min with 204
+  through squid, the guest showing `qemu-guest-agent` active with its cloud-config
+  comments intact, and the manager staying up with the Secret pointed away.
+- Key files: `internal/provider/kvm/{seed.go,conn.go,kvm.go}`,
+  `internal/controller/proxy_controller.go`, `internal/provider/config_kvm.go`,
+  `api/v1alpha1/proxy_types.go`.
+
+## 2026-09-10 21:04 UTC — KVM provider passes in-cluster; guest agent merged into cloud-config
+
+- Two KVM Proxies with real cloud-configs reach `Ready` from inside the kind cluster
+  (2 min from apply, IP via the guest agent, 204 through squid from the LAN).
+- The `qemu-guest-agent` bootstrap is now merged into each Proxy's cloud-config
+  (`packages` + `runcmd`) instead of shipped as vendor-data: cloud-init's default
+  merger drops vendor-data keys the user-data also defines, so any Proxy with its own
+  `packages:` never got the agent and stayed `Provisioning` forever. `vendorData` is
+  now plain extra vendor-data with no default; non-cloud-config user-data (scripts,
+  `#include`) still gets the bootstrap as vendor-data. A cloud-config that is not a
+  YAML mapping fails the Proxy permanently instead of booting without its config.
+- The manager pod mounts a 64 MiB `emptyDir` on `/tmp`: the seed ISO writer stages
+  files on disk and the root filesystem is read-only.
+- Errors already classified with a provider sentinel no longer drop the cached
+  libvirt SSH connection; only libvirt and transport errors do.
+- Key files: `internal/provider/kvm/{seed.go,conn.go,errors.go}`,
+  `config/manager/manager.yaml`, `docs/kvm-provider.md`.
+
 ## 2026-09-10 10:07 UTC — KVM provider validated on a real hypervisor; UEFI default; libvirt call tracing
 
 - `make test-integration-kvm` passes end to end against a libvirt 11.6 / QEMU 10.1

README.md

@@ -164,6 +164,12 @@ Cloud-init from a Secret: the Secret **must** carry the label
 labelled Secrets, so an unlabelled one is invisible (the Proxy reports
 `CloudInitError`). Rotating the Secret's content triggers VM replacement.
 
+Stuck deletion: a Proxy stays `Terminating` while the operator keeps failing
+to clean its instance up — credentials revoked, a provider or a KVM host
+dropped from the config. Annotate it with
+`crawl.example.com/skip-cleanup: "true"` to abandon the instance and release
+the object. Whatever is still running then has to be removed by hand.
+
 ## Providers
 
 `Managed` proxies are provisioned by a provider — a small compute backend

api/v1alpha1/proxy_types.go

@@ -62,6 +62,15 @@ const (
 	// must be replaced.
 	AnnotationSpecHash = "crawl.example.com/spec-hash"
 
+	// AnnotationSkipCleanup abandons the provider-side instance: set it to
+	// "true" and deletion removes the finalizer without asking the provider
+	// to clean anything up. It is the escape hatch for a cleanup that cannot
+	// succeed — a provider dropped from the config, credentials revoked, a
+	// backend that answers permanently — where the object would otherwise
+	// stay in Terminating forever. The instance is left running and becomes
+	// the administrator's to remove.
+	AnnotationSkipCleanup = "crawl.example.com/skip-cleanup"
+
 	// LabelCloudInit must be set (to "true") on every Secret referenced by
 	// spec.cloudInit.secretRef: the manager's cache only holds Secrets
 	// carrying this label, so an unlabelled Secret is invisible to the

config/manager/kustomization.yaml

@@ -5,5 +5,5 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 images:
 - name: controller
-  newName: example.com/egress-proxies-operator
-  newTag: v0.0.1
+  newName: egress-proxies-operator
+  newTag: kvm-dev

config/manager/manager.yaml

@@ -154,6 +154,11 @@ spec:
         - name: kvm-ssh
           mountPath: /etc/proxy-operator/kvm
           readOnly: true
+        # Scratch space: the root filesystem is read-only, and the kvm
+        # provider builds each VM's cloud-init seed ISO through a temp
+        # directory (a few hundred KB, deleted right after upload).
+        - name: tmp
+          mountPath: /tmp
       volumes:
       - name: providers-config
         configMap:
@@ -162,5 +167,8 @@ spec:
         secret:
           secretName: kvm-ssh
           optional: true
+      - name: tmp
+        emptyDir:
+          sizeLimit: 64Mi
       serviceAccountName: controller-manager
       terminationGracePeriodSeconds: 10

config/manager/providers_config.yaml

@@ -16,3 +16,39 @@ data:
     providers:
       - name: kubernetes
         type: kubernetes
+      - name: kvm-home
+        type: kvm
+        kvm:
+          # SSH to each host with the libvirt unix socket forwarded through the
+          # connection; nothing to open on the hypervisors beyond sshd. Both
+          # files come from the optional kvm-ssh Secret mounted by
+          # config/manager/manager.yaml — see docs/kvm-provider.md.
+          sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
+          knownHostsFile: /etc/proxy-operator/kvm/known_hosts
+          # insecureIgnoreHostKey: true   # skip host-key checks (not recommended)
+          # diskSizeGb: 10                # per-VM overlay; cloud-init grows the root fs into it
+          # vendorData: |                 # extra vendor-data; the qemu-guest-agent bootstrap is merged
+          #   #cloud-config               # into each Proxy's cloud-config, not configured here
+          #   ...
+          images:
+            # placement.image names a key here (or is an https:// URL directly).
+            # Fetched into each host's pool on first use and cached there as
+            # base-<key>.qcow2; do not change an entry while Proxies use it.
+            debian-13:
+              url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
+              # sha256: "<64 hex>"        # optional; verified after download
+              # firmware: uefi            # default; "bios" only for images with no EFI partition
+              #                           # (Debian 13 hangs in GRUB under BIOS headless: Debian bug #1111240)
+            ubuntu-24.04:
+              url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
+          machineTypes:                   # placement.machineType names a key here
+            small:  {vcpus: 1, memoryMiB: 1024}
+            medium: {vcpus: 2, memoryMiB: 2048}
+          hosts:                          # placement.zone names a host; optional with one host
+            - name: beelink
+              address: beelink.tail8c69c.ts.net:22
+              user: novakj                  # must be in the libvirt group
+              bridge: br0                 # to a LAN with DHCP; the VM's IP comes from there
+              # socket: /var/run/libvirt/libvirt-sock
+              # pool: default
+

config/samples/providers-config.yaml

@@ -30,8 +30,9 @@ providers:
       knownHostsFile: /etc/proxy-operator/kvm/known_hosts
       # insecureIgnoreHostKey: true   # skip host-key checks (not recommended)
       # diskSizeGb: 10                # per-VM overlay; cloud-init grows the root fs into it
-      # vendorData: |                 # default installs qemu-guest-agent for IP discovery
-      #   #cloud-config
+      # installGuestAgent: false      # skip the vendor-data script that installs qemu-guest-agent
+      # vendorData: |                 # extra vendor-data, carried beside the agent script as a
+      #   #cloud-config               # MIME multipart. The Proxy's own cloud-init is never rewritten.
       #   ...
       images:
         # placement.image names a key here (or is an https:// URL directly).
@@ -48,9 +49,9 @@ providers:
         small:  {vcpus: 1, memoryMiB: 1024}
         medium: {vcpus: 2, memoryMiB: 2048}
       hosts:                          # placement.zone names a host; optional with one host
-        - name: nas1
-          address: nas1.home.lan:22
-          user: virt                  # must be in the libvirt group
+        - name: beelink
+          address: beelink:22
+          user: novakj                  # must be in the libvirt group
           bridge: br0                 # to a LAN with DHCP; the VM's IP comes from there
           # socket: /var/run/libvirt/libvirt-sock
           # pool: default

docs/architecture.md

@@ -489,6 +489,15 @@ they read as choices, not accidents. Chronological by build step.
   cluster-wide Secret read RBAC — an unrestricted cache would hold every
   Secret in scope). Unlabelled referenced Secrets are invisible by
   construction, surfacing as `CloudInitError`.
+- **Deletion never gives up on its own, but can be released by hand:** a
+  permanent cleanup failure keeps retrying with the finalizer on, because
+  dropping it automatically would leak a running instance on any provider
+  that answers "permission denied". Where `Get` fails permanently the
+  reconciler still calls `Delete`, since it is idempotent and may finish
+  where `Get` cannot converge. `crawl.example.com/skip-cleanup: "true"`
+  abandons the instance and releases the object; it is checked before the
+  provider lookup, so it also frees a Proxy whose provider is gone from the
+  config.
 - **Events RBAC from the plan is omitted:** nothing wires an
   EventRecorder in the prototype, and granting verbs nothing uses would
   be RBAC lint noise. Add the marker together with the recorder if events

docs/kvm-provider.md

@@ -27,11 +27,12 @@ Per host, once:
    `edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default (libvirt picks
    the OVMF binary itself); a domain start fails with "no firmware
    found" without it. Only `firmware: bios` images skip this.
-6. Optionally, **outbound internet from the VMs**: the default
-   vendor-data installs `qemu-guest-agent` from the distro mirrors on
-   first boot. Without it the VM still works — the operator falls back to
-   the host's ARP table to find the IP, which fills in once the guest
-   sends traffic.
+6. Optionally, **outbound internet from the VMs**: a vendor-data script
+   installs `qemu-guest-agent` from the distro mirrors on first boot.
+   Without it the VM still works — the operator falls back to the host's
+   ARP table to find the IP, which fills in once the guest sends traffic —
+   but the ARP table only sees traffic to the host itself, so on a plain
+   bridge the agent is in practice required.
 
 Check from the operator's point of view before wiring it in:
 
@@ -76,9 +77,10 @@ ssh virt@nas1.home.lan ip link show br0
    `/etc/proxy-operator/kvm` as two files, `id_ed25519` and
    `known_hosts`, which is what `sshKeyFile` and `knownHostsFile` in the
    provider config point at. The deployment starts fine without the
-   Secret as long as no `kvm` provider is configured. Rotating the key:
-   re-run the command (or the make target) and restart the manager; it
-   reads both files once at startup.
+   Secret, even with a `kvm` provider configured: only that provider's
+   proxies fail, and they recover once the Secret exists. Rotating the
+   key: re-run the command (or the make target) and restart the manager;
+   both files are read once, on the first connection, and cached.
 
 2. Add a provider entry to the providers config
    (`config/manager/providers_config.yaml` in-cluster; full sample in
@@ -102,9 +104,13 @@ ssh virt@nas1.home.lan ip link show br0
            bridge: br0
    ```
 
-   The operator reads the key and known_hosts at startup and refuses to
-   start if either is missing — a bad path fails fast, not on the first
-   Proxy.
+   The operator checks both files at startup and logs a `WARNING:` line
+   when either is unreadable, but it does not refuse to start: the
+   provider set is built all-or-nothing, so exiting here would crash-loop
+   the manager and take the `kubernetes` and `gcp` providers with it.
+   The files are read for real on the first connection. A missing file is
+   a transient failure, so proxies heal by themselves once the Secret
+   lands; a key that does not parse is permanent and needs a fix.
 
 3. Apply a Proxy (`config/samples/proxy_kvm.yaml`) and watch it:
 
@@ -129,13 +135,15 @@ ssh virt@nas1.home.lan ip link show br0
 | `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against |
 | `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely |
 | `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it |
-| `vendorData` | | installs `qemu-guest-agent` | cloud-init vendor-data placed on every seed next to the Proxy's user-data |
+| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on |
+| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table |
+| `images.<key>` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only — it becomes the cached volume name `base-<key>.<format>` |
 | `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) |
 | `images.<key>.sha256` | | | expected digest; verified after download, a change triggers a re-download |
 | `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
 | `images.<key>.firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition; Debian 13 hangs in GRUB under BIOS on a headless VM, see *Known gaps* |
 | `machineTypes.<key>` | ≥ 1 | | `{vcpus, memoryMiB}` size profile |
-| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique |
+| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique. Letters, digits, dots, dashes and underscores only — it becomes part of the providerID `hosts/<name>/domains/<vm>` |
 | `hosts[].address` | yes | | SSH endpoint, `host:port` |
 | `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
 | `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
@@ -174,6 +182,28 @@ console, the guest-agent channel, a virtio RNG and no graphics. Its
 creation time) — that is how orphan GC recognises its own VMs, since
 libvirt has no labels.
 
+The seed's `user-data` is the Proxy's `cloudInit`, **byte for byte**. Any
+format cloud-init accepts works, including a Jinja template, a MIME
+archive, a shell script or a gzipped payload; the operator never parses or
+re-renders it.
+
+The guest agent rides along in `vendor-data`, as a **shell script** rather
+than a cloud-config. That is the only shape cloud-init cannot drop:
+vendor-data cloud-config is merged under the user's own document with
+mergers that keep the user-data's value for every key both define and never
+append lists, so a Proxy carrying its own `packages:` silently lost the
+agent and sat in `Provisioning` forever. The strategy cannot be overridden
+from either document — `merge_how` in user-data governs only the first
+source, and `merge_how` in vendor-data is consumed and discarded before the
+document is stored. A vendor-data script skips all of it: cloud-init gives
+vendor-data its own handler chain, writes scripts to a separate directory
+and runs them from `cc_scripts_vendor`, after the user's packages are
+installed and before the user's `runcmd`. The script is idempotent, picks
+whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk` the image has, and
+logs into `/var/log/cloud-init-output.log`. Set `installGuestAgent: false`
+to leave it out. When `vendorData` is also configured, the two travel as
+separate parts of a MIME multipart vendor-data.
+
 ## The image cache
 
 The base image is downloaded **by the operator, through libvirt**, into
@@ -247,6 +277,14 @@ The provider logs through controller-runtime's logger, named `kvm`, with
   cleaned when the same Proxy reconciles again (Delete removes volumes by
   name even with no domain), but a Proxy CR that is gone for good leaves
   them for `virsh vol-list` to find.
+- **A host removed from the config takes its VMs with it.** `Delete` for a
+  providerID naming an unconfigured host is a no-op success, so the Proxy
+  finishes deleting instead of pinning its finalizer, but the domain and its
+  volumes stay on the hypervisor and orphan GC cannot see them either (it
+  only sweeps configured hosts). Delete the Proxies before retiring a host,
+  or clean up afterwards with `virsh undefine --nvram` plus `virsh vol-delete`.
+  `Get` still fails permanently for such a providerID on purpose: a typo in
+  `hosts` must not look like a missing VM and trigger a rebuild elsewhere.
 - **The first image fetch is synchronous.** It holds a reconcile worker
   for the download (LAN speed, a few hundred MB, once per image per
   host). Other Proxies keep reconciling on the remaining workers.
@@ -254,7 +292,12 @@ The provider logs through controller-runtime's logger, named `kvm`, with
   under *The image cache*.
 - **IP discovery needs either guest traffic or the agent.** A VM on a
   bridge with no DHCP never gets an address and stays `Provisioning`
-  forever; check `virsh console`.
+  forever; check `virsh console`. The same symptom with a booted, addressed
+  VM means the guest agent never came up: check `systemctl status
+  qemu-guest-agent`, `cloud-init status --long` and the vendor script's
+  output in `/var/log/cloud-init-output.log` (a VM without outbound
+  internet cannot install it). A user-data that disables vendor-data
+  (`vendor_data: {enabled: false}`) also removes the agent, by design.
 - **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
   ([Debian bug #1111240](https://bugs.debian.org/1111240), open, in
   `grub2`): trixie's image ships `terminal_output gfxterm serial` in its

docs/plans-executions/2026-09-04-1404-kvm-provider.md

@@ -10,7 +10,7 @@ Plan: [2026-09-04-1404-kvm-provider.md](../plans/2026-09-04-1404-kvm-provider.md
 - [x] Step 4 — Provider methods (`kvm.go`) over a fake API
 - [x] Step 5 — Wiring, manifests, samples, docs, CHANGELOG
 - [x] Step 6 — Integration test against a real hypervisor (written and compiled; first real run is part of Step 7)
-- [ ] Step 7 — Manual validation + MR
+- [x] Step 7 — Manual validation + MR
 
 ## Step 0 — Save plan + branch
 
@@ -557,3 +557,117 @@ on the host it needs `virsh undefine --nvram` plus its two volumes by
 hand. The `ListByTag` sweep costs one `virDomainGetMetadata` per
 domain on the host (21 here, ~0.4 ms each over the tunnel), which is
 fine at this scale but is the call to watch on a busy hypervisor.
+
+### Run 7: in-cluster, two defects the integration test could not see
+
+The e2e pass ran the operator inside the kind cluster
+(`egress-proxies-operator:kvm-dev`, built with `make docker-build`,
+loaded with `kind load docker-image`, deployed via the kustomize
+`config/manager` with the `kvm-home` provider pointing at beelink) and
+two Proxies with real cloud-configs: one plain squid, one with a debug
+login user. Both defects come from the difference between the test
+harness and the real thing, not from libvirt.
+
+**Read-only root filesystem.** The manager runs with
+`readOnlyRootFilesystem: true`, and `iso9660.NewWriter` stages the seed
+in a temp directory, so the first Create failed with `creating iso
+writer: read-only file system`. Fix: a 64 MiB `emptyDir` on `/tmp` in
+`config/manager/manager.yaml`. Side finding: the error is classified
+`ErrPermanent` by `writeSeed`, yet `hostPool.call` treated any
+non-libvirt error as a transport failure and dropped the SSH connection,
+so every retry redialled. `call` now leaves the connection alone for
+errors already carrying a provider sentinel (`isClassified`), with a
+test.
+
+**The guest agent never got installed.** Both VMs booted (confirmed on
+the console by hand) but sat in `Provisioning` for 25 minutes: the
+trace showed `virDomainInterfaceAddresses` from the agent source failing
+with `ErrAgentUnresponsive` on every poll and the ARP source empty. The
+integration test had passed with `CloudInit: "#cloud-config\n"`; the
+real Proxies have `packages: [squid]` and `runcmd:`. cloud-init merges
+vendor-data under user-data with its default mergers, which keep the
+user-data's value for any key both define and never append lists, so the
+vendor-data's `packages`/`runcmd` were dropped wholesale. Confirmed
+locally with the dev box's cloud-init 26.1 before changing anything:
+
+```sh
+python3 - <<'EOF'
+from cloudinit import mergers
+user = {"packages": ["squid"], "runcmd": ["systemctl restart squid"]}
+vendor = {"packages": ["qemu-guest-agent"], "runcmd": [["systemctl", "enable", "--now", "qemu-guest-agent"]]}
+print(mergers.construct(mergers.default_mergers()).merge(dict(user), dict(vendor)))
+EOF
+# {'packages': ['squid'], 'runcmd': ['systemctl restart squid']}
+```
+
+Fix: `withGuestAgent` in `seed.go` merges the bootstrap into the Proxy's
+cloud-config itself (append to `packages` and `runcmd`, skip if present,
+re-render with `sigs.k8s.io/yaml`), so vendor-data is no longer defaulted
+and `vendorData` is plain extra data. User-data that is not a cloud-config
+document (script, `#include`, MIME) still gets the bootstrap as
+vendor-data since there is nothing to collide with; a cloud-config that
+does not parse as a mapping fails the Proxy as `ErrPermanent`. The
+rendered document for the squid Proxy passed `cloud-init schema
+--config-file`. The logging test's fixture (`#cloud-config` + a bare
+sentinel line) is not a mapping and had to become `hostname: <sentinel>`.
+
+Rebuilt, reloaded, restarted, deleted and recreated both Proxies: both
+`Ready`/`HEALTHY True` in 2 min 1 s from `kubectl apply`, IP via the
+agent 110 s after `virDomainCreate` (the guest installs squid and the
+agent from the mirrors first), and `curl -x http://<ip>:3128
+https://www.gstatic.com/generate_204` from the dev box returns 204
+through both.
+
+Worth noting: the `#cloud-config` round-trip drops comments and reorders
+keys alphabetically; semantics are preserved (numbers, booleans and
+block scalars survive, checked in tests) but a user diffing the seed
+against their CR will see a different document. Vendor-data supplied via
+`vendorData` is subject to the same cloud-init merge rule and is now
+documented as such. The kind pod also restarted twice during the session
+from "Pod sandbox changed" (the node briefly lost the API server), which
+is unrelated to the operator. Still uncommitted on purpose: the
+`kustomization.yaml` image override (`egress-proxies-operator:kvm-dev`)
+and the `beelink` host entries in `config/manager/providers_config.yaml`
+and `config/samples/providers-config.yaml`, which are this machine's dev
+setup rather than the repo's defaults.
+
+### Run 8: review fixes
+
+A `/code-review high` pass over the branch found five defects, fixed in
+[plan 2026-09-10-2222](../plans/2026-09-10-2222-kvm-review-fixes.md) with
+its own execution log. Two of them land on Run 7's conclusions above and
+supersede them:
+
+- The guest-agent **merge described in Run 7 is gone**. It could not be made
+  safe for user-data that is not a plain cloud-config, and its YAML
+  round-trip re-typed unquoted scalars. The bootstrap now ships as a
+  vendor-data shell script, which cloud-init never merges, and the Proxy's
+  cloud-init reaches the seed byte for byte.
+- The read-only-filesystem fix from Run 7 still stands, but the eager SSH
+  key load it sat next to does not: a missing `kvm-ssh` Secret aborted the
+  whole provider set and crash-looped the manager. The material is read on
+  first use now.
+
+Re-validated the same way: integration test against beelink, both Proxies
+recreated in kind, 204 through squid.
+
+### MR opened
+
+[PR #5](https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/pulls/5),
+`feat/kvm-provider` → `main`, 19 commits. `tea` is not installed on the
+dev box and the HTTPS remote has no credential helper, so the push used a
+one-off helper reading a Gitea access token from `~/.config/gitea-token`,
+and the MR was created with the API instead of `tea pr create`:
+
+```sh
+git -c credential.helper='!f() { echo username=kacerr; echo "password=$(tr -d "[:space:]" < ~/.config/gitea-token)"; }; f' \
+  push -u origin feat/kvm-provider
+curl -s -H "Authorization: token $(tr -d '[:space:]' < ~/.config/gitea-token)" \
+  -H 'Content-Type: application/json' -d @pr.json \
+  https://gitea.home.hrajfrisbee.cz/api/v1/repos/kacerr/egress-proxies-operator/pulls
+# pr.json: {"title": ..., "head": "feat/kvm-provider", "base": "main", "body": ...}
+```
+
+The push printed a libsecret "store failed" warning from the system
+credential store; harmless, the token was not persisted anywhere but the
+file.

docs/plans-executions/2026-09-10-2222-kvm-review-fixes.md

@@ -0,0 +1,229 @@
+# Execution: fix the five code-review findings on `feat/kvm-provider`
+Plan: [docs/plans/2026-09-10-2222-kvm-review-fixes.md](../plans/2026-09-10-2222-kvm-review-fixes.md)
+
+- [x] Step 0 — Save plan into the repo
+- [x] Step 1 — Unwedge deletion (finding 1)
+- [x] Step 2 — Validate names that become identifiers (finding 2)
+- [x] Step 3 — Read the SSH material lazily (finding 3)
+- [x] Step 4 — Guest agent via vendor-data script (findings 4 and 5)
+- [x] Step 5 — Validate, record, push
+
+## Step 0 — plan committed
+
+Copied out of `~/.claude/plans/` and committed alone before touching code, as
+the repo's plan rules require.
+
+## Step 1 — Unwedge deletion
+
+Three changes, in the order the bug unfolds.
+
+`reconcileDelete` calls `Get` first and only drops the finalizer when it
+answers not-found, so any permanent error there retried forever. It now
+switches on the class: a permanent `Get` still calls `Delete`, which is
+idempotent, and releases the object when that succeeds. Polling a `Get`
+that cannot converge was the wedge; `Delete` is the only call that can
+still make progress.
+
+`crawl.example.com/skip-cleanup: "true"` abandons the instance and removes
+the finalizer. It is checked **before** the provider lookup, which also
+frees a Proxy whose provider was deleted from the config — a second wedge
+sitting three lines above the reported one, with the same shape. No
+`EventRecorder` exists on the reconciler and none was added, so the
+abandonment is an Info log; `docs/architecture.md` already records that
+decision for the whole controller.
+
+In the KVM provider, `Delete` for a providerID naming an unconfigured host
+now logs and returns success. The precedent is two functions further down:
+`removeVolumes` treats a missing pool the same way, with a comment saying
+outright that refusing "would pin the Proxy's finalizer".
+
+**`Get` deliberately still fails permanently for that case.** Making it
+answer not-found would have been a one-line fix for the wedge, but the live
+reconcile path reads not-found as "the VM is gone" and rebuilds it
+elsewhere. A typo in `hosts` would then quietly duplicate every Proxy on
+that host, with the originals still running where GC cannot see them, since
+`ListByTag` only sweeps configured hosts. The asymmetry is now pinned by a
+test that asserts both halves at once.
+
+## Step 2 — Validate names that become identifiers
+
+One regexp, `^[A-Za-z0-9][A-Za-z0-9_.-]*$`, applied to `hosts[].name` and to
+the `images` map keys. Dots stay legal because a host name like
+`nas1.home.lan` is ordinary and harmless; the character that actually
+breaks things is `/`.
+
+The review only reported the host name. The image key has the same defect
+and is arguably worse: it is not merely embedded in an ID but pasted into a
+file name, `base-<key>.<format>`, so `../etc/x` would be a path traversal
+into the storage pool rather than a parse failure. Both are validated in the
+same pass. Machine-type keys are left alone deliberately — they are only
+ever compared against `placement.machineType`, never embedded anywhere.
+
+Worth noting: this is validation of the operator's own config file, which
+only a cluster admin can write, so the traversal was never a privilege
+boundary. It is still cheaper to reject at load than to explain a stuck
+Proxy later. Both new table cases assert the message text, so the wording
+that names the consequence is now pinned by tests.
+
+## Step 3 — Read the SSH material lazily
+
+`kvm.New` no longer loads the key; `lazyAuth` does, on the first connection,
+caching the result once it works. `registry.Build` is all-or-nothing by
+design, so the old constructor error did not merely fail the KVM provider,
+it exited the manager before any controller started.
+
+The plan said "lazy dialer", and the first attempt was exactly that — one
+closure wrapping both the loading and the dial. It could not be tested: the
+fixture's SSH server is real, but the socket behind it is an echo server, so
+every successful dial dies in the libvirt handshake and the assertion could
+not tell "the key worked" from "the key did not". Splitting the seam in two,
+`lazyAuth` for the material and `lazyDialer` for the wiring, made both
+testable: the healing path against `lazyAuth`, and a separate test that the
+dialer really authenticates, asserted by the error being past SSH and by the
+server having seen the forward request.
+
+Error classes carry the intent. A missing file stays unclassified, so
+`provider.Class` defaults it to transient and a Proxy heals on its own when
+the Secret appears — the common case, since the Secret is usually created
+after the manager. A key that does not parse wraps `ErrPermanent`, because
+retrying cannot fix it. `newSSHAuth` distinguishes the two with
+`errors.Is(err, fs.ErrNotExist)`.
+
+Worth noting: the startup check is kept, just non-fatally, logged with the
+`WARNING:` prefix the repo already uses where logr has no warn level. That
+preserves the diagnostic the eager version was written for. The caching test
+deliberately deletes the key file after a successful load and asserts the
+cached value is reused, which pins the rotation behaviour the docs promise:
+a rotated Secret needs a manager restart. `TestNew_badKeyPathFailsFast` was
+inverted rather than deleted, so the old contract stays visible in the diff
+next to the new one.
+
+## Step 4 — Guest agent via vendor-data script
+
+The review filed two findings against `withGuestAgent` — detection too
+narrow, and YAML re-typing on the round-trip. Reading cloud-init 26.1's
+sources while planning turned both into one finding: the operator should
+not be editing the user's document at all.
+
+Vendor-data gets its own handler chain (`stages.py:_default_vendordata_handlers`),
+with separate script and cloud-config paths. A vendor-data **script** is
+therefore written to the vendor script directory and run by
+`cc_scripts_vendor`, never touching the config merger that drops colliding
+keys. On the final-stage module list it lands after
+`package_update_upgrade_install` and before the user's `runcmd`, so the
+agent now comes up *earlier* than the old `runcmd` append did.
+
+That also closed the door the previous approach had left ajar. The merge
+cannot be rescued from inside either document: `merge_how` in user-data
+governs only the first source, and `merge_how` in vendor-data is popped by
+`dict_extract_mergers` and discarded before the document is written. There
+was no fix that kept the merge.
+
+`withGuestAgent`, `isCloudConfig` and `appendUnique` are gone, along with
+seed.go's `sigs.k8s.io/yaml` dependency. `installGuestAgent` (default true)
+turns the script off. When `vendorData` is also configured the two ship as
+a MIME multipart, with the user's document declared `text/plain` — a type
+cloud-init re-sniffs from the content's own header line, so this code never
+classifies what the user wrote.
+
+Before deleting the old path, its damage was captured rather than taken on
+trust. Round-tripping a plausible cloud-config through it:
+
+```sh
+# temporary test in the package, deleted afterwards
+go test -run TestZZProbeRetyping -v ./internal/provider/kvm/
+# in:  version: 1.10
+# out: version: 1.1
+```
+
+`1.10` arriving as `1.1` is worse than the reviewer's `passwd` example,
+since nothing about it looks wrong in the seed. Comments were dropped and
+keys reordered on the same pass.
+
+The generated vendor-data was then checked against the cloud-init actually
+installed on this machine, rather than against the docs:
+
+```sh
+python3 -c 'from cloudinit import user_data, handlers; ...'
+# vendor-script:    multipart=False  sniffed=text/x-shellscript
+# vendor-multipart: multipart=True   part 1 text/plain -> text/cloud-config
+#                                    part 2 text/x-shellscript
+```
+
+and the script itself through `sh -n`.
+
+Worth noting: the script has to choose a package manager where cloud-init's
+`packages:` key was distro-agnostic. It covers apt, dnf, yum, zypper and
+apk, and both systemd and OpenRC for the enable step, but an image outside
+that set now needs `installGuestAgent: false` and the agent baked in. The
+seed's user-data is byte-identical to the Proxy's `cloudInit` from here on,
+which is what the new `TestBuildSeedISO_userDataIsNeverRewritten` pins
+across seven formats. One consequence worth remembering: a user-data that
+sets `vendor_data: {enabled: false}` disables the agent, and cloud-init
+documents that as intended behaviour.
+
+## Step 5 — Validation
+
+Unit suite, `go vet` and `gofmt` clean across the repo. Then the two real
+runs.
+
+**Integration against beelink**, unchanged environment from the KVM plan's
+log, passed in 22 s on a warm cache, with
+`virDomainInterfaceAddresses source=DomainInterfaceAddressesSrcAgent
+interfaces=2` — the agent installed by the new vendor script answering, on
+a stock Debian 13 image.
+
+**In-cluster**, both Proxies deleted and recreated against the rebuilt
+image: Ready with an address 2 min 3 s after `kubectl apply`, 204 through
+squid from the dev box for both. Inside the debug VM:
+
+```sh
+ssh novakj@192.168.0.94 'systemctl is-active qemu-guest-agent; cloud-init status; \
+  sudo head -4 /var/lib/cloud/instance/user-data.txt; sudo ls /var/lib/cloud/instance/scripts/vendor/'
+# active
+# status: done
+# #cloud-config
+# # Login for debugging: user "novakj", password "kvm-dev", passwordless sudo.
+# ...
+# part-001
+```
+
+Two things worth having seen with your own eyes there: the user's **comment
+lines survived** into the guest, which the old merge stripped, and the
+script really did land in the vendor directory rather than being merged
+into the user's config.
+
+**Negative check for the Secret**, done by pointing the deployment's volume
+at a name that does not exist rather than deleting the real Secret:
+
+```sh
+kubectl -n egress-proxies-operator-system patch deploy egress-proxies-operator-controller-manager \
+  --type=json -p '[{"op":"replace","path":"/spec/template/spec/volumes/1/secret/secretName","value":"kvm-ssh-absent"}]'
+```
+
+The manager came up and stayed up, 0 restarts, logging
+`WARNING: kvm ssh material is not usable ... open /etc/proxy-operator/kvm/id_ed25519:
+no such file or directory`. The Kubernetes Proxies kept reconciling; the
+KVM ones failed per reconcile and retried. Before this branch the same
+config exited the process before any controller started. Patched back
+afterwards, everything Ready.
+
+Worth noting: the KVM Proxies kept reporting `Ready` during that window
+rather than degrading visibly. That is existing behaviour, not a
+regression — a transient provider error rides the workqueue and does not
+clear a healthy status — but it does mean the only signal of a missing
+Secret is the log and the reconcile errors. Also, the ListByTag sweep on
+beelink still reports `managed=2` where one domain is an orphan from an
+older failed run; unrelated to this work, and the same leftover noted in
+the KVM plan's Run 6.
+
+Worth noting: the review filed this as a KVM finding, but the fix is mostly
+in the controller, because KVM is the only provider that resolves a
+providerID segment against operator config at all. GCP and Kubernetes hand
+the lookup to the backend, which answers 404 for a retired zone or a deleted
+namespace, so they reach not-found and release on their own. Anything else
+that classifies permanently on the delete path — revoked RBAC on the
+Kubernetes provider is the realistic one — was equally stuck before this
+step and is what the annotation is really for. The KVM table case
+`"host not configured"` moved out of `TestDelete_errors` into its own named
+test, since it is no longer an error.

docs/plans/2026-09-10-2222-kvm-review-fixes.md

@@ -0,0 +1,195 @@
+# Plan: Fix the five code-review findings on `feat/kvm-provider`
+**Created:** 2026-09-10 22:22 UTC
+
+## Context
+
+PR #5 (`feat/kvm-provider` → `main`, 19 commits) is open and awaiting merge. A
+`/code-review high` pass over the branch returned five confirmed findings: two that can
+pin a Proxy's finalizer forever, one startup crash-loop, and two in the guest-agent merge
+committed earlier today. All five are fixed on the same branch so the PR updates in place,
+then re-validated against the real hypervisor before merging.
+
+Four decisions, made by the user during planning:
+
+1. **Wedged deletes** — attempt `Delete` even when `Get` fails permanently, make an
+   unconfigured KVM host a no-op delete, and add an annotation that abandons cleanup for
+   whatever still cannot finish.
+2. **Missing `kvm-ssh` Secret** — read the SSH material lazily, so only KVM Proxies fail.
+3. **Guest agent** — stop rewriting the user's cloud-config. Ship the bootstrap as a
+   vendor-data shell script instead.
+4. Validation is unit tests, the integration test against beelink, and the kind e2e. The
+   uncommitted dev-only files stay exactly as they are and out of the PR.
+
+## Why the guest-agent fix is a rewrite, not a patch
+
+Both cloud-init findings come from the same root cause: this operator edits the user's
+document. Reading cloud-init 26.1's sources settles that it never has to.
+
+Vendor-data is consumed by its **own** handler chain with separate script and config paths
+(`stages.py:_default_vendordata_handlers`). A vendor-data **shell script** is therefore
+written to the vendor script directory and run by `cc_scripts_vendor`, never passing
+through the config merger that drops colliding keys. On the final-stage module list it
+runs after `package_update_upgrade_install` and before the user's own `runcmd`, so the
+agent comes up early regardless of what the user's config does.
+
+That also closes the door the previous approach left open. The merge cannot be rescued
+from inside the documents: `merge_how` in the user-data governs only the first source, and
+`merge_how` in vendor-data is popped and discarded by `dict_extract_mergers` before the
+document is ever written. There is no supported way to make vendor-data cloud-config keys
+survive a user-data that defines the same key.
+
+Consequences: user-data reaches the seed byte for byte in any format, so Jinja templates,
+MIME archives, gzip payloads and shell scripts stop being special cases (finding 4), and
+nothing re-serialises YAML, so unquoted scalars keep their type (finding 5). The cost is
+that the script picks the package manager itself where cloud-init's `packages` key was
+distro-agnostic.
+
+## Why the delete fix is shaped this way
+
+KVM is the only provider that resolves a providerID segment against operator config. GCP
+and Kubernetes let the backend answer "not found", so retiring a location yields
+`ErrNotFound` and the finalizer drops. `Provider.host()`
+([kvm.go:118-139](internal/provider/kvm/kvm.go#L118-L139)) returns `ErrPermanent`, and
+`reconcileDelete` ([proxy_controller.go:281-291](internal/controller/proxy_controller.go#L281-L291))
+drops the finalizer only on `ErrNotFound`. `deletionFailure`
+([:327-336](internal/controller/proxy_controller.go#L327-L336)) says in its own comment
+that it retries "until an operator intervenes", and no intervention mechanism exists. The
+annotation is that mechanism, and it also rescues the unconfigured-provider wedge at
+[:276-279](internal/controller/proxy_controller.go#L276-L279).
+
+`Get` keeps returning `ErrPermanent` for an unknown host deliberately. Turning it into
+`ErrNotFound` would make a typo in the host list look like a missing VM on the live path,
+and the reconciler would recreate every affected Proxy elsewhere while the originals keep
+running where garbage collection cannot see them.
+
+## Steps
+
+One commit per step on `feat/kvm-provider`, each ending with
+`Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>`.
+
+### Step 0 — Save this plan into the repo
+Copy to `docs/plans/2026-09-10-2222-kvm-review-fixes.md` with the same header timestamp and
+commit it alone, before any code change. Start
+`docs/plans-executions/2026-09-10-2222-kvm-review-fixes.md` with the status checklist when
+Step 1 lands.
+
+### Step 1 — Unwedge deletion (finding 1)
+- `api/v1alpha1/proxy_types.go`: add `AnnotationSkipCleanup = "crawl.example.com/skip-cleanup"`
+  beside `AnnotationSpecHash` ([:59-63](api/v1alpha1/proxy_types.go#L59-L63)), documented as
+  abandoning the backend instance.
+- `internal/controller/proxy_controller.go`, `reconcileDelete`: honour the annotation after
+  the empty-providerID branch and **before** the provider lookup, so it also covers a
+  provider that is no longer configured. Log the abandoned providerID at Info.
+  `ProxyReconciler` has no `EventRecorder`; do not add one for this.
+- Same function: replace the two `if`s after `Get` with a switch. On
+  `provider.Class(err) == provider.ErrPermanent`, call `Delete` anyway, since it is
+  idempotent, and remove the finalizer when it succeeds rather than requeueing into a `Get`
+  that can never converge. On failure, `deletionFailure(errors.Join(getErr, delErr))`.
+  Guard on `err != nil`, since `Class(nil)` is nil.
+- `internal/provider/kvm/kvm.go`, `Delete`: an unconfigured host segment means there is
+  nothing this operator can clean, so log it and return success, mirroring `removeVolumes`
+  ([kvm.go:506-514](internal/provider/kvm/kvm.go#L506-L514)) for a missing pool. `Get` is
+  unchanged.
+- Tests: extend the deletion table in `internal/controller/reconcile_test.go`
+  ([:357-393](internal/controller/reconcile_test.go#L357-L393)) with permanent-Get plus
+  succeeding Delete, permanent-Get plus failing Delete, the annotation with the provider
+  never called, and the annotation on an unconfigured provider. In `kvm_test.go`, the
+  `"host not configured"` case at [:737](internal/provider/kvm/kvm_test.go#L737) flips from
+  `ErrPermanent` to success; add a named test beside `TestDelete_missingPoolIsNotAnError`
+  ([:715](internal/provider/kvm/kvm_test.go#L715)).
+- Docs: the annotation in `README.md` (~line 163) and `docs/architecture.md` (~line 487);
+  a *Known gaps* entry in `docs/kvm-provider.md` saying a domain on a host removed from
+  config can no longer be cleaned up by the operator, the Proxy still finishes deleting,
+  and the VM must be removed by hand because garbage collection only sweeps configured
+  hosts.
+
+### Step 2 — Validate names that become identifiers (finding 2)
+- `internal/provider/config_kvm.go`: add a name regexp beside `sha256Hex`
+  ([:6](internal/provider/config_kvm.go#L6)) allowing letters, digits, dot, dash and
+  underscore. Apply it to `hosts[].name` after the existing empty check
+  ([:129-131](internal/provider/config_kvm.go#L129-L131)) and to each `images` map key,
+  which becomes the cached volume name `base-<key>.<format>`
+  ([image.go:49-55](internal/provider/kvm/image.go#L49-L55)) and half of the image-cache
+  lock key ([image.go:96](internal/provider/kvm/image.go#L96)). `machineTypes` keys are only
+  compared, never embedded, so leave them.
+- Keep the file's first-error-wins style and `fmt.Sprintf` field paths. The host message
+  should name the consequence: the value becomes part of `hosts/<name>/domains/<vm>`, which
+  `parseProviderID` splits on `/` ([id.go:11-21](internal/provider/kvm/id.go#L11-L21)).
+- Tests: two entries in the `TestLoadConfig_kvmInvalid` table
+  ([config_kvm_test.go:76](internal/provider/config_kvm_test.go#L76)), existing
+  name/yaml/wantErrSub shape.
+
+### Step 3 — Read the SSH material lazily (finding 3)
+- `internal/provider/kvm/conn.go`: keep `newSSHAuth`, and add a lazy dialer that builds it
+  on first dial and caches it once it succeeds, so a `kvm-ssh` Secret created after the
+  manager starts heals on the next reconcile with no restart.
+- `internal/provider/kvm/kvm.go`, `New`: stop calling `newSSHAuth`; pass the lazy dialer.
+  Use the context parameter it currently ignores to log a warning when either file is
+  unreadable at startup, keeping the diagnostic without the exit.
+- Error classes: a missing or unreadable file stays unclassified, so `Class` defaults it to
+  transient and the Proxy retries until the Secret appears; an unparseable key or malformed
+  known_hosts wraps `ErrPermanent`. Distinguish with `errors.Is(err, fs.ErrNotExist)`.
+- Fix the two doc comments promising startup failure
+  ([kvm.go:79-81](internal/provider/kvm/kvm.go#L79-L81),
+  [conn.go:182-185](internal/provider/kvm/conn.go#L182-L185)) and the matching claim in
+  `docs/kvm-provider.md` (~lines 107-109). `DISCOVERY_TOKEN` already sets the house
+  convention for an optional mount: degrade with a loud warning.
+- Tests in `conn_test.go`, which already writes key files: `New` succeeds with no key
+  present, the first dial fails transiently, a dial after the file appears succeeds, and a
+  corrupt key is permanent.
+
+### Step 4 — Guest agent via vendor-data script (findings 4 and 5)
+- `internal/provider/kvm/seed.go`: delete `withGuestAgent`, `isCloudConfig`, `appendUnique`
+  and the `sigs.k8s.io/yaml` import. User-data goes onto the seed exactly as the Proxy
+  supplied it.
+- Add the bootstrap as a `#!/bin/sh` vendor-data script: skip when the agent is already
+  present, otherwise install `qemu-guest-agent` through whichever of `apt-get`, `dnf`,
+  `yum`, `zypper` or `apk` exists, then `systemctl enable --now`. Run `apt-get update`
+  before installing, since nothing else guarantees a fresh index when the user's config has
+  no `packages` key. Keep it idempotent and let it log to stdout, which lands in
+  `cloud-init-output.log`.
+- When the provider config also sets `vendorData`, wrap both into a MIME multipart
+  vendor-data: the user's document as one part, the script as another. Cloud-init detects
+  multipart by finding `mime-version:` in the first 4096 bytes, so the header must lead.
+- `internal/provider/config_kvm.go`: add `installGuestAgent`, defaulting to on. Off means
+  the seed carries only the user's own vendor-data and IP discovery falls back to the host
+  ARP table. Nothing fails permanently any more, so the error path in `writeSeed`
+  ([kvm.go:252-255](internal/provider/kvm/kvm.go#L252-L255)) narrows to MIME assembly.
+- Tests in `seed_test.go`: user-data byte-identical for cloud-config, Jinja, MIME, gzip and
+  script inputs; the script present in vendor-data; the opt-out; the multipart case
+  including the `mime-version:` position; keep the existing ISO tests.
+- Docs: rewrite the *What lands on the host* passage added earlier today, the `vendorData`
+  row and a new `installGuestAgent` row in the config table, and the ARP note in
+  *Known gaps*. Cite the cloud-init behaviour that forced the change.
+
+### Step 5 — Validate, record, push
+- `gofmt -l internal/`, `go vet ./...`, `go test -short ./...`.
+- `make test-integration-kvm` against beelink, environment as recorded in the KVM execution
+  log.
+- Rebuild the image, `kind load`, restart the manager, delete and recreate both Proxies,
+  curl a test URL through each.
+- `CHANGELOG.md` entry at the top with a real `date` timestamp, noting it supersedes the
+  merge approach described in today's earlier entry. Add a Run 8 section to
+  `docs/plans-executions/2026-09-04-1404-kvm-provider.md` plus this plan's own execution
+  file. Push with the token helper; PR #5 updates in place.
+
+## Verification
+- Unit: the controller cases prove a permanent `Get` no longer pins the finalizer and that
+  the annotation short-circuits before the provider is touched. `TestDelete_*` proves an
+  unconfigured host is a clean delete while `TestGet_*` stays permanent, which is what keeps
+  the live path from recreating VMs.
+- Seed: assert the uploaded user-data equals the Proxy's `cloudInit` byte for byte, which is
+  the property the old merge could not offer.
+- Integration on beelink: Create, IP via the guest agent, ListByTag, Delete. In the guest,
+  `cloud-init status --long` clean and `systemctl is-active qemu-guest-agent` active.
+- In-cluster: both Proxies reach Ready with an IP and return 204 through squid. Their
+  cloud-config defines its own `packages`, which is the exact case that failed in Run 7.
+- Negative check in kind: rename the `kvm-ssh` Secret away and restart. The pod must stay
+  up, the Kubernetes Proxies must stay Ready, and only the KVM Proxies degrade. Restore the
+  Secret and confirm they recover without a restart.
+
+## Out of scope
+- The two items the review noted but did not report: deleting a cached base image while
+  running VMs back onto it, and an ad-hoc image URL always being treated as qcow2. Both are
+  documented known gaps and neither is a regression from this branch.
+- The uncommitted dev-only files, by the user's decision.

internal/controller/proxy_controller.go

@@ -270,8 +270,14 @@ func (r *ProxyReconciler) reconcileDelete(ctx context.Context, p *crawlv1alpha1.
 	if p.Status.ProviderID == "" {
 		// Nothing was ever recorded as created; orphan GC reaps any stray
 		// instance a crashed create might have left behind.
-		controllerutil.RemoveFinalizer(p, crawlv1alpha1.FinalizerName)
-		return ctrl.Result{}, r.Update(ctx, p)
+		return r.releaseFinalizer(ctx, p)
+	}
+	if p.Annotations[crawlv1alpha1.AnnotationSkipCleanup] == "true" {
+		// Checked before the provider lookup so it also releases a Proxy whose
+		// provider is gone from the config. The instance is left running.
+		logf.FromContext(ctx).Info("skip-cleanup annotation set; abandoning the instance",
+			"providerID", p.Status.ProviderID, "provider", p.Spec.Provider)
+		return r.releaseFinalizer(ctx, p)
 	}
 	prov, ok := r.Providers[p.Spec.Provider]
 	if !ok {
@@ -279,11 +285,22 @@ func (r *ProxyReconciler) reconcileDelete(ctx context.Context, p *crawlv1alpha1.
 			"provider %q is not configured; cannot clean up instance %s", p.Spec.Provider, p.Status.ProviderID)
 	}
 	_, err := prov.Get(ctx, p.Status.ProviderID)
-	if provider.Class(err) == provider.ErrNotFound {
-		controllerutil.RemoveFinalizer(p, crawlv1alpha1.FinalizerName)
-		return ctrl.Result{}, r.Update(ctx, p)
-	}
-	if err != nil {
+	switch {
+	case err == nil:
+	case provider.Class(err) == provider.ErrNotFound:
+		return r.releaseFinalizer(ctx, p)
+	case provider.Class(err) == provider.ErrPermanent:
+		// Get will never answer for this instance — a host retired from the
+		// config, a providerID the provider can no longer resolve. Delete is
+		// idempotent and may still finish, so try it instead of polling a Get
+		// that cannot converge, and release the object when it does.
+		if delErr := prov.Delete(ctx, p.Status.ProviderID); delErr != nil {
+			return r.deletionFailure(errors.Join(err, delErr))
+		}
+		logf.FromContext(ctx).Info("instance unreachable but cleanup succeeded; removing the finalizer",
+			"providerID", p.Status.ProviderID, "error", err.Error())
+		return r.releaseFinalizer(ctx, p)
+	default:
 		return r.deletionFailure(err)
 	}
 	if err := prov.Delete(ctx, p.Status.ProviderID); err != nil {
@@ -294,6 +311,11 @@ func (r *ProxyReconciler) reconcileDelete(ctx context.Context, p *crawlv1alpha1.
 	return ctrl.Result{RequeueAfter: r.DeletionPoll}, nil
 }
 
+func (r *ProxyReconciler) releaseFinalizer(ctx context.Context, p *crawlv1alpha1.Proxy) (ctrl.Result, error) {
+	controllerutil.RemoveFinalizer(p, crawlv1alpha1.FinalizerName)
+	return ctrl.Result{}, r.Update(ctx, p)
+}
+
 func (r *ProxyReconciler) reconcileExternal(_ context.Context, p *crawlv1alpha1.Proxy) (ctrl.Result, error) {
 	if p.Spec.Endpoint == nil {
 		// CEL guarantees an endpoint on any object that went through the API

internal/controller/reconcile_test.go

@@ -2,6 +2,7 @@ package controller
 
 import (
 	"context"
+	"slices"
 	"testing"
 	"time"
 
@@ -78,6 +79,10 @@ func notFoundErr() error {
 	return provider.Wrap(provider.ErrNotFound, "get", "stub", "some-id", nil)
 }
 
+func permanentErr() error {
+	return provider.Wrap(provider.ErrPermanent, "get", "stub", "some-id", nil)
+}
+
 func testScheme(t *testing.T) *runtime.Scheme {
 	t.Helper()
 	s := runtime.NewScheme()
@@ -125,6 +130,15 @@ func withSpecHashAnnotation(hash string) func(*crawlv1alpha1.Proxy) {
 	}
 }
 
+func withSkipCleanup() func(*crawlv1alpha1.Proxy) {
+	return func(p *crawlv1alpha1.Proxy) {
+		if p.Annotations == nil {
+			p.Annotations = map[string]string{}
+		}
+		p.Annotations[crawlv1alpha1.AnnotationSkipCleanup] = "true"
+	}
+}
+
 func deleting() func(*crawlv1alpha1.Proxy) {
 	return func(p *crawlv1alpha1.Proxy) {
 		now := metav1.Now()
@@ -375,6 +389,59 @@ func TestReconcile_actionTable(t *testing.T) {
 				assertProxyGone(t, r)
 			},
 		},
+		{
+			// A permanent Get (a retired KVM host, a providerID the provider
+			// cannot resolve) used to retry forever with the finalizer on.
+			name:       "deletion with an unreachable instance still deletes and releases",
+			proxy:      managedProxy(deleting(), withProviderID("stub-id-1")),
+			stub:       &stubProvider{getErr: permanentErr()},
+			wantResult: ctrl.Result{},
+			verify: func(t *testing.T, r *ProxyReconciler, stub *stubProvider) {
+				assertProxyGone(t, r)
+				if stub.deleteCalls != 1 {
+					t.Errorf("deleteCalls = %d, want 1 — Delete must be tried when Get cannot answer", stub.deleteCalls)
+				}
+			},
+		},
+		{
+			name:  "deletion keeps the finalizer when cleanup itself fails",
+			proxy: managedProxy(deleting(), withProviderID("stub-id-1")),
+			stub: &stubProvider{
+				getErr:    permanentErr(),
+				deleteErr: provider.Wrap(provider.ErrPermanent, "delete", "stub", "stub-id-1", nil),
+			},
+			wantResult: ctrl.Result{},
+			wantErr:    true,
+			verify: func(t *testing.T, r *ProxyReconciler, _ *stubProvider) {
+				// getProxy fails the test if the object is gone, which is what
+				// finalizer removal would have caused.
+				if p := getProxy(t, r); !slices.Contains(p.Finalizers, crawlv1alpha1.FinalizerName) {
+					t.Error("finalizer removed although cleanup failed; the instance would leak silently")
+				}
+			},
+		},
+		{
+			name:       "skip-cleanup annotation abandons the instance",
+			proxy:      managedProxy(deleting(), withProviderID("stub-id-1"), withSkipCleanup()),
+			stub:       &stubProvider{getInst: &provider.Instance{ID: "stub-id-1", State: provider.StateRunning}},
+			wantResult: ctrl.Result{},
+			verify: func(t *testing.T, r *ProxyReconciler, stub *stubProvider) {
+				assertProxyGone(t, r)
+				if stub.getCalls != 0 || stub.deleteCalls != 0 {
+					t.Errorf("getCalls/deleteCalls = %d/%d, want 0/0 — the provider must not be asked", stub.getCalls, stub.deleteCalls)
+				}
+			},
+		},
+		{
+			name: "skip-cleanup annotation releases a proxy whose provider is gone",
+			proxy: managedProxy(deleting(), withProviderID("stub-id-1"), withSkipCleanup(),
+				func(p *crawlv1alpha1.Proxy) { p.Spec.Provider = "retired" }),
+			stub:       &stubProvider{},
+			wantResult: ctrl.Result{},
+			verify: func(t *testing.T, r *ProxyReconciler, _ *stubProvider) {
+				assertProxyGone(t, r)
+			},
+		},
 		{
 			name:  "deletion deletes the instance and polls",
 			proxy: managedProxy(deleting(), withProviderID("stub-id-1")),

internal/provider/config_kvm.go

@@ -22,10 +22,18 @@ type KVMConfig struct {
 	// DiskSizeGB is the capacity of each VM's overlay disk. Default 10. The
 	// guest's cloud-init growpart expands the root filesystem into it.
 	DiskSizeGB int64 `json:"diskSizeGb,omitempty"`
-	// VendorData is cloud-init vendor-data placed on every seed next to the
-	// Proxy's own user-data. Default: installs and starts qemu-guest-agent so
-	// the provider can read the VM's IP even before it shows up in ARP.
+	// VendorData is optional cloud-init vendor-data placed on every seed next
+	// to the Proxy's own user-data. Note that cloud-init drops any top-level
+	// key of a vendor-data cloud-config that the user-data also defines.
+	// When InstallGuestAgent is on, this document and the agent script are
+	// carried as separate parts of a MIME multipart vendor-data.
 	VendorData string `json:"vendorData,omitempty"`
+	// InstallGuestAgent adds a vendor-data script that installs and starts
+	// qemu-guest-agent, which is how the provider learns a VM's address
+	// before the host's ARP table has seen traffic from it. Default true.
+	// Turn it off for images that already ship the agent, or that cannot
+	// reach a package mirror; discovery then depends on ARP alone.
+	InstallGuestAgent *bool `json:"installGuestAgent,omitempty"`
 	// Images are named base images; placement.image names a key here (or is
 	// an http(s) URL directly). The provider downloads each image into a
 	// host's storage pool on first use and caches it there.
@@ -80,8 +88,31 @@ type KVMHost struct {
 	Bridge string `json:"bridge"`
 }
 
+// Accepted KVMImage.Format and KVMImage.Firmware values.
+const (
+	KVMFormatQcow2  = "qcow2"
+	KVMFormatRaw    = "raw"
+	KVMFirmwareUEFI = "uefi"
+	KVMFirmwareBIOS = "bios"
+)
+
+// IsHTTPURL reports whether s is an absolute http(s) URL with a host, the
+// only kind of image source the kvm provider can fetch.
+func IsHTTPURL(s string) bool {
+	u, err := url.Parse(s)
+	return err == nil && (u.Scheme == "http" || u.Scheme == "https") && u.Host != ""
+}
+
 var sha256Hex = regexp.MustCompile(`^[0-9a-fA-F]{64}$`)
 
+// kvmIdentifier constrains the config names that are pasted into
+// identifiers elsewhere: a host name goes into the providerID
+// hosts/<name>/domains/<vm>, which is split on "/" to parse, and an image
+// key becomes the file name of the cached base volume. A slash in either
+// produces an ID or a path that never resolves again, so the VM leaks with
+// nothing able to address it.
+var kvmIdentifier = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*$`)
+
 func validateKVM(prefix string, c *KVMConfig) error {
 	if c == nil {
 		return fmt.Errorf("%s: kvm block is required", prefix)
@@ -97,18 +128,21 @@ func validateKVM(prefix string, c *KVMConfig) error {
 	}
 	for name, img := range c.Images {
 		p := fmt.Sprintf("%s: kvm.images[%q]", prefix, name)
-		u, err := url.Parse(img.URL)
-		if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+		if !kvmIdentifier.MatchString(name) {
+			return fmt.Errorf("%s: key must be letters, digits, dots, dashes or underscores"+
+				" — it becomes the cached volume name base-<key>.<format>", p)
+		}
+		if !IsHTTPURL(img.URL) {
 			return fmt.Errorf("%s.url must be an http(s) URL (got %q)", p, img.URL)
 		}
 		if img.SHA256 != "" && !sha256Hex.MatchString(img.SHA256) {
 			return fmt.Errorf("%s.sha256 must be 64 hex characters", p)
 		}
-		if img.Format != "" && img.Format != "qcow2" && img.Format != "raw" {
-			return fmt.Errorf("%s.format must be qcow2 or raw (got %q)", p, img.Format)
+		if img.Format != "" && img.Format != KVMFormatQcow2 && img.Format != KVMFormatRaw {
+			return fmt.Errorf("%s.format must be %s or %s (got %q)", p, KVMFormatQcow2, KVMFormatRaw, img.Format)
 		}
-		if img.Firmware != "" && img.Firmware != "uefi" && img.Firmware != "bios" {
-			return fmt.Errorf("%s.firmware must be uefi or bios (got %q)", p, img.Firmware)
+		if img.Firmware != "" && img.Firmware != KVMFirmwareUEFI && img.Firmware != KVMFirmwareBIOS {
+			return fmt.Errorf("%s.firmware must be %s or %s (got %q)", p, KVMFirmwareUEFI, KVMFirmwareBIOS, img.Firmware)
 		}
 	}
 	if len(c.MachineTypes) == 0 {
@@ -128,6 +162,10 @@ func validateKVM(prefix string, c *KVMConfig) error {
 		if h.Name == "" {
 			return fmt.Errorf("%s: name is required", p)
 		}
+		if !kvmIdentifier.MatchString(h.Name) {
+			return fmt.Errorf("%s: name %q must be letters, digits, dots, dashes or underscores"+
+				" — it becomes part of the providerID hosts/<name>/domains/<vm>", p, h.Name)
+		}
 		if seen[h.Name] {
 			return fmt.Errorf("%s: duplicate host name %q", p, h.Name)
 		}

internal/provider/config_kvm_test.go

@@ -129,6 +129,19 @@ hosts:
 			yaml:       minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n  small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n  - {name: nas1, address: a:22, user: virt, bridge: br0}\n  - {name: nas1, address: b:22, user: virt, bridge: br0}\n"),
 			wantErrSub: `duplicate host name "nas1"`,
 		},
+		{
+			// A slash would produce hosts/dc1/rack2/domains/<vm>, which
+			// parseProviderID can never split back, so the VM would be
+			// unreachable for Get, Delete and GC alike.
+			name:       "host name with a slash",
+			yaml:       minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n  small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n  - {name: dc1/rack2, address: a:22, user: virt, bridge: br0}\n"),
+			wantErrSub: `name "dc1/rack2" must be letters, digits, dots, dashes or underscores`,
+		},
+		{
+			name:       "image key with a slash",
+			yaml:       minimal("sshKeyFile: /key\nknownHostsFile: /kh\nimages:\n  ../etc/d: {url: https://x/y.qcow2}\n" + hostsAndTypes),
+			wantErrSub: `images["../etc/d"]: key must be letters, digits, dots, dashes or underscores`,
+		},
 		{
 			name:       "host missing address",
 			yaml:       minimal("sshKeyFile: /key\nknownHostsFile: /kh\nmachineTypes:\n  small: {vcpus: 1, memoryMiB: 1024}\nhosts:\n  - {name: nas1, user: virt, bridge: br0}\n"),

internal/provider/kvm/conn.go

@@ -5,6 +5,7 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"io/fs"
 	"net"
 	"os"
 	"sync"
@@ -135,10 +136,13 @@ func (p *hostPool) drop(host string, c *hostClient) {
 // call runs fn against host's connection. go-libvirt RPCs take no context
 // and block until the daemon answers, so when ctx ends first the connection
 // is closed — which unblocks fn with an error — and ctx.Err() is returned.
-// A transport error (anything that is not a libvirt.Error) also drops the
-// cached connection so the next call redials instead of reusing a dead
-// link. When log has V(1) enabled, every RPC fn makes is traced through it
-// (see tracingAPI), as are connection events.
+// A transport error also drops the cached connection so the next call
+// redials instead of reusing a dead link. "Transport error" means anything
+// that is neither a libvirt.Error nor an error fn already classified with a
+// provider sentinel (a failed seed build, a bad placement): those are local
+// verdicts, not evidence about the link. When log has V(1) enabled, every
+// RPC fn makes is traced through it (see tracingAPI), as are connection
+// events.
 func (p *hostPool) call(ctx context.Context, log logr.Logger, host provider.KVMHost, fn func(api libvirtAPI) error) error {
 	log = log.WithName("libvirt")
 	c, err := p.get(ctx, log, host)
@@ -153,7 +157,7 @@ func (p *hostPool) call(ctx context.Context, log logr.Logger, host provider.KVMH
 		log.V(1).Info("libvirt connection dropped", "reason", "context cancelled", "error", ctx.Err().Error())
 		p.drop(host.Name, c)
 	})
-	if err != nil && !isLibvirtError(err) && !errors.Is(err, ctx.Err()) {
+	if err != nil && !isLibvirtError(err) && !isClassified(err) && !errors.Is(err, ctx.Err()) {
 		log.V(1).Info("libvirt connection dropped", "reason", "transport error", "error", err.Error())
 		p.drop(host.Name, c)
 	}
@@ -177,14 +181,17 @@ func awaitOrCancel(ctx context.Context, fn func() error, abort func()) error {
 }
 
 // sshAuth holds the pieces of the SSH client config shared by every host:
-// the private key and the host-key policy, both loaded once at startup so a
-// missing or unreadable file fails the operator fast rather than on the
-// first Create.
+// the private key and the host-key policy. Both are loaded on the first
+// connection rather than at startup — see lazySSH.
 type sshAuth struct {
 	signer  ssh.Signer
 	hostKey ssh.HostKeyCallback
 }
 
+// newSSHAuth loads the key material. A file that is not there yet stays
+// unclassified, so the provider treats it as transient and heals when the
+// Secret appears; content that will never parse is permanent, since
+// retrying cannot fix it.
 func newSSHAuth(cfg provider.KVMConfig) (*sshAuth, error) {
 	key, err := os.ReadFile(cfg.SSHKeyFile)
 	if err != nil {
@@ -192,34 +199,79 @@ func newSSHAuth(cfg provider.KVMConfig) (*sshAuth, error) {
 	}
 	signer, err := ssh.ParsePrivateKey(key)
 	if err != nil {
-		return nil, fmt.Errorf("parsing kvm.sshKeyFile: %w", err)
+		return nil, fmt.Errorf("%w: parsing kvm.sshKeyFile: %v", provider.ErrPermanent, err)
 	}
 	hostKey := ssh.InsecureIgnoreHostKey() //nolint:gosec // opt-in via config, documented as insecure
 	if !cfg.InsecureIgnoreHostKey {
 		hostKey, err = knownhosts.New(cfg.KnownHostsFile)
-		if err != nil {
+		if errors.Is(err, fs.ErrNotExist) {
 			return nil, fmt.Errorf("loading kvm.knownHostsFile: %w", err)
 		}
+		if err != nil {
+			return nil, fmt.Errorf("%w: loading kvm.knownHostsFile: %v", provider.ErrPermanent, err)
+		}
 	}
 	return &sshAuth{signer: signer, hostKey: hostKey}, nil
 }
 
-// dialer returns the production hostDialer: SSH to the host, forward its
-// libvirt unix socket through the connection, and open a libvirt session
-// over that.
-func (a *sshAuth) dialer() hostDialer {
-	return func(ctx context.Context, host provider.KVMHost) (*hostClient, error) {
-		client, err := a.dialSSH(ctx, host)
-		if err != nil {
-			return nil, err
-		}
-		hc, err := connectLibvirt(ctx, client, host.Socket)
-		if err != nil {
-			client.Close()
-			return nil, err
-		}
-		return hc, nil
+// lazySSH defers loading the SSH material until it is first needed and
+// caches it once it works. The kvm-ssh Secret is an optional mount: the
+// manager has to start without it and keep every other provider running,
+// and a Secret created afterwards has to heal the KVM proxies on the next
+// reconcile rather than needing a restart.
+type lazySSH struct {
+	cfg provider.KVMConfig
+
+	mu   sync.Mutex
+	auth *sshAuth
+}
+
+// newLazySSH keeps only the SSH fields of cfg, since the loader lives as
+// long as the Provider.
+func newLazySSH(cfg provider.KVMConfig) *lazySSH {
+	return &lazySSH{cfg: provider.KVMConfig{
+		SSHKeyFile:            cfg.SSHKeyFile,
+		KnownHostsFile:        cfg.KnownHostsFile,
+		InsecureIgnoreHostKey: cfg.InsecureIgnoreHostKey,
+	}}
+}
+
+func (l *lazySSH) load() (*sshAuth, error) {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	if l.auth != nil {
+		return l.auth, nil
 	}
+	auth, err := newSSHAuth(l.cfg)
+	if err != nil {
+		return nil, err
+	}
+	l.auth = auth
+	return auth, nil
+}
+
+// dial is the production hostDialer.
+func (l *lazySSH) dial(ctx context.Context, host provider.KVMHost) (*hostClient, error) {
+	a, err := l.load()
+	if err != nil {
+		return nil, err
+	}
+	return a.dial(ctx, host)
+}
+
+// dial SSHes to the host, forwards its libvirt unix socket through the
+// connection, and opens a libvirt session over that.
+func (a *sshAuth) dial(ctx context.Context, host provider.KVMHost) (*hostClient, error) {
+	client, err := a.dialSSH(ctx, host)
+	if err != nil {
+		return nil, err
+	}
+	hc, err := connectLibvirt(ctx, client, host.Socket)
+	if err != nil {
+		client.Close()
+		return nil, err
+	}
+	return hc, nil
 }
 
 // dialSSH establishes the SSH connection. The handshake is bounded by ctx's

internal/provider/kvm/conn_test.go

@@ -7,6 +7,7 @@ import (
 	"crypto/rand"
 	"encoding/pem"
 	"errors"
+	"fmt"
 	"io"
 	"net"
 	"os"
@@ -109,41 +110,48 @@ func TestHostPool_call_redialsAfterDisconnect(t *testing.T) {
 	}
 }
 
-func TestHostPool_call_transportErrorDropsConnection(t *testing.T) {
+// Only a transport error is evidence about the link. A daemon-reported
+// libvirt error, or a verdict fn reached locally, must not cost the
+// connection.
+func TestHostPool_call_dropsConnectionOnlyOnTransportError(t *testing.T) {
 	t.Parallel()
-	d := &fakeDialer{}
-	p := newHostPool(d.dial)
-	err := p.call(context.Background(), discard, testHost, func(libvirtAPI) error { return io.ErrUnexpectedEOF })
-	if !errors.Is(err, io.ErrUnexpectedEOF) {
-		t.Fatalf("call() error = %v, want the transport error passed through", err)
+	tests := []struct {
+		name        string
+		err         error
+		wantDropped bool
+	}{
+		{"transport error", io.ErrUnexpectedEOF, true},
+		{"libvirt error", lvErr(libvirt.ErrNoDomain), false},
+		{"locally classified error", fmt.Errorf("%w: creating iso writer: read-only file system", provider.ErrPermanent), false},
 	}
-	if !d.clients[0].closed.Load() {
-		t.Error("connection not closed after a transport error")
-	}
-	if err := p.call(context.Background(), discard, testHost, noop); err != nil {
-		t.Fatalf("call() after transport error = %v", err)
-	}
-	if d.dialCount() != 2 {
-		t.Errorf("dials = %d, want 2", d.dialCount())
-	}
-}
-
-func TestHostPool_call_libvirtErrorKeepsConnection(t *testing.T) {
-	t.Parallel()
-	d := &fakeDialer{}
-	p := newHostPool(d.dial)
-	err := p.call(context.Background(), discard, testHost, func(libvirtAPI) error { return lvErr(libvirt.ErrNoDomain) })
-	if !hasCode(err, libvirt.ErrNoDomain) {
-		t.Fatalf("call() error = %v, want the libvirt error passed through", err)
-	}
-	if err := p.call(context.Background(), discard, testHost, noop); err != nil {
-		t.Fatalf("call() error = %v", err)
-	}
-	if d.dialCount() != 1 {
-		t.Errorf("dials = %d, want 1: a daemon-reported error must not cost the connection", d.dialCount())
-	}
-	if d.clients[0].closed.Load() {
-		t.Error("connection closed after a libvirt error")
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			d := &fakeDialer{}
+			p := newHostPool(d.dial)
+			log, snapshot := captureLogger(1)
+			err := p.call(context.Background(), log, testHost, func(libvirtAPI) error { return tc.err })
+			if !errors.Is(err, tc.err) {
+				t.Fatalf("call() error = %v, want %v passed through", err, tc.err)
+			}
+			if got := d.clients[0].closed.Load(); got != tc.wantDropped {
+				t.Errorf("connection closed = %v, want %v", got, tc.wantDropped)
+			}
+			joined := strings.Join(snapshot(), "\n")
+			if logged := strings.Contains(joined, `"msg"="libvirt connection dropped"`) && strings.Contains(joined, `"reason"="transport error"`); logged != tc.wantDropped {
+				t.Errorf("transport drop logged = %v, want %v:\n%s", logged, tc.wantDropped, joined)
+			}
+			if err := p.call(context.Background(), discard, testHost, noop); err != nil {
+				t.Fatalf("next call() error = %v", err)
+			}
+			wantDials := 1
+			if tc.wantDropped {
+				wantDials = 2
+			}
+			if d.dialCount() != wantDials {
+				t.Errorf("dials = %d, want %d", d.dialCount(), wantDials)
+			}
+		})
 	}
 }
 
@@ -387,13 +395,19 @@ func newSSHFixture(t *testing.T, answerGlobalRequests bool) *sshFixture {
 	}
 }
 
-func TestSSHAuth_dialSSH_forwardsUnixSocket(t *testing.T) {
-	t.Parallel()
-	f := newSSHFixture(t, true)
-	auth, err := newSSHAuth(f.cfg)
+func mustSSHAuth(t *testing.T, cfg provider.KVMConfig) *sshAuth {
+	t.Helper()
+	auth, err := newSSHAuth(cfg)
 	if err != nil {
 		t.Fatalf("newSSHAuth() error = %v", err)
 	}
+	return auth
+}
+
+func TestSSHAuth_dialSSH_forwardsUnixSocket(t *testing.T) {
+	t.Parallel()
+	f := newSSHFixture(t, true)
+	auth := mustSSHAuth(t, f.cfg)
 	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
 	defer cancel()
 	client, err := auth.dialSSH(ctx, f.host)
@@ -427,10 +441,7 @@ func TestSSHAuth_dialSSH_rejectsUnknownHostKey(t *testing.T) {
 	if err := os.WriteFile(f.cfg.KnownHostsFile, []byte(line+"\n"), 0o600); err != nil {
 		t.Fatal(err)
 	}
-	auth, err := newSSHAuth(f.cfg)
-	if err != nil {
-		t.Fatalf("newSSHAuth() error = %v", err)
-	}
+	auth := mustSSHAuth(t, f.cfg)
 	client, err := auth.dialSSH(context.Background(), f.host)
 	if err == nil {
 		client.Close()
@@ -447,11 +458,7 @@ func TestSSHAuth_dialSSH_insecureIgnoreHostKey(t *testing.T) {
 	f := newSSHFixture(t, true)
 	f.cfg.KnownHostsFile = ""
 	f.cfg.InsecureIgnoreHostKey = true
-	auth, err := newSSHAuth(f.cfg)
-	if err != nil {
-		t.Fatalf("newSSHAuth() error = %v", err)
-	}
-	client, err := auth.dialSSH(context.Background(), f.host)
+	client, err := mustSSHAuth(t, f.cfg).dialSSH(context.Background(), f.host)
 	if err != nil {
 		t.Fatalf("dialSSH() error = %v", err)
 	}
@@ -465,11 +472,7 @@ func TestSSHAuth_dialSSH_rejectsWrongClientKey(t *testing.T) {
 	if err := os.WriteFile(f.keyFile, otherPEM, 0o600); err != nil {
 		t.Fatal(err)
 	}
-	auth, err := newSSHAuth(f.cfg)
-	if err != nil {
-		t.Fatalf("newSSHAuth() error = %v", err)
-	}
-	client, err := auth.dialSSH(context.Background(), f.host)
+	client, err := mustSSHAuth(t, f.cfg).dialSSH(context.Background(), f.host)
 	if err == nil {
 		client.Close()
 		t.Fatal("dialSSH() succeeded with a key the server does not know")
@@ -482,13 +485,9 @@ func TestSSHAuth_dialSSH_rejectsWrongClientKey(t *testing.T) {
 func TestSSHAuth_dialSSH_cancelledContext(t *testing.T) {
 	t.Parallel()
 	f := newSSHFixture(t, true)
-	auth, err := newSSHAuth(f.cfg)
-	if err != nil {
-		t.Fatalf("newSSHAuth() error = %v", err)
-	}
 	ctx, cancel := context.WithCancel(context.Background())
 	cancel()
-	if client, err := auth.dialSSH(ctx, f.host); err == nil {
+	if client, err := mustSSHAuth(t, f.cfg).dialSSH(ctx, f.host); err == nil {
 		client.Close()
 		t.Fatal("dialSSH() succeeded with an already-cancelled context")
 	}
@@ -524,14 +523,101 @@ func TestNewSSHAuth_missingFiles(t *testing.T) {
 	}
 }
 
+// The kvm-ssh Secret is an optional mount, so the provider has to be
+// constructible without it and has to start working once it appears — the
+// alternative was a constructor error that crash-looped the whole manager.
+func TestLazySSH_load_healsWhenTheKeyAppears(t *testing.T) {
+	t.Parallel()
+	f := newSSHFixture(t, true)
+	keyPEM, err := os.ReadFile(f.keyFile)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := os.Remove(f.keyFile); err != nil {
+		t.Fatal(err)
+	}
+	auth := newLazySSH(f.cfg).load
+
+	if _, err := auth(); err == nil {
+		t.Fatal("loading with no key file succeeded, want an error")
+	} else if class := classify(err); class != provider.ErrTransient {
+		t.Errorf("class = %v, want ErrTransient so the proxy retries once the Secret lands", class)
+	}
+	if err := os.WriteFile(f.keyFile, keyPEM, 0o600); err != nil {
+		t.Fatal(err)
+	}
+	loaded, err := auth()
+	if err != nil {
+		t.Fatalf("loading after the key appeared: %v", err)
+	}
+	// Cached from here on: the file may vanish again (a Secret rotated out)
+	// without breaking connections that already work.
+	if err := os.Remove(f.keyFile); err != nil {
+		t.Fatal(err)
+	}
+	if again, err := auth(); err != nil || again != loaded {
+		t.Errorf("auth() = %v, %v; want the cached value reused", again, err)
+	}
+}
+
+func TestLazySSH_load_unparseableKeyIsPermanent(t *testing.T) {
+	t.Parallel()
+	f := newSSHFixture(t, true)
+	if err := os.WriteFile(f.keyFile, []byte("not a key"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	_, err := newLazySSH(f.cfg).load()
+	if classify(err) != provider.ErrPermanent {
+		t.Errorf("class = %v, want ErrPermanent: retrying cannot fix a key that does not parse", classify(err))
+	}
+}
+
+// The dialer must authenticate with the material the lazy loader produced,
+// not a stale or empty one. The echo socket is not a libvirt server, so the
+// call gets as far as the libvirt handshake and no further; what matters is
+// that it is past SSH by then.
+func TestLazySSH_dial_authenticatesOverSSH(t *testing.T) {
+	t.Parallel()
+	f := newSSHFixture(t, true)
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+
+	_, err := newLazySSH(f.cfg).dial(ctx, f.host)
+	if err == nil {
+		t.Fatal("dial to an echo socket succeeded, want a libvirt handshake error")
+	}
+	if strings.Contains(err.Error(), "sshKeyFile") || strings.Contains(err.Error(), "ssh handshake") {
+		t.Errorf("dial failed before libvirt: %v", err)
+	}
+	if paths := f.server.requestedPaths(); len(paths) != 1 || paths[0] != f.host.Socket {
+		t.Errorf("server saw forward requests %v, want [%s]", paths, f.host.Socket)
+	}
+}
+
+// New must not fail on missing key material, or one misconfigured provider
+// aborts registry.Build and the manager exits before any controller starts.
+func TestNew_missingSSHMaterialStillBuilds(t *testing.T) {
+	t.Parallel()
+	p, err := New(context.Background(), provider.ProviderConfig{
+		Name: "kvm-home", Type: "kvm",
+		KVM: &provider.KVMConfig{
+			SSHKeyFile:     filepath.Join(t.TempDir(), "absent"),
+			KnownHostsFile: filepath.Join(t.TempDir(), "absent"),
+			Hosts:          []provider.KVMHost{{Name: "nas1", Address: "nas1:22", User: "virt", Bridge: "br0"}},
+		},
+	})
+	if err != nil || p == nil {
+		t.Fatalf("New() = %v, %v; want a usable provider", p, err)
+	}
+	if _, err := p.Get(context.Background(), "hosts/nas1/domains/proxy-x"); provider.Class(err) != provider.ErrTransient {
+		t.Errorf("Get class = %v, want ErrTransient", provider.Class(err))
+	}
+}
+
 func TestKeepalive_closesClientWhenServerStalls(t *testing.T) {
 	t.Parallel()
 	f := newSSHFixture(t, false)
-	auth, err := newSSHAuth(f.cfg)
-	if err != nil {
-		t.Fatalf("newSSHAuth() error = %v", err)
-	}
-	client, err := auth.dialSSH(context.Background(), f.host)
+	client, err := mustSSHAuth(t, f.cfg).dialSSH(context.Background(), f.host)
 	if err != nil {
 		t.Fatalf("dialSSH() error = %v", err)
 	}
@@ -552,11 +638,7 @@ func TestKeepalive_closesClientWhenServerStalls(t *testing.T) {
 func TestKeepalive_keepsHealthyClientOpen(t *testing.T) {
 	t.Parallel()
 	f := newSSHFixture(t, true)
-	auth, err := newSSHAuth(f.cfg)
-	if err != nil {
-		t.Fatalf("newSSHAuth() error = %v", err)
-	}
-	client, err := auth.dialSSH(context.Background(), f.host)
+	client, err := mustSSHAuth(t, f.cfg).dialSSH(context.Background(), f.host)
 	if err != nil {
 		t.Fatalf("dialSSH() error = %v", err)
 	}
@@ -613,42 +695,28 @@ func TestHostPool_call_noTracingBelowV1(t *testing.T) {
 	}
 }
 
-func TestHostPool_call_logsDropReason(t *testing.T) {
+func TestHostPool_call_logsCancelDropReason(t *testing.T) {
 	t.Parallel()
-	t.Run("transport error", func(t *testing.T) {
-		t.Parallel()
-		d := &fakeDialer{}
-		p := newHostPool(d.dial)
-		log, snapshot := captureLogger(1)
-		_ = p.call(context.Background(), log, testHost, func(libvirtAPI) error { return io.ErrUnexpectedEOF })
-		joined := strings.Join(snapshot(), "\n")
-		if !strings.Contains(joined, `"msg"="libvirt connection dropped"`) || !strings.Contains(joined, `"reason"="transport error"`) {
-			t.Errorf("output missing the drop line:\n%s", joined)
-		}
-	})
-	t.Run("context cancelled", func(t *testing.T) {
-		t.Parallel()
-		d := &fakeDialer{}
-		p := newHostPool(d.dial)
-		log, snapshot := captureLogger(1)
-		ctx, cancel := context.WithCancel(context.Background())
-		cancel()
-		released := make(chan struct{})
-		err := p.call(ctx, log, testHost, func(libvirtAPI) error {
-			<-d.clients[0].done
-			close(released)
-			return io.EOF
-		})
-		if !errors.Is(err, context.Canceled) {
-			t.Fatalf("call() error = %v, want context.Canceled", err)
-		}
-		<-released
-		joined := strings.Join(snapshot(), "\n")
-		if !strings.Contains(joined, `"reason"="context cancelled"`) {
-			t.Errorf("output missing the cancel drop line:\n%s", joined)
-		}
-		if strings.Contains(joined, `"reason"="transport error"`) {
-			t.Errorf("cancel was also reported as a transport error:\n%s", joined)
-		}
+	d := &fakeDialer{}
+	p := newHostPool(d.dial)
+	log, snapshot := captureLogger(1)
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	released := make(chan struct{})
+	err := p.call(ctx, log, testHost, func(libvirtAPI) error {
+		<-d.clients[0].done
+		close(released)
+		return io.EOF
 	})
+	if !errors.Is(err, context.Canceled) {
+		t.Fatalf("call() error = %v, want context.Canceled", err)
+	}
+	<-released
+	joined := strings.Join(snapshot(), "\n")
+	if !strings.Contains(joined, `"reason"="context cancelled"`) {
+		t.Errorf("output missing the cancel drop line:\n%s", joined)
+	}
+	if strings.Contains(joined, `"reason"="transport error"`) {
+		t.Errorf("cancel was also reported as a transport error:\n%s", joined)
+	}
 }

internal/provider/kvm/errors.go

@@ -19,10 +19,8 @@ import (
 // function to recognise. libvirt exposes no quota signal, so nothing maps
 // to QuotaExceeded.
 func classify(err error) error {
-	for _, class := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} {
-		if errors.Is(err, class) {
-			return class
-		}
+	if isClassified(err) {
+		return provider.Class(err)
 	}
 	var lerr libvirt.Error
 	if !errors.As(err, &lerr) {
@@ -51,6 +49,18 @@ func isLibvirtError(err error) bool {
 	return errors.As(err, &lerr)
 }
 
+// isClassified reports whether err already carries one of the provider's
+// sentinels, i.e. the package reached a verdict itself rather than hitting
+// a transport failure.
+func isClassified(err error) bool {
+	for _, class := range []error{provider.ErrNotFound, provider.ErrQuotaExceeded, provider.ErrPermanent, provider.ErrTransient} {
+		if errors.Is(err, class) {
+			return true
+		}
+	}
+	return false
+}
+
 func hasCode(err error, code libvirt.ErrorNumber) bool {
 	var lerr libvirt.Error
 	return errors.As(err, &lerr) && libvirt.ErrorNumber(lerr.Code) == code

internal/provider/kvm/fake_test.go

@@ -31,7 +31,6 @@ type fakeAPI struct {
 }
 
 type fakeVol struct {
-	name     string
 	format   string
 	capacity int64
 	backing  string
@@ -94,7 +93,7 @@ func (f *fakeAPI) vol(pool, name string) *fakeVol {
 func (f *fakeAPI) putVol(pool, name, format string, data []byte) {
 	f.mu.Lock()
 	defer f.mu.Unlock()
-	f.pools[pool][name] = &fakeVol{name: name, format: format, capacity: int64(len(data)), data: data}
+	f.pools[pool][name] = &fakeVol{format: format, capacity: int64(len(data)), data: data}
 }
 
 func (f *fakeAPI) volNames(pool string) []string {
@@ -348,7 +347,7 @@ func (f *fakeAPI) StorageVolCreateXML(pool libvirt.StoragePool, raw string, _ li
 	if _, exists := vols[v.Name]; exists {
 		return libvirt.StorageVol{}, lvError(libvirt.ErrStorageVolExist, "storage volume '%s' exists already", v.Name)
 	}
-	fv := &fakeVol{name: v.Name, format: v.Target.Format.Type, capacity: v.Capacity.Value}
+	fv := &fakeVol{format: v.Target.Format.Type, capacity: v.Capacity.Value}
 	if v.BackingStore != nil {
 		fv.backing = v.BackingStore.Path
 	}

internal/provider/kvm/image.go

@@ -8,7 +8,6 @@ import (
 	"fmt"
 	"io"
 	"net/http"
-	"net/url"
 	"strings"
 	"sync"
 	"time"
@@ -22,11 +21,8 @@ import (
 // imageSource is a resolved placement.image: either an entry from the
 // provider's images map (Key set) or a URL given directly.
 type imageSource struct {
-	Key      string
-	URL      string
-	SHA256   string
-	Format   string
-	Firmware string
+	Key string
+	provider.KVMImage
 }
 
 // resolveImage maps placement.image to a source. Anything that is neither a
@@ -34,11 +30,10 @@ type imageSource struct {
 // surface, not retry.
 func resolveImage(cfg provider.KVMConfig, image string) (imageSource, error) {
 	if img, ok := cfg.Images[image]; ok {
-		return imageSource{Key: image, URL: img.URL, SHA256: img.SHA256, Format: img.Format, Firmware: img.Firmware}, nil
+		return imageSource{Key: image, KVMImage: img}, nil
 	}
-	u, err := url.Parse(image)
-	if err == nil && (u.Scheme == "http" || u.Scheme == "https") && u.Host != "" {
-		return imageSource{URL: image, Format: formatQcow2, Firmware: firmwareUEFI}, nil
+	if provider.IsHTTPURL(image) {
+		return imageSource{KVMImage: imageDefaults(provider.KVMImage{URL: image})}, nil
 	}
 	return imageSource{}, fmt.Errorf("%w: placement.image %q is neither a key in kvm.images nor an http(s) URL",
 		provider.ErrPermanent, image)

internal/provider/kvm/image_test.go

@@ -37,12 +37,11 @@ type imageServer struct {
 	status   int
 	noLength bool
 	delay    time.Duration
-	body     []byte
 }
 
 func newImageServer(t *testing.T) *imageServer {
 	t.Helper()
-	s := &imageServer{status: http.StatusOK, body: testImage}
+	s := &imageServer{status: http.StatusOK}
 	s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
 		s.hits.Add(1)
 		time.Sleep(s.delay)
@@ -55,16 +54,16 @@ func newImageServer(t *testing.T) *imageServer {
 		if s.noLength {
 			w.(http.Flusher).Flush()
 		} else {
-			w.Header().Set("Content-Length", strconv.Itoa(len(s.body)))
+			w.Header().Set("Content-Length", strconv.Itoa(len(testImage)))
 		}
-		_, _ = w.Write(s.body)
+		_, _ = w.Write(testImage)
 	}))
 	t.Cleanup(s.Close)
 	return s
 }
 
 func testSource(srv *imageServer, sha string) imageSource {
-	return imageSource{Key: "debian-13", URL: srv.URL + "/debian.qcow2", SHA256: sha, Format: formatQcow2}
+	return imageSource{Key: "debian-13", KVMImage: provider.KVMImage{URL: srv.URL + "/debian.qcow2", SHA256: sha, Format: formatQcow2}}
 }
 
 func ensureTest(t *testing.T, api *fakeAPI, src imageSource) (string, error) {
@@ -88,9 +87,9 @@ func TestResolveImage(t *testing.T) {
 		want    imageSource
 		wantErr bool
 	}{
-		{"configured key", "debian-13", imageSource{Key: "debian-13", URL: "https://cloud.debian.org/d.qcow2", SHA256: "abc", Format: "qcow2", Firmware: "uefi"}, false},
-		{"direct https url", "https://x.example/img.qcow2", imageSource{URL: "https://x.example/img.qcow2", Format: "qcow2", Firmware: "uefi"}, false},
-		{"direct http url", "http://mirror/img.qcow2", imageSource{URL: "http://mirror/img.qcow2", Format: "qcow2", Firmware: "uefi"}, false},
+		{"configured key", "debian-13", imageSource{Key: "debian-13", KVMImage: provider.KVMImage{URL: "https://cloud.debian.org/d.qcow2", SHA256: "abc", Format: "qcow2", Firmware: "uefi"}}, false},
+		{"direct https url", "https://x.example/img.qcow2", imageSource{KVMImage: provider.KVMImage{URL: "https://x.example/img.qcow2", Format: "qcow2", Firmware: "uefi"}}, false},
+		{"direct http url", "http://mirror/img.qcow2", imageSource{KVMImage: provider.KVMImage{URL: "http://mirror/img.qcow2", Format: "qcow2", Firmware: "uefi"}}, false},
 		{"unknown key", "ubuntu", imageSource{}, true},
 		{"file url", "file:///tmp/x.qcow2", imageSource{}, true},
 		{"empty", "", imageSource{}, true},
@@ -100,9 +99,7 @@ func TestResolveImage(t *testing.T) {
 			t.Parallel()
 			got, err := resolveImage(cfg, tc.image)
 			if tc.wantErr {
-				if !errors.Is(err, provider.ErrPermanent) {
-					t.Fatalf("resolveImage(%q) error = %v, want ErrPermanent", tc.image, err)
-				}
+				wantClass(t, err, provider.ErrPermanent)
 				return
 			}
 			if err != nil {
@@ -117,16 +114,16 @@ func TestResolveImage(t *testing.T) {
 
 func TestImageSource_volumeNames(t *testing.T) {
 	t.Parallel()
-	keyed := imageSource{Key: "debian-13", URL: "https://a/b.qcow2", Format: "qcow2"}
+	keyed := imageSource{Key: "debian-13", KVMImage: provider.KVMImage{URL: "https://a/b.qcow2", Format: "qcow2"}}
 	if keyed.volumeName() != "base-debian-13.qcow2" || keyed.markerName() != "base-debian-13.qcow2.sha256" {
 		t.Errorf("keyed names = %s / %s", keyed.volumeName(), keyed.markerName())
 	}
-	direct := imageSource{URL: "https://a/b.qcow2", Format: "qcow2"}
+	direct := imageSource{KVMImage: provider.KVMImage{URL: "https://a/b.qcow2", Format: "qcow2"}}
 	name := direct.volumeName()
 	if !strings.HasPrefix(name, "base-") || !strings.HasSuffix(name, ".qcow2") || len(name) != len("base-")+16+len(".qcow2") {
 		t.Errorf("direct-url name = %q, want base-<16 hex>.qcow2", name)
 	}
-	other := imageSource{URL: "https://a/c.qcow2", Format: "qcow2"}
+	other := imageSource{KVMImage: provider.KVMImage{URL: "https://a/c.qcow2", Format: "qcow2"}}
 	if other.volumeName() == name {
 		t.Error("different URLs produced the same volume name")
 	}
@@ -181,57 +178,47 @@ func TestImageCache_ensure_noChecksumConfigured(t *testing.T) {
 	}
 }
 
-func TestImageCache_ensure_alreadyCached(t *testing.T) {
+func TestImageCache_ensure_existingVolumes(t *testing.T) {
 	t.Parallel()
-	srv := newImageServer(t)
-	api := newFakeAPI("default")
-	api.putVol("default", "base-debian-13.qcow2", "qcow2", testImage)
-	api.putVol("default", "base-debian-13.qcow2.sha256", "raw", []byte(testImageSHA()))
-	path, err := ensureTest(t, api, testSource(srv, testImageSHA()))
-	if err != nil {
-		t.Fatalf("ensure() error = %v", err)
+	tests := []struct {
+		name     string
+		base     []byte
+		marker   []byte
+		wantHits int32
+	}{
+		{"already cached", testImage, []byte(testImageSHA()), 0},
+		{"marker missing is a partial upload", []byte("half an image"), nil, 1},
+		{"configured checksum changed", []byte("an older release"), []byte(strings.Repeat("0", 64)), 1},
 	}
-	if path != "/var/lib/libvirt/images/base-debian-13.qcow2" {
-		t.Errorf("path = %q", path)
-	}
-	if srv.hits.Load() != 0 {
-		t.Errorf("http hits = %d, want 0 for a cached image", srv.hits.Load())
-	}
-	if api.hasCall("StorageVolCreateXML") || api.hasCall("StorageVolDelete") {
-		t.Errorf("cached image was touched: calls = %v", api.calls)
-	}
-}
-
-func TestImageCache_ensure_markerMissingRedownloads(t *testing.T) {
-	t.Parallel()
-	srv := newImageServer(t)
-	api := newFakeAPI("default")
-	api.putVol("default", "base-debian-13.qcow2", "qcow2", []byte("half an image"))
-	if _, err := ensureTest(t, api, testSource(srv, testImageSHA())); err != nil {
-		t.Fatalf("ensure() error = %v", err)
-	}
-	if srv.hits.Load() != 1 {
-		t.Errorf("http hits = %d, want 1: a base without its marker is a partial upload", srv.hits.Load())
-	}
-	if base := api.vol("default", "base-debian-13.qcow2"); base == nil || !bytes.Equal(base.data, testImage) {
-		t.Error("partial base volume was not replaced")
-	}
-}
-
-func TestImageCache_ensure_checksumChangedRedownloads(t *testing.T) {
-	t.Parallel()
-	srv := newImageServer(t)
-	api := newFakeAPI("default")
-	api.putVol("default", "base-debian-13.qcow2", "qcow2", []byte("an older release"))
-	api.putVol("default", "base-debian-13.qcow2.sha256", "raw", []byte(strings.Repeat("0", 64)))
-	if _, err := ensureTest(t, api, testSource(srv, testImageSHA())); err != nil {
-		t.Fatalf("ensure() error = %v", err)
-	}
-	if srv.hits.Load() != 1 {
-		t.Errorf("http hits = %d, want 1 after the configured checksum changed", srv.hits.Load())
-	}
-	if marker := api.vol("default", "base-debian-13.qcow2.sha256"); marker == nil || string(marker.data) != testImageSHA() {
-		t.Errorf("marker not updated: %+v", marker)
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			srv := newImageServer(t)
+			api := newFakeAPI("default")
+			api.putVol("default", "base-debian-13.qcow2", "qcow2", tc.base)
+			if tc.marker != nil {
+				api.putVol("default", "base-debian-13.qcow2.sha256", "raw", tc.marker)
+			}
+			path, err := ensureTest(t, api, testSource(srv, testImageSHA()))
+			if err != nil {
+				t.Fatalf("ensure() error = %v", err)
+			}
+			if path != "/var/lib/libvirt/images/base-debian-13.qcow2" {
+				t.Errorf("path = %q", path)
+			}
+			if srv.hits.Load() != tc.wantHits {
+				t.Errorf("http hits = %d, want %d", srv.hits.Load(), tc.wantHits)
+			}
+			if touched := api.hasCall("StorageVolCreateXML") || api.hasCall("StorageVolDelete"); touched != (tc.wantHits > 0) {
+				t.Errorf("volumes touched = %v, want %v: calls = %v", touched, tc.wantHits > 0, api.calls)
+			}
+			if base := api.vol("default", "base-debian-13.qcow2"); base == nil || !bytes.Equal(base.data, testImage) {
+				t.Error("base volume does not hold the verified image")
+			}
+			if marker := api.vol("default", "base-debian-13.qcow2.sha256"); marker == nil || string(marker.data) != testImageSHA() {
+				t.Errorf("marker = %+v, want the verified digest", marker)
+			}
+		})
 	}
 }
 
@@ -240,9 +227,7 @@ func TestImageCache_ensure_checksumMismatchIsPermanent(t *testing.T) {
 	srv := newImageServer(t)
 	api := newFakeAPI("default")
 	_, err := ensureTest(t, api, testSource(srv, strings.Repeat("a", 64)))
-	if !errors.Is(err, provider.ErrPermanent) {
-		t.Fatalf("ensure() error = %v, want ErrPermanent", err)
-	}
+	wantClass(t, err, provider.ErrPermanent)
 	if names := api.volNames("default"); len(names) != 0 {
 		t.Errorf("volumes left behind after a checksum mismatch: %v", names)
 	}
@@ -283,8 +268,9 @@ func TestImageCache_ensure_noContentLengthIsPermanent(t *testing.T) {
 	srv.noLength = true
 	api := newFakeAPI("default")
 	_, err := ensureTest(t, api, testSource(srv, ""))
-	if !errors.Is(err, provider.ErrPermanent) || !strings.Contains(err.Error(), "Content-Length") {
-		t.Fatalf("ensure() error = %v, want ErrPermanent about Content-Length", err)
+	wantClass(t, err, provider.ErrPermanent)
+	if !strings.Contains(err.Error(), "Content-Length") {
+		t.Errorf("ensure() error = %v, want one about Content-Length", err)
 	}
 }
 

internal/provider/kvm/kvm.go

@@ -26,10 +26,10 @@ const (
 	defaultDiskSizeGB = 10
 	defaultSocket     = "/var/run/libvirt/libvirt-sock"
 	defaultPool       = "default"
-	formatQcow2       = "qcow2"
-	formatRaw         = "raw"
-	firmwareUEFI      = "uefi"
-	firmwareBIOS      = "bios"
+	formatQcow2       = provider.KVMFormatQcow2
+	formatRaw         = provider.KVMFormatRaw
+	firmwareUEFI      = provider.KVMFirmwareUEFI
+	firmwareBIOS      = provider.KVMFirmwareBIOS
 
 	// undefineFlags removes everything libvirt may keep alongside a domain
 	// definition, so an undefine never fails on leftover state.
@@ -42,18 +42,9 @@ func withDefaults(cfg provider.KVMConfig) provider.KVMConfig {
 	if cfg.DiskSizeGB == 0 {
 		cfg.DiskSizeGB = defaultDiskSizeGB
 	}
-	if cfg.VendorData == "" {
-		cfg.VendorData = defaultVendorData
-	}
 	images := make(map[string]provider.KVMImage, len(cfg.Images))
 	for k, img := range cfg.Images {
-		if img.Format == "" {
-			img.Format = formatQcow2
-		}
-		if img.Firmware == "" {
-			img.Firmware = firmwareUEFI
-		}
-		images[k] = img
+		images[k] = imageDefaults(img)
 	}
 	cfg.Images = images
 	hosts := make([]provider.KVMHost, len(cfg.Hosts))
@@ -79,30 +70,41 @@ type Provider struct {
 	now    func() time.Time
 }
 
+func imageDefaults(img provider.KVMImage) provider.KVMImage {
+	if img.Format == "" {
+		img.Format = formatQcow2
+	}
+	if img.Firmware == "" {
+		img.Firmware = firmwareUEFI
+	}
+	return img
+}
+
 // New builds a Provider that reaches its hosts over SSH. The key and
-// known_hosts files are read here so a bad path fails at startup rather
-// than on the first Create. Satisfies registry.Constructor.
-func New(_ context.Context, pc provider.ProviderConfig) (provider.Provider, error) {
+// known_hosts files are tried once here and kept when usable, but a failure
+// is only a warning, retried on the first connection: they come from an
+// optional Secret mount, and a constructor that failed without it would
+// abort the whole provider set and crash-loop the manager, taking the
+// kubernetes and gcp providers down with it. Satisfies registry.Constructor.
+func New(ctx context.Context, pc provider.ProviderConfig) (provider.Provider, error) {
 	cfg := provider.KVMConfig{}
 	if pc.KVM != nil {
 		cfg = *pc.KVM
 	}
-	auth, err := newSSHAuth(cfg)
-	if err != nil {
-		return nil, fmt.Errorf("kvm provider %q: %w", pc.Name, err)
+	loader := newLazySSH(cfg)
+	if _, err := loader.load(); err != nil {
+		logf.FromContext(ctx).WithName("kvm").WithValues("provider", pc.Name).Info(
+			"WARNING: kvm ssh material is not usable; proxies on this provider will fail until it is",
+			"error", err.Error())
 	}
-	return newWithDialer(pc, auth.dialer(), nil), nil
+	return newWithDialer(pc.Name, cfg, loader.dial, nil), nil
 }
 
 // newWithDialer is the test seam: the dialer decides what libvirtAPI each
 // host resolves to, and the HTTP client is what image downloads go through.
-func newWithDialer(pc provider.ProviderConfig, dial hostDialer, httpClient *http.Client) *Provider {
-	cfg := provider.KVMConfig{}
-	if pc.KVM != nil {
-		cfg = *pc.KVM
-	}
+func newWithDialer(name string, cfg provider.KVMConfig, dial hostDialer, httpClient *http.Client) *Provider {
 	return &Provider{
-		name:   pc.Name,
+		name:   name,
 		cfg:    withDefaults(cfg),
 		hosts:  newHostPool(dial),
 		images: newImageCache(httpClient),
@@ -249,7 +251,11 @@ func (p *Provider) writeSeed(log logr.Logger, api libvirtAPI, pool libvirt.Stora
 	if err := deleteIfExists(api, pool, seedName); err != nil {
 		return libvirt.StorageVol{}, err
 	}
-	iso, err := buildSeedISO(name, userData, p.cfg.VendorData)
+	vendorData, err := buildVendorData(p.cfg.VendorData, p.cfg.InstallGuestAgent == nil || *p.cfg.InstallGuestAgent)
+	if err != nil {
+		return libvirt.StorageVol{}, fmt.Errorf("%w: %v", provider.ErrPermanent, err)
+	}
+	iso, err := buildSeedISO(name, userData, vendorData)
 	if err != nil {
 		return libvirt.StorageVol{}, fmt.Errorf("%w: %v", provider.ErrPermanent, err)
 	}
@@ -450,7 +456,8 @@ func mapState(s libvirt.DomainState, ip string) provider.InstanceState {
 // domain is not destroyed again, a missing domain is not undefined, and
 // the per-VM volumes are removed by name even when the domain is already
 // gone, so a crash between volume creation and define is cleaned up on the
-// next reconcile.
+// next reconcile. A providerID naming a host that is no longer configured
+// is a no-op success rather than an error.
 func (p *Provider) Delete(ctx context.Context, providerID string) error {
 	hostName, name, err := parseProviderID(providerID)
 	if err != nil {
@@ -458,7 +465,14 @@ func (p *Provider) Delete(ctx context.Context, providerID string) error {
 	}
 	host, err := p.host(hostName)
 	if err != nil {
-		return p.wrapErr("delete", providerID, err)
+		// The host is no longer configured, so this operator has no way to
+		// reach the domain and nothing left to clean. Same reasoning as
+		// removeVolumes over a missing pool: refusing would pin the Proxy's
+		// finalizer. Whatever is still running there is the admin's to remove,
+		// and orphan GC cannot see it either — see docs/kvm-provider.md.
+		p.logger(ctx).Info("host is not configured; delete has nothing to clean up",
+			"host", hostName, "providerID", providerID)
+		return nil
 	}
 	log := p.logger(ctx).WithValues("host", host.Name, "name", name)
 	err = p.hosts.call(ctx, log, host, func(api libvirtAPI) error {

internal/provider/kvm/kvm_test.go

@@ -30,8 +30,8 @@ func TestWithDefaults(t *testing.T) {
 	if cfg.DiskSizeGB != defaultDiskSizeGB {
 		t.Errorf("diskSizeGb = %d, want %d", cfg.DiskSizeGB, defaultDiskSizeGB)
 	}
-	if cfg.VendorData != defaultVendorData {
-		t.Errorf("vendorData = %q, want the default", cfg.VendorData)
+	if cfg.VendorData != "" {
+		t.Errorf("vendorData = %q, want none: the guest-agent bootstrap is merged into the user-data, not defaulted here", cfg.VendorData)
 	}
 	if cfg.Images["debian"].Format != "qcow2" || cfg.Images["raw"].Format != "raw" {
 		t.Errorf("image formats = %q/%q, want qcow2 default and raw kept", cfg.Images["debian"].Format, cfg.Images["raw"].Format)
@@ -95,7 +95,8 @@ func testConfig(hosts ...string) provider.ProviderConfig {
 }
 
 func newTestProvider(fakes hostFakes, hosts ...string) *Provider {
-	p := newWithDialer(testConfig(hosts...), fakes.dial, nil)
+	pc := testConfig(hosts...)
+	p := newWithDialer(pc.Name, *pc.KVM, fakes.dial, nil)
 	p.now = func() time.Time { return testNow }
 	return p
 }
@@ -103,7 +104,7 @@ func newTestProvider(fakes hostFakes, hosts ...string) *Provider {
 // cacheImage pre-populates the pool with the configured image and its
 // marker so Create never touches the network.
 func cacheImage(api *fakeAPI) {
-	src := imageSource{Key: "debian-13", URL: testImageURL, Format: formatQcow2}
+	src := imageSource{Key: "debian-13", KVMImage: provider.KVMImage{URL: testImageURL, Format: formatQcow2}}
 	api.putVol("default", src.volumeName(), formatQcow2, []byte("qcow2"))
 	api.putVol("default", src.markerName(), formatRaw, []byte(testImageSHA()))
 }
@@ -187,6 +188,13 @@ func assertCreated(t *testing.T, api *fakeAPI) {
 	if seed.format != formatRaw || len(seed.data) == 0 || int64(len(seed.data)) != seed.capacity {
 		t.Errorf("seed = format %s, %d bytes, capacity %d; want raw with capacity == uploaded size", seed.format, len(seed.data), seed.capacity)
 	}
+	_, files := readSeed(t, seed.data)
+	if files["user-data"] != testCloudInit {
+		t.Errorf("seed user-data = %q, want the Proxy's cloud-init verbatim", files["user-data"])
+	}
+	if !strings.Contains(files["vendor-data"], "qemu-guest-agent") {
+		t.Errorf("seed vendor-data = %q, want the guest-agent script", files["vendor-data"])
+	}
 }
 
 func TestCreate_freshHost(t *testing.T) {
@@ -234,7 +242,7 @@ func TestCreate_fetchesImageOnFirstUse(t *testing.T) {
 	api := newFakeAPI("default")
 	cfg := testConfig("nas1")
 	cfg.KVM.Images["debian-13"] = provider.KVMImage{URL: srv.URL + "/debian.qcow2", SHA256: testImageSHA()}
-	p := newWithDialer(cfg, hostFakes{"nas1": api}.dial, nil)
+	p := newWithDialer(cfg.Name, *cfg.KVM, hostFakes{"nas1": api}.dial, nil)
 
 	if _, err := p.Create(context.Background(), testRequest()); err != nil {
 		t.Fatalf("Create: %v", err)
@@ -472,28 +480,25 @@ func TestCreate_hostFailures(t *testing.T) {
 	t.Parallel()
 	cases := []struct {
 		name  string
-		setup func() (hostFakes, string)
+		setup func() hostFakes
 		class error
 	}{
 		{
 			name:  "host unreachable",
-			setup: func() (hostFakes, string) { return hostFakes{}, "nas1" },
+			setup: func() hostFakes { return hostFakes{} },
 			class: provider.ErrTransient,
 		},
 		{
-			name: "pool missing",
-			setup: func() (hostFakes, string) {
-				api := newFakeAPI("other")
-				return hostFakes{"nas1": api}, "nas1"
-			},
+			name:  "pool missing",
+			setup: func() hostFakes { return hostFakes{"nas1": newFakeAPI("other")} },
 			class: provider.ErrPermanent,
 		},
 		{
 			name: "domain xml rejected",
-			setup: func() (hostFakes, string) {
+			setup: func() hostFakes {
 				api := readyHost()
 				api.fail["DomainDefineXML"] = lvError(libvirt.ErrConfigUnsupported, "no q35 here")
-				return hostFakes{"nas1": api}, "nas1"
+				return hostFakes{"nas1": api}
 			},
 			class: provider.ErrPermanent,
 		},
@@ -501,8 +506,7 @@ func TestCreate_hostFailures(t *testing.T) {
 	for _, tc := range cases {
 		t.Run(tc.name, func(t *testing.T) {
 			t.Parallel()
-			fakes, host := tc.setup()
-			p := newTestProvider(fakes, host)
+			p := newTestProvider(tc.setup(), "nas1")
 			_, err := p.Create(context.Background(), testRequest())
 			wantClass(t, err, tc.class)
 		})
@@ -722,6 +726,28 @@ func TestDelete_missingPoolIsNotAnError(t *testing.T) {
 	}
 }
 
+// A host dropped from the config leaves the operator no way to reach the
+// domain, so Delete has nothing left to clean. Failing instead would pin
+// the Proxy's finalizer forever, since the reconciler's delete path cannot
+// make progress without a Delete that can succeed.
+func TestDelete_unconfiguredHostIsNotAnError(t *testing.T) {
+	t.Parallel()
+	api := readyHost()
+	managedDomain(t, api, testName, testUID, true, nil)
+	p := newTestProvider(hostFakes{"nas1": api}, "nas1")
+
+	if err := p.Delete(context.Background(), "hosts/nas9/domains/"+testName); err != nil {
+		t.Fatalf("Delete over an unconfigured host: %v", err)
+	}
+	if api.domain(testName) == nil {
+		t.Error("domain on the configured host was touched; only the named host's state may change")
+	}
+	// Get still refuses, so the live path cannot mistake a config typo for a
+	// missing VM and recreate it somewhere else.
+	_, err := p.Get(context.Background(), "hosts/nas9/domains/"+testName)
+	wantClass(t, err, provider.ErrPermanent)
+}
+
 func TestDelete_errors(t *testing.T) {
 	t.Parallel()
 	cases := []struct {
@@ -731,7 +757,6 @@ func TestDelete_errors(t *testing.T) {
 		class error
 	}{
 		{"malformed id", "nope", func(*testing.T) hostFakes { return hostFakes{"nas1": readyHost()} }, provider.ErrPermanent},
-		{"host not configured", "hosts/nas9/domains/x", func(*testing.T) hostFakes { return hostFakes{"nas1": readyHost()} }, provider.ErrPermanent},
 		{"host unreachable", "hosts/nas1/domains/x", func(*testing.T) hostFakes { return hostFakes{} }, provider.ErrTransient},
 		{
 			"undefine fails", "hosts/nas1/domains/" + testName,
@@ -880,15 +905,6 @@ func TestMapState(t *testing.T) {
 	}
 }
 
-func TestNew_badKeyPathFailsFast(t *testing.T) {
-	t.Parallel()
-	cfg := testConfig("nas1")
-	cfg.KVM.SSHKeyFile = t.TempDir() + "/missing"
-	if _, err := New(context.Background(), cfg); err == nil {
-		t.Fatal("New succeeded with a missing key file")
-	}
-}
-
 // runAllOps drives one full lifecycle over a ready fake host so logging
 // tests see every provider method and every libvirt call.
 func runAllOps(t *testing.T, ctx context.Context, req provider.CreateRequest) {
@@ -956,7 +972,7 @@ func TestLogging_verbosityTiers(t *testing.T) {
 			ctx, snapshot := captureContext(tc.verbosity)
 			req := testRequest()
 			const sentinel = "SENTINEL-cloud-init-must-never-be-logged"
-			req.CloudInit = "#cloud-config\n" + sentinel + "\n"
+			req.CloudInit = "#cloud-config\nhostname: " + sentinel + "\n"
 
 			runAllOps(t, ctx, req)
 

internal/provider/kvm/metadata.go

@@ -37,15 +37,13 @@ func newInstanceMetadata(uid string, createdAt time.Time) instanceMetadata {
 // by the lookup, so only the local name is matched here.
 func parseInstanceMetadata(s string) (instanceMetadata, error) {
 	var raw struct {
-		XMLName   xml.Name `xml:"instance"`
-		Managed   string   `xml:"managed"`
-		UID       string   `xml:"uid"`
-		CreatedAt string   `xml:"createdAt"`
+		XMLName xml.Name `xml:"instance"`
+		instanceMetadata
 	}
 	if err := xml.Unmarshal([]byte(s), &raw); err != nil {
 		return instanceMetadata{}, fmt.Errorf("parsing domain metadata: %w", err)
 	}
-	return instanceMetadata{Managed: raw.Managed, UID: raw.UID, CreatedAt: raw.CreatedAt}, nil
+	return raw.instanceMetadata, nil
 }
 
 func (m instanceMetadata) managed() bool {

internal/provider/kvm/metadata_test.go

@@ -29,47 +29,37 @@ func TestInstanceMetadata_roundTrip(t *testing.T) {
 	}
 }
 
-func TestParseInstanceMetadata_libvirtPrefixedForm(t *testing.T) {
-	t.Parallel()
-	// libvirt re-serialises the stored element with whatever prefix it
-	// chooses; only the namespace URI is contractual.
-	in := `<op:instance xmlns:op="` + metadataURI + `">` +
-		`<op:managed>true</op:managed><op:uid>uid-2</op:uid>` +
-		`<op:createdAt>2026-09-04T10:00:00Z</op:createdAt></op:instance>`
-	m, err := parseInstanceMetadata(in)
-	if err != nil {
-		t.Fatalf("parseInstanceMetadata() error = %v", err)
-	}
-	if !m.managed() || m.UID != "uid-2" || m.createdAt().Hour() != 10 {
-		t.Errorf("parsed = %+v", m)
-	}
-}
-
-func TestParseInstanceMetadata_libvirtBareForm(t *testing.T) {
-	t.Parallel()
-	// What libvirt 11.6 actually returns from DomainGetMetadata: the element
-	// selected by our URI, serialised without any namespace declaration.
-	in := `<instance><managed>true</managed><uid>uid-3</uid>` +
-		`<createdAt>2026-09-10T07:30:00Z</createdAt></instance>`
-	m, err := parseInstanceMetadata(in)
-	if err != nil {
-		t.Fatalf("parseInstanceMetadata() error = %v", err)
-	}
-	if !m.managed() || m.UID != "uid-3" || m.createdAt().Minute() != 30 {
-		t.Errorf("parsed = %+v", m)
-	}
-}
-
-func TestParseInstanceMetadata_notManaged(t *testing.T) {
+func TestParseInstanceMetadata(t *testing.T) {
 	t.Parallel()
 	tests := []struct {
-		name    string
-		in      string
-		wantErr bool
+		name        string
+		in          string
+		wantErr     bool
+		wantManaged bool
+		wantUID     string
+		wantCreated time.Time
 	}{
-		{"managed false", `<instance xmlns="` + metadataURI + `"><managed>false</managed><uid>u</uid></instance>`, false},
-		{"foreign element", `<foo xmlns="urn:other"><managed>true</managed></foo>`, true},
-		{"garbage", `not xml`, true},
+		{
+			// libvirt re-serialises the stored element with whatever prefix it
+			// chooses; only the namespace URI is contractual.
+			name: "libvirt prefixed form",
+			in: `<op:instance xmlns:op="` + metadataURI + `">` +
+				`<op:managed>true</op:managed><op:uid>uid-2</op:uid>` +
+				`<op:createdAt>2026-09-04T10:00:00Z</op:createdAt></op:instance>`,
+			wantManaged: true, wantUID: "uid-2", wantCreated: time.Date(2026, 9, 4, 10, 0, 0, 0, time.UTC),
+		},
+		{
+			// What libvirt 11.6 actually returns from DomainGetMetadata: the
+			// element selected by our URI, serialised without any namespace
+			// declaration.
+			name: "libvirt bare form",
+			in: `<instance><managed>true</managed><uid>uid-3</uid>` +
+				`<createdAt>2026-09-10T07:30:00Z</createdAt></instance>`,
+			wantManaged: true, wantUID: "uid-3", wantCreated: time.Date(2026, 9, 10, 7, 30, 0, 0, time.UTC),
+		},
+		{name: "managed false", in: `<instance xmlns="` + metadataURI + `"><managed>false</managed><uid>u</uid></instance>`, wantUID: "u"},
+		{name: "foreign element", in: `<foo xmlns="urn:other"><managed>true</managed></foo>`, wantErr: true},
+		{name: "garbage", in: `not xml`, wantErr: true},
 	}
 	for _, tc := range tests {
 		t.Run(tc.name, func(t *testing.T) {
@@ -78,8 +68,8 @@ func TestParseInstanceMetadata_notManaged(t *testing.T) {
 			if (err != nil) != tc.wantErr {
 				t.Fatalf("error = %v, wantErr %v", err, tc.wantErr)
 			}
-			if m.managed() {
-				t.Errorf("managed() = true for %q", tc.in)
+			if m.managed() != tc.wantManaged || m.UID != tc.wantUID || !m.createdAt().Equal(tc.wantCreated) {
+				t.Errorf("parsed = %+v, want managed=%v uid=%q createdAt=%v", m, tc.wantManaged, tc.wantUID, tc.wantCreated)
 			}
 		})
 	}

internal/provider/kvm/seed.go

@@ -3,6 +3,9 @@ package kvm
 import (
 	"bytes"
 	"fmt"
+	"io"
+	"mime/multipart"
+	"net/textproto"
 	"strings"
 
 	"github.com/kdomanski/iso9660"
@@ -11,20 +14,97 @@ import (
 // seedLabel is the volume label cloud-init's NoCloud datasource looks for.
 const seedLabel = "cidata"
 
-// defaultVendorData is the cloud-init vendor-data shipped on every seed
-// unless the config overrides it. cloud-init merges it under the Proxy's own
-// user-data, so stock images grow a guest agent (which lets the provider
-// read the VM's IP before ARP sees traffic) without the user's cloud-init
-// having to know about it.
-const defaultVendorData = `#cloud-config
-packages:
-  - qemu-guest-agent
-runcmd:
-  - [systemctl, enable, --now, qemu-guest-agent]
+// guestAgentScript installs and starts the QEMU guest agent, which is how
+// the provider learns a VM's address before the host's ARP table has seen
+// any traffic from it.
+//
+// It ships as vendor-data, and as a *script* rather than a cloud-config,
+// because that is the only shape cloud-init cannot drop. Vendor-data
+// cloud-config is merged under the Proxy's own user-data with mergers that
+// keep the user-data's value for every key both documents define and never
+// append lists, so a Proxy carrying its own packages: silently lost the
+// agent. The strategy cannot be overridden from either document: merge_how
+// in user-data governs only the first source, and merge_how in vendor-data
+// is consumed and discarded before the document is stored. A vendor-data
+// script bypasses all of it — cloud-init gives vendor-data its own handler
+// chain, writes scripts to a separate directory and runs them from
+// cc_scripts_vendor, after the user's packages are installed and before the
+// user's runcmd.
+const guestAgentScript = `#!/bin/sh
+# Added by egress-proxies-operator so the hypervisor can report this VM's
+# address. Set installGuestAgent: false on the provider to leave it out.
+set -e
+if ! command -v qemu-ga >/dev/null 2>&1 && [ ! -x /usr/sbin/qemu-ga ]; then
+	if command -v apt-get >/dev/null 2>&1; then
+		apt-get update
+		DEBIAN_FRONTEND=noninteractive apt-get install -y qemu-guest-agent
+	elif command -v dnf >/dev/null 2>&1; then
+		dnf install -y qemu-guest-agent
+	elif command -v yum >/dev/null 2>&1; then
+		yum install -y qemu-guest-agent
+	elif command -v zypper >/dev/null 2>&1; then
+		zypper --non-interactive install qemu-guest-agent
+	elif command -v apk >/dev/null 2>&1; then
+		apk add qemu-guest-agent
+	else
+		echo "no known package manager; install qemu-guest-agent in the image" >&2
+		exit 1
+	fi
+fi
+if command -v systemctl >/dev/null 2>&1; then
+	systemctl enable --now qemu-guest-agent
+elif command -v rc-update >/dev/null 2>&1; then
+	rc-update add qemu-guest-agent default
+	rc-service qemu-guest-agent start
+fi
 `
 
-// buildSeedISO renders a NoCloud seed: user-data verbatim, vendor-data as
-// given, and a meta-data whose instance-id is the deterministic VM name, so
+// buildVendorData renders the seed's vendor-data. With nothing configured
+// it is the guest-agent script alone; with both, a MIME multipart carrying
+// the configured document and the script as separate parts, since
+// vendor-data is a single file and the two must not be concatenated.
+func buildVendorData(configured string, installAgent bool) (string, error) {
+	switch {
+	case !installAgent:
+		return configured, nil
+	case strings.TrimSpace(configured) == "":
+		return guestAgentScript, nil
+	default:
+		return multipartVendorData(configured, guestAgentScript)
+	}
+}
+
+// multipartVendorData packs several documents into one vendor-data blob.
+// cloud-init decides a payload is multipart by finding "mime-version:"
+// within its first 4096 bytes, so those headers lead. The configured
+// document goes in as text/plain on purpose: cloud-init re-sniffs that type
+// from the content's own header line, which keeps this code out of the
+// business of classifying what the user wrote.
+func multipartVendorData(configured, script string) (string, error) {
+	var body bytes.Buffer
+	w := multipart.NewWriter(&body)
+	parts := []struct{ contentType, content string }{
+		{"text/plain", configured},
+		{"text/x-shellscript", script},
+	}
+	for _, p := range parts {
+		part, err := w.CreatePart(textproto.MIMEHeader{"Content-Type": {p.contentType}})
+		if err != nil {
+			return "", fmt.Errorf("building vendor-data part: %w", err)
+		}
+		if _, err := io.WriteString(part, p.content); err != nil {
+			return "", fmt.Errorf("writing vendor-data part: %w", err)
+		}
+	}
+	if err := w.Close(); err != nil {
+		return "", fmt.Errorf("closing vendor-data: %w", err)
+	}
+	return fmt.Sprintf("MIME-Version: 1.0\r\nContent-Type: multipart/mixed; boundary=%q\r\n\r\n%s",
+		w.Boundary(), body.String()), nil
+}
+
+// buildSeedISO renders a NoCloud seed: user-data and vendor-data verbatim,
+// and a meta-data whose instance-id is the deterministic VM name, so
 // cloud-init never mistakes a reboot for a new instance. The ISO's volume
 // descriptor carries the build time, so two builds differ in a few
 // timestamp bytes; the files and label are identical.

internal/provider/kvm/seed_test.go

@@ -4,6 +4,8 @@ import (
 	"bytes"
 	"io"
 	"maps"
+	"mime"
+	"mime/multipart"
 	"strings"
 	"testing"
 
@@ -42,7 +44,7 @@ func readSeed(t *testing.T, iso []byte) (label string, files map[string]string)
 func TestBuildSeedISO_contents(t *testing.T) {
 	t.Parallel()
 	userData := "#cloud-config\npackages: [squid]\n"
-	iso, err := buildSeedISO("proxy-abc123def456ghij", userData, defaultVendorData)
+	iso, err := buildSeedISO("proxy-abc123def456ghij", userData, guestAgentScript)
 	if err != nil {
 		t.Fatalf("buildSeedISO() error = %v", err)
 	}
@@ -56,7 +58,7 @@ func TestBuildSeedISO_contents(t *testing.T) {
 	if files["user-data"] != userData {
 		t.Errorf("user-data = %q, want verbatim cloud-init", files["user-data"])
 	}
-	if files["vendor-data"] != defaultVendorData {
+	if files["vendor-data"] != guestAgentScript {
 		t.Errorf("vendor-data = %q", files["vendor-data"])
 	}
 	if got, want := files["meta-data"], "instance-id: proxy-abc123def456ghij\nlocal-hostname: proxy-abc123def456ghij\n"; got != want {
@@ -89,7 +91,7 @@ func TestBuildSeedISO_deterministicContent(t *testing.T) {
 
 func TestBuildSeedISO_emptyUserData(t *testing.T) {
 	t.Parallel()
-	iso, err := buildSeedISO("proxy-x", "", defaultVendorData)
+	iso, err := buildSeedISO("proxy-x", "", guestAgentScript)
 	if err != nil {
 		t.Fatalf("buildSeedISO() error = %v", err)
 	}
@@ -98,3 +100,134 @@ func TestBuildSeedISO_emptyUserData(t *testing.T) {
 		t.Errorf("user-data = %q (present=%v), want an empty file to still exist", body, ok)
 	}
 }
+
+// The whole point of shipping the agent as vendor-data: whatever the Proxy
+// wrote reaches the guest untouched, in any of cloud-init's formats. The
+// previous implementation parsed and re-rendered cloud-config user-data,
+// which dropped comments, reordered keys and re-typed unquoted scalars
+// (version: 1.10 arrived as 1.1).
+func TestBuildSeedISO_userDataIsNeverRewritten(t *testing.T) {
+	t.Parallel()
+	tests := []struct {
+		name     string
+		userData string
+	}{
+		{"cloud-config with its own packages", "#cloud-config\n# keep me\npackages:\n  - squid\nversion: 1.10\nchpasswd: {list: 'u:0123456789'}\n"},
+		{"jinja template", "## template: jinja\n#cloud-config\nhostname: {{ v1.local_hostname }}\n"},
+		{"shell script", "#!/bin/sh\napt-get install -y squid\n"},
+		{"include", "#include\nhttps://example.invalid/x\n"},
+		{"mime multipart", "MIME-Version: 1.0\r\nContent-Type: multipart/mixed; boundary=\"x\"\r\n\r\n--x\r\nContent-Type: text/cloud-config\r\n\r\n#cloud-config\npackages: [squid]\n\r\n--x--\r\n"},
+		{"gzip payload", "\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03binary\x00bytes"},
+		{"leading blank line", "\n\n#cloud-config\npackages: [squid]\n"},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			vendor, err := buildVendorData("", true)
+			if err != nil {
+				t.Fatalf("buildVendorData() error = %v", err)
+			}
+			iso, err := buildSeedISO("proxy-x", tc.userData, vendor)
+			if err != nil {
+				t.Fatalf("buildSeedISO() error = %v", err)
+			}
+			_, files := readSeed(t, iso)
+			if files["user-data"] != tc.userData {
+				t.Errorf("user-data was rewritten:\n got %q\nwant %q", files["user-data"], tc.userData)
+			}
+			if !strings.Contains(files["vendor-data"], "qemu-guest-agent") {
+				t.Errorf("vendor-data = %q, want the agent script", files["vendor-data"])
+			}
+		})
+	}
+}
+
+func TestBuildVendorData_agentScriptAlone(t *testing.T) {
+	t.Parallel()
+	got, err := buildVendorData("", true)
+	if err != nil {
+		t.Fatalf("buildVendorData() error = %v", err)
+	}
+	if got != guestAgentScript {
+		t.Errorf("vendor-data = %q, want the bare script", got)
+	}
+	if !strings.HasPrefix(got, "#!") {
+		t.Error("vendor-data must start with a shebang or cloud-init treats it as something other than a script")
+	}
+}
+
+func TestBuildVendorData_optOutKeepsConfiguredOnly(t *testing.T) {
+	t.Parallel()
+	configured := "#cloud-config\nntp: {enabled: true}\n"
+	tests := []struct {
+		name       string
+		configured string
+		want       string
+	}{
+		{"nothing configured", "", ""},
+		{"configured document passes through", configured, configured},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			got, err := buildVendorData(tc.configured, false)
+			if err != nil {
+				t.Fatalf("buildVendorData() error = %v", err)
+			}
+			if got != tc.want {
+				t.Errorf("vendor-data = %q, want %q", got, tc.want)
+			}
+			if strings.Contains(got, "qemu-guest-agent") {
+				t.Error("agent script present although installGuestAgent is off")
+			}
+		})
+	}
+}
+
+// Both documents have to survive in one file, and cloud-init only looks for
+// the multipart marker in the first 4096 bytes.
+func TestBuildVendorData_multipartCarriesBoth(t *testing.T) {
+	t.Parallel()
+	configured := "#cloud-config\nntp:\n  enabled: true\n"
+	got, err := buildVendorData(configured, true)
+	if err != nil {
+		t.Fatalf("buildVendorData() error = %v", err)
+	}
+	head := got
+	if len(head) > 4096 {
+		head = head[:4096]
+	}
+	if !strings.Contains(strings.ToLower(head), "mime-version:") {
+		t.Fatalf("no mime-version: in the first 4096 bytes; cloud-init would treat this as a single document:\n%s", head)
+	}
+	_, params, err := mime.ParseMediaType(strings.SplitN(got, "\r\n\r\n", 2)[0][strings.Index(got, "Content-Type:")+len("Content-Type: "):])
+	if err != nil {
+		t.Fatalf("parsing the multipart content type: %v", err)
+	}
+	body := strings.SplitN(got, "\r\n\r\n", 2)[1]
+	r := multipart.NewReader(strings.NewReader(body), params["boundary"])
+	var bodies, types []string
+	for {
+		part, err := r.NextPart()
+		if err == io.EOF {
+			break
+		}
+		if err != nil {
+			t.Fatalf("reading part: %v", err)
+		}
+		content, err := io.ReadAll(part)
+		if err != nil {
+			t.Fatalf("reading part body: %v", err)
+		}
+		bodies = append(bodies, string(content))
+		types = append(types, part.Header.Get("Content-Type"))
+	}
+	if len(bodies) != 2 || bodies[0] != configured || bodies[1] != guestAgentScript {
+		t.Fatalf("parts = %q, want the configured document then the script", bodies)
+	}
+	// text/plain makes cloud-init re-sniff the user's document from its own
+	// header line, so this code never has to classify it.
+	if types[0] != "text/plain" || types[1] != "text/x-shellscript" {
+		t.Errorf("part types = %v, want [text/plain text/x-shellscript]", types)
+	}
+}

internal/provider/kvm/trace.go

@@ -123,7 +123,10 @@ func (t *tracingAPI) DomainGetMetadata(d libvirt.Domain, typ int32, uri libvirt.
 func (t *tracingAPI) DomainInterfaceAddresses(d libvirt.Domain, source uint32, flags uint32) ([]libvirt.DomainInterface, error) {
 	c := t.begin("DomainInterfaceAddresses", "domain", d.Name, "source", libvirt.DomainInterfaceAddressesSource(source))
 	ifaces, err := t.inner.DomainInterfaceAddresses(d, source, flags)
-	c.detail("addresses", formatAddresses(ifaces)).done(err, "interfaces", len(ifaces))
+	if t.log.V(2).Enabled() {
+		c.detail("addresses", formatAddresses(ifaces))
+	}
+	c.done(err, "interfaces", len(ifaces))
 	return ifaces, err
 }
 
@@ -140,11 +143,14 @@ func formatAddresses(ifaces []libvirt.DomainInterface) []string {
 func (t *tracingAPI) ConnectListAllDomains(needResults int32, flags libvirt.ConnectListAllDomainsFlags) ([]libvirt.Domain, uint32, error) {
 	c := t.begin("ConnectListAllDomains", flagKV(int64(flags))...)
 	doms, n, err := t.inner.ConnectListAllDomains(needResults, flags)
-	names := make([]string, 0, len(doms))
-	for _, d := range doms {
-		names = append(names, d.Name)
+	if t.log.V(2).Enabled() {
+		names := make([]string, 0, len(doms))
+		for _, d := range doms {
+			names = append(names, d.Name)
+		}
+		c.detail("names", names)
 	}
-	c.detail("names", names).done(err, "count", len(doms))
+	c.done(err, "count", len(doms))
 	return doms, n, err
 }