Let a Proxy finish deleting when its instance can no longer be reached

reconcileDelete asked Get first and dropped the finalizer only on
not-found, so a permanent Get retried forever: a KVM host removed from
the config, or a providerID the provider cannot resolve, left the Proxy
in Terminating with no way out. It now tries Delete anyway, which is
idempotent and can still finish where Get cannot converge, and releases
the object when that succeeds.

crawl.example.com/skip-cleanup abandons the instance for the cases that
stay stuck, such as revoked credentials. It is checked before the
provider lookup, so it also frees a Proxy whose provider is gone from
the config.

In the KVM provider, Delete over an unconfigured host is now a no-op
success, mirroring removeVolumes over a missing pool. Get keeps failing
permanently there on purpose: the live path reads not-found as "rebuild
it", so a typo in hosts must not look like a missing VM.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-10 22:30:15 +00:00
co-authored by Claude Opus 5
parent ce365e2602
commit ba24d7e3d9
9 changed files with 213 additions and 6 deletions
@@ -0,0 +1,57 @@
# Execution: fix the five code-review findings on `feat/kvm-provider`
Plan: [docs/plans/2026-09-10-2222-kvm-review-fixes.md](../plans/2026-09-10-2222-kvm-review-fixes.md)
- [x] Step 0 — Save plan into the repo
- [x] Step 1 — Unwedge deletion (finding 1)
- [ ] Step 2 — Validate names that become identifiers (finding 2)
- [ ] Step 3 — Read the SSH material lazily (finding 3)
- [ ] Step 4 — Guest agent via vendor-data script (findings 4 and 5)
- [ ] Step 5 — Validate, record, push
## Step 0 — plan committed
Copied out of `~/.claude/plans/` and committed alone before touching code, as
the repo's plan rules require.
## Step 1 — Unwedge deletion
Three changes, in the order the bug unfolds.
`reconcileDelete` calls `Get` first and only drops the finalizer when it
answers not-found, so any permanent error there retried forever. It now
switches on the class: a permanent `Get` still calls `Delete`, which is
idempotent, and releases the object when that succeeds. Polling a `Get`
that cannot converge was the wedge; `Delete` is the only call that can
still make progress.
`crawl.example.com/skip-cleanup: "true"` abandons the instance and removes
the finalizer. It is checked **before** the provider lookup, which also
frees a Proxy whose provider was deleted from the config — a second wedge
sitting three lines above the reported one, with the same shape. No
`EventRecorder` exists on the reconciler and none was added, so the
abandonment is an Info log; `docs/architecture.md` already records that
decision for the whole controller.
In the KVM provider, `Delete` for a providerID naming an unconfigured host
now logs and returns success. The precedent is two functions further down:
`removeVolumes` treats a missing pool the same way, with a comment saying
outright that refusing "would pin the Proxy's finalizer".
**`Get` deliberately still fails permanently for that case.** Making it
answer not-found would have been a one-line fix for the wedge, but the live
reconcile path reads not-found as "the VM is gone" and rebuilds it
elsewhere. A typo in `hosts` would then quietly duplicate every Proxy on
that host, with the originals still running where GC cannot see them, since
`ListByTag` only sweeps configured hosts. The asymmetry is now pinned by a
test that asserts both halves at once.
Worth noting: the review filed this as a KVM finding, but the fix is mostly
in the controller, because KVM is the only provider that resolves a
providerID segment against operator config at all. GCP and Kubernetes hand
the lookup to the backend, which answers 404 for a retired zone or a deleted
namespace, so they reach not-found and release on their own. Anything else
that classifies permanently on the delete path — revoked RBAC on the
Kubernetes provider is the realistic one — was equally stuck before this
step and is what the annotation is really for. The KVM table case
`"host not configured"` moved out of `TestDelete_errors` into its own named
test, since it is no longer an error.