Let a Proxy finish deleting when its instance can no longer be reached
reconcileDelete asked Get first and dropped the finalizer only on not-found, so a permanent Get retried forever: a KVM host removed from the config, or a providerID the provider cannot resolve, left the Proxy in Terminating with no way out. It now tries Delete anyway, which is idempotent and can still finish where Get cannot converge, and releases the object when that succeeds. crawl.example.com/skip-cleanup abandons the instance for the cases that stay stuck, such as revoked credentials. It is checked before the provider lookup, so it also frees a Proxy whose provider is gone from the config. In the KVM provider, Delete over an unconfigured host is now a no-op success, mirroring removeVolumes over a missing pool. Get keeps failing permanently there on purpose: the live path reads not-found as "rebuild it", so a typo in hosts must not look like a missing VM. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -489,6 +489,15 @@ they read as choices, not accidents. Chronological by build step.
|
||||
cluster-wide Secret read RBAC — an unrestricted cache would hold every
|
||||
Secret in scope). Unlabelled referenced Secrets are invisible by
|
||||
construction, surfacing as `CloudInitError`.
|
||||
- **Deletion never gives up on its own, but can be released by hand:** a
|
||||
permanent cleanup failure keeps retrying with the finalizer on, because
|
||||
dropping it automatically would leak a running instance on any provider
|
||||
that answers "permission denied". Where `Get` fails permanently the
|
||||
reconciler still calls `Delete`, since it is idempotent and may finish
|
||||
where `Get` cannot converge. `crawl.example.com/skip-cleanup: "true"`
|
||||
abandons the instance and releases the object; it is checked before the
|
||||
provider lookup, so it also frees a Proxy whose provider is gone from the
|
||||
config.
|
||||
- **Events RBAC from the plan is omitted:** nothing wires an
|
||||
EventRecorder in the prototype, and granting verbs nothing uses would
|
||||
be RBAC lint noise. Add the marker together with the recorder if events
|
||||
|
||||
@@ -264,6 +264,14 @@ The provider logs through controller-runtime's logger, named `kvm`, with
|
||||
cleaned when the same Proxy reconciles again (Delete removes volumes by
|
||||
name even with no domain), but a Proxy CR that is gone for good leaves
|
||||
them for `virsh vol-list` to find.
|
||||
- **A host removed from the config takes its VMs with it.** `Delete` for a
|
||||
providerID naming an unconfigured host is a no-op success, so the Proxy
|
||||
finishes deleting instead of pinning its finalizer, but the domain and its
|
||||
volumes stay on the hypervisor and orphan GC cannot see them either (it
|
||||
only sweeps configured hosts). Delete the Proxies before retiring a host,
|
||||
or clean up afterwards with `virsh undefine --nvram` plus `virsh vol-delete`.
|
||||
`Get` still fails permanently for such a providerID on purpose: a typo in
|
||||
`hosts` must not look like a missing VM and trigger a rebuild elsewhere.
|
||||
- **The first image fetch is synchronous.** It holds a reconcile worker
|
||||
for the download (LAN speed, a few hundred MB, once per image per
|
||||
host). Other Proxies keep reconciling on the remaining workers.
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
# Execution: fix the five code-review findings on `feat/kvm-provider`
|
||||
Plan: [docs/plans/2026-09-10-2222-kvm-review-fixes.md](../plans/2026-09-10-2222-kvm-review-fixes.md)
|
||||
|
||||
- [x] Step 0 — Save plan into the repo
|
||||
- [x] Step 1 — Unwedge deletion (finding 1)
|
||||
- [ ] Step 2 — Validate names that become identifiers (finding 2)
|
||||
- [ ] Step 3 — Read the SSH material lazily (finding 3)
|
||||
- [ ] Step 4 — Guest agent via vendor-data script (findings 4 and 5)
|
||||
- [ ] Step 5 — Validate, record, push
|
||||
|
||||
## Step 0 — plan committed
|
||||
|
||||
Copied out of `~/.claude/plans/` and committed alone before touching code, as
|
||||
the repo's plan rules require.
|
||||
|
||||
## Step 1 — Unwedge deletion
|
||||
|
||||
Three changes, in the order the bug unfolds.
|
||||
|
||||
`reconcileDelete` calls `Get` first and only drops the finalizer when it
|
||||
answers not-found, so any permanent error there retried forever. It now
|
||||
switches on the class: a permanent `Get` still calls `Delete`, which is
|
||||
idempotent, and releases the object when that succeeds. Polling a `Get`
|
||||
that cannot converge was the wedge; `Delete` is the only call that can
|
||||
still make progress.
|
||||
|
||||
`crawl.example.com/skip-cleanup: "true"` abandons the instance and removes
|
||||
the finalizer. It is checked **before** the provider lookup, which also
|
||||
frees a Proxy whose provider was deleted from the config — a second wedge
|
||||
sitting three lines above the reported one, with the same shape. No
|
||||
`EventRecorder` exists on the reconciler and none was added, so the
|
||||
abandonment is an Info log; `docs/architecture.md` already records that
|
||||
decision for the whole controller.
|
||||
|
||||
In the KVM provider, `Delete` for a providerID naming an unconfigured host
|
||||
now logs and returns success. The precedent is two functions further down:
|
||||
`removeVolumes` treats a missing pool the same way, with a comment saying
|
||||
outright that refusing "would pin the Proxy's finalizer".
|
||||
|
||||
**`Get` deliberately still fails permanently for that case.** Making it
|
||||
answer not-found would have been a one-line fix for the wedge, but the live
|
||||
reconcile path reads not-found as "the VM is gone" and rebuilds it
|
||||
elsewhere. A typo in `hosts` would then quietly duplicate every Proxy on
|
||||
that host, with the originals still running where GC cannot see them, since
|
||||
`ListByTag` only sweeps configured hosts. The asymmetry is now pinned by a
|
||||
test that asserts both halves at once.
|
||||
|
||||
Worth noting: the review filed this as a KVM finding, but the fix is mostly
|
||||
in the controller, because KVM is the only provider that resolves a
|
||||
providerID segment against operator config at all. GCP and Kubernetes hand
|
||||
the lookup to the backend, which answers 404 for a retired zone or a deleted
|
||||
namespace, so they reach not-found and release on their own. Anything else
|
||||
that classifies permanently on the delete path — revoked RBAC on the
|
||||
Kubernetes provider is the realistic one — was equally stuck before this
|
||||
step and is what the annotation is really for. The KVM table case
|
||||
`"host not configured"` moved out of `TestDelete_errors` into its own named
|
||||
test, since it is no longer an error.
|
||||
Reference in New Issue
Block a user