Delete stops and destroys a VM of ours under the per-name lock, then
deletes its seed on the VM's node, or on every node when no VM of ours
was left. A hibernated VM of ours is Permanent, as in Create. The
Checker verifies each node concurrently (version, node status, ISO
storage, templates) and adds CertificateInvalid, CredentialsUnavailable,
StorageNotFound and TemplateNotFound to the shared reasons. The
ProviderHealth type comment gains proxmox.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Four unrelated working-tree changes, none of them touching operator
behaviour:
- review/*.md -> docs/reviews/2026-09-17-kvm2-provider-review/, byte for
byte, so the kvm2 review sits with the rest of the docs under a name
that says which review it was.
- AGENTS.md: replaced the generic kubebuilder scaffold guide with what is
actually true of this repo — the real component layout, the lint/test
commands that work here, and the behavioural gotchas (replace-on-change
hashing, the cloud-init Secret label, in-memory leases).
- .claude/settings.json: the allow list still carried /Users/jan.novak
paths from the macOS checkout; re-pointed at /home/novakj and collapsed
the one-off entries into prefixes.
- config/samples/proxy_kvm2_crawl_only_one.yaml: a one-VM cut of the crawl
pool, the rest of the members left commented out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
proxy_kvm2.yaml shows one VM; nothing showed the shape a crawler actually
leases from. Add a five-Proxy pool on the kvm2-home entry: four share one
labelled cloud-init Secret, and the fifth carries its own inline block
because it runs squid on 8080 and spec.port has to match the http_port a
shared Secret would pin.
It stays out of the samples kustomization so `apply -k config/samples`
doesn't boot a pool along with the one-per-provider samples. All six
objects validate against the installed CRD under --dry-run=server.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes out the review-fix campaign. The kvm2 provider was deployed
in-cluster and validated against beelink -- the first time any of this
code had run against real libvirt rather than the in-memory fake.
config/manager/providers_config.yaml ships the kvm2-home entry enabled,
since the operator is now running it for real.
review/ carries the code review the four H-commits reference, so H1..H4
resolve to something rather than dangling: scope, the seven-chunk sweep,
the escalations, the ten deep dives and the final report. .claude/agents
carries the two reviewer definitions that produced them.
What the validation did not prove is recorded next to what it did: H2's
error paths need a body that fails part way through, and a healthy 433 MB
download never finishes short, so the unverified libvirtd short-stream
behaviour is still open. The kvm2 integration test has also still never
been run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Some lines ignored the zap encoder entirely, so `kubectl logs | jq` could
never work:
I0917 19:50:02.580097 1 leaderelection.go:258] "Attempting to acquire
leader lease..." lock="egress-proxies-operator-system/b47711d1..."
That is klog's own stderr writer, not ours. client-go's leader elector
calls klog.FromContext(ctx), and controller-runtime hands the elector a
context with no logr in it -- mgr.Start receives ctrl.SetupSignalHandler(),
which carries no logger. FromContext falls back to Background(), and with
no logger registered that is klog's text sink. Nothing here imported or
configured klog; it was an indirect dependency that around 60 linked
packages use, including client-go/tools/{leaderelection,cache,record} and
client-go/transport. controller-runtime does not route it either -- its
only klog.SetLogger call is in its own test suite.
Plain klog.SetLogger, deliberately not SetLoggerWithOptions with
ContextualLogger(true). The contextual form is what klog's docs recommend
because it skips a klogr round trip, but it also disables klog's verbosity
gate and hands the decision to zap, whose level under Development is Debug
-- so client-go's V(1) chatter, suppressed today by klog's own -v=0, would
have started appearing. Plain SetLogger keeps klog's check in front and
still redirects what it does emit. Same fix to the format, no new volume,
which matters because volume is what prompted this.
The discovery API's http.Server set no ErrorLog, so net/http's own faults
(a failed Accept, a panic escaping a handler) went to the stdlib logger on
stderr. It now drains into logr through logr.ToSlogHandler, the pattern
internal/provider/gcp/wirelog.go already uses for the same purpose.
The deployment gets --zap-encoder=json and --zap-time-encoding=rfc3339.
The flag beats the in-code Development default -- addDefaults fills
NewEncoder only when it is nil -- and the time encoding is not optional,
since the JSON encoder defaults to epoch seconds. Verbosity stays at debug
on purpose; --zap-log-level=info drops the per-RPC lines.
zap.Options{Development: true} stays as the in-code default: flipping it
would change the format for every deployment of this operator, and the
flags get the same result here without that reach.
k8s.io/klog/v2 moves from indirect to direct in go.mod. It was already in
the module graph, so nothing new is downloaded.
No tests: three changes, all wiring, in a file with no test coverage, and
pinning klog.SetLogger means mutating process-global state that leaks into
every other test in the package. Verified in-cluster instead --
`kubectl logs | jq -e -s 'length > 0'` now passes, which fails if any
single line is not JSON.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baseVolumeName was base-<key>.<format>, byte-identical to what the kvm
provider builds in the same pool -- for URL images too, since kvm takes
hex(sum[:8]) and kvm2 takes hex(sum[:])[:16], the same eight bytes -- with
a completely separate lock table. The shipped config has a live kvm-home
entry and a commented-out kvm2-home entry on the same host, the same key
debian-13 and the same default pool, and the comment invites the operator
to enable it. Both integration tests use key test-image. On beelink both
live overlays back onto kvm's base-debian-13.qcow2.
Since cachedImage accepts any 64-hex marker when no sha256 is configured
and never re-hashes the base, that meant two entries pointing one key at
different URLs silently shared a file; a concurrent cold fetch let one
entry delete the other's half-uploaded base and boot a guest from the
remains; and a sha256 edit deleted the base in place under live overlays,
which libvirt permits without complaint.
The name now carries the provider entry: base-<entry>-<key>.<format>.
imageSource gained an Entry field and resolveImage takes the provider
name, so every production call site follows from imageSource.volumeName().
That closes the cross-entry sharing outright, kvm<->kvm2 and kvm2<->kvm2
alike, and turns the per-Provider imageLocks from a latent bug into a
correct one: with names unique per entry, two entries cannot contend for a
file, so no process-wide lock table is needed.
The entry name now reaches a file name, so New checks it against the
existing kvmIdentifier rule, exported as provider.IsKVMIdentifier rather
than duplicated. Provider names are otherwise validated for all types at
once and only for being non-empty and unique, so a name with a slash would
have produced a volume nothing could address.
One companion fix, because the naming alone does not stop it:
cachedImage's digest-mismatch branch deleted the marker and the base and
refetched. It now returns ErrPermanent naming both digests and telling the
operator what to do. That branch was the remaining path that unlinks a
good file under running domains, and the contract already said "add a new
key instead" -- the code was contradicting it. The no-marker and
non-hex-marker branches keep deleting, since there the base is debris.
Not fixed, and said so in the docs rather than left to look closed:
editing images.<key>.url is still a silent no-op, since nothing records
the URL; and cross-process fencing still needs the deferred atomic publish
(upload to a temp name, verify, publish), which lands where H2 put the
discard.
Migration is a non-event. The base-debian-13.qcow2 on beelink is kvm's,
kvm keeps using it untouched, and kvm2 downloads its own copy once per
host. Nothing is replaced under a running guest; the pool needs room for
both, which the prerequisites now say.
The eviction recipe in the guide named base-debian-13.qcow2, which after
this change is kvm's file -- following it would have deleted the wrong
provider's base. It now greps for the prefix and deletes the marker first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.
- A timed-out check no longer drops the link. The check goes through the new
hostPool.probe, which abandons its RPCs when the context ends and leaves
the connection alone, so an image upload or a Create running on the same
link is not cut off. A link that really died is closed by the keepalive.
One abandoned probe per host may stay waiting; until it returns, the next
check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
hostEntry used a mutex, so a check could wait out a reconcile redialing a
dead host — minutes — past its own timeout. The slot is a one-slot channel
and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
The flat 15s default contradicted the validation that the timeout may not
exceed the interval, so --provider-check-interval=12s alone failed to
start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A leader-only runnable checks every configured provider concurrently
under one timeout per round (first round at start, then every interval),
keeps the latest results for the metrics collector, logs targets going
down, changing reason or coming back, and publishes one ProviderHealth
object per provider: created with a managed-by label when missing, then
a status merge patch. Per-target lastTransitionTime survives rounds in
which ready does not flip. Objects of providers that are no longer
configured are deleted once at start; with checks turned off, all are.
Reads go through the API reader, so no cluster-wide informer is needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One operator-owned object per configured provider will carry the
periodic check result: a Ready condition, per-target verdicts with their
own transition times, and a ready/total count for kubectl get. There is
no spec; only a viewer role is shipped since the operator is the sole
writer. Written by hand (kubebuilder is not installed), registered in
the existing AddKnownTypes call rather than a second init().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default manager config gains a kvm-home provider reaching beelink
over Tailscale, the provider sample names the same host, and the
manager kustomization uses the locally built kvm-dev image.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging the bootstrap into each Proxy's cloud-config was the wrong
mechanism twice over. Detection only matched a literal #cloud-config
first line, so a jinja template or a MIME archive still fell back to
vendor-data, where cloud-init's default mergers drop it — the exact
failure the merge existed to prevent. And the YAML round-trip re-typed
unquoted scalars: version: 1.10 reached the guest as 1.1, comments were
dropped and keys reordered.
Neither is worth patching, because vendor-data delivered as a shell
script bypasses the merger entirely: cloud-init hands vendor-data to its
own handler chain, writes scripts to a separate directory and runs them
from cc_scripts_vendor, after the user's packages and before the user's
runcmd. The merge could not have been rescued anyway — merge_how in
user-data governs only the first source, and merge_how in vendor-data is
discarded before the document is stored.
The Proxy's cloud-init now reaches the seed byte for byte in any format.
installGuestAgent (default true) turns the script off; a configured
vendorData travels beside it as a MIME multipart, declared text/plain so
cloud-init re-sniffs it and this code never classifies user content.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In the kind cluster two KVM Proxies with real cloud-configs booted fine
but never left Provisioning: the guest agent was not installed, so the
provider could not learn the IP. cloud-init merges vendor-data under
user-data with its default mergers, which keep the user-data's value
for any key both define and never append lists, so a cloud-config with
its own `packages:` silently dropped the vendor-data's packages and
runcmd. The integration test passed only because its user-data was
empty.
withGuestAgent now appends qemu-guest-agent to `packages` and the
enable command to `runcmd` of the Proxy's own cloud-config (skipping
entries already present) and re-renders it. Vendor-data is no longer
defaulted; `vendorData` is plain extra data. User-data that is not a
cloud-config document (script, #include, MIME) still gets the bootstrap
as vendor-data. A cloud-config that is not a YAML mapping, or whose
packages/runcmd is not a list, fails the Proxy as ErrPermanent.
Validated in-cluster: both Proxies Ready and healthy in two minutes, 204
through squid from the LAN. Docs, sample comment, CHANGELOG and the
plan execution log (Run 7) updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The manager runs with a read-only root filesystem and the seed ISO
writer stages files in a temp directory, so the first in-cluster Create
failed with "creating iso writer: read-only file system". Mount a 64 MiB
emptyDir on /tmp.
That error is classified ErrPermanent by writeSeed, yet hostPool.call
treated every non-libvirt error as a transport failure and dropped the
cached SSH connection, redialling on each retry. Errors that already
carry a provider sentinel are local verdicts, not evidence about the
link; leave the connection alone for those.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The known-gaps entry, decision log, sample comment and builder comment
blamed the host; the cause is trixie's GRUB config ("terminal_output
gfxterm serial" with a serial console and no video device), reported
and reproduced upstream and still open. UEFI remains the default for the
reason recorded; bookworm images are noted as unaffected.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 5 on the real host bisected the guest hang: the Debian 13 image
hangs after GRUB under SeaBIOS with or without our seed and with either
CPU model, and boots under OVMF. The image's legacy boot chain is the
problem, not the provider.
images.<key>.firmware is "uefi" (default) or "bios". The domain XML
emits <os firmware="efi"> with secure boot off, so libvirt selects the
OVMF binary and manages the NVRAM file, which the existing nvram undefine
flag removes. Direct-URL images default to UEFI too. OVMF joins the
hypervisor prerequisites; sample, config table, known gaps, decision log
and the execution log record the finding.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Registers "kvm" in the provider builtins, mounts an optional kvm-ssh
Secret at /etc/proxy-operator/kvm, adds a kvm-home providers-config
sample and a proxy_kvm.yaml Proxy sample, documents the provider in
docs/kvm-provider.md (prerequisites, config reference, image cache and
its do-not-replace-in-use rule, known gaps), and updates the README
and architecture provider sections and decision log.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
config/manager/kustomization.yaml: commit the images: stanza that
`kustomize edit set image` (run by make deploy, including inside make
test-e2e) writes into this tracked file. It showed up as unexplained
drift twice; committing it once ends that -- the edit is idempotent, so
future deploy/e2e runs produce no diff. The example.com image name is
the e2e suite's placeholder default and gets overridden by IMG= on any
real deploy.
Execution log: the Status checklist's Step 3 line still said "Mock
provider" from before the pivot; a fresh session resuming from the
checklist alone would have been misled.
Co-Authored-By: Claude <noreply@anthropic.com>
Follows the approved lean-down plan
(docs/plans/2026-08-08-1335-lean-scaffold-cleanup.md). Removes from the
application's deployed footprint:
- config/network-policy/ and its commented enable line -- the user does
not need network policies at the moment.
- The webhook-only halves of config/default/kustomization.yaml: the
commented ../webhook and ../certmanager resource lines, the
manager_webhook_patch.yaml reference, the serving-cert ->
Validating/Mutating WebhookConfiguration cainjection replacement
blocks, and the crdkustomizecainjection* scaffold markers -- anchors
only for `kubebuilder create webhook`, which is a permanent non-goal.
- The two commented [WEBHOOK] blocks in config/crd/kustomization.yaml
plus the now-empty patches: key; kept the one-line
crdkustomizeresource marker since `kubebuilder create api` could
legitimately run again.
- config/crd/kustomizeconfig.yaml, whose only consumer was the removed
configurations: block.
Explicitly kept per user direction: all of config/prometheus/, the
paired metrics-TLS-via-cert-manager plumbing (cert_metrics_manager_patch
+ the metrics-certs/ServiceMonitor replacement halves), all RBAC
manifests including the admin/editor/viewer helper roles, and all
developer tooling.
Also records in the execution log why the webhook machinery existed at
all: kubebuilder init emits it unconditionally, verified against the
v4.15.0 binary that no init flag can suppress it -- scaffold-then-prune
is the only supported path, and the pruning pass should have happened
at Step 0.
Verified: kustomize build clean on config/default and config/crd,
go build/vet clean with and without -tags=e2e, make test green with
coverage identical to pre-cleanup.
Co-Authored-By: Claude <noreply@anthropic.com>
Full ProxySpec/ProxyStatus/Proxy types per the plan: PlacementSpec,
CloudInitSpec, EndpointSpec, HealthCheckSpec, SecretKeySelector, all
defaults, and 7 CEL XValidation rules enforcing mode/provider
immutability, provider/endpoint required-iff-Managed/External, and
cloud-init exactly-one-of inline/secretRef.
Applies the four corrections identified during planning that would
otherwise be silent bugs: MaxLeases as *int32 (so an explicit 0 survives
Go round-trips instead of re-defaulting to 5), HealthCheck's
default={} marker (so nested defaults apply even when the field is
omitted entirely), MinLength=1 on Provider/CloudInit.Inline (so the CEL
has() checks stay simple), and listType=map on Conditions.
Adds pure helpers (EffectivePort, EffectiveHost, HealthCheckOrDefault,
MaxLeasesOrDefault) with table-driven tests, for use by the health
engine, discovery API, and spec-hash computation in later steps.
Patches the scaffolded placeholder controller test's resource literal to
a schema-valid spec so it survives the new CRD validation — the test
itself is rewritten wholesale in Step 4 alongside the real reconciler.
Regenerated deepcopy and the CRD; make test green (envtest confirmed all
7 CEL rules enforced by a real apiserver).
Co-Authored-By: Claude <noreply@anthropic.com>
Bootstraps the empty repo into a kubebuilder go/v4 project: group
crawl.example.com, version v1alpha1, kind Proxy (namespaced). Keeps the
existing module path and preserves the repo's Go/testing/changelog
conventions from CLAUDE.md untouched.
Drops the scaffolded GitHub Actions workflows since the remote is Gitea,
not GitHub. Everything else is default kubebuilder output, unmodified,
so later diffs stay reviewable against a known baseline.
Full implementation plan: docs/plans/2026-08-07-1747-proxy-operator.md
Co-Authored-By: Claude <noreply@anthropic.com>
kacerrandClaude Opus 5.5