The kvm2 block reuses KVMConfig; validateKVM gains a block-name variant so
messages say kvm2. The package skeleton has a non-dialing New, strict
providerID parsing, the state table and a 34-row classify test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The health check shares each host's libvirt connection with the reconciles,
so it must not be able to take that connection away from them.
- A timed-out check no longer drops the link. The check goes through the new
hostPool.probe, which abandons its RPCs when the context ends and leaves
the connection alone, so an image upload or a Create running on the same
link is not cut off. A link that really died is closed by the keepalive.
One abandoned probe per host may stay waiting; until it returns, the next
check reports Timeout at once (errProbeBusy).
- Waiting for a host's connection slot now ends with the caller's context.
hostEntry used a mutex, so a check could wait out a reconcile redialing a
dead host — minutes — past its own timeout. The slot is a one-slot channel
and acquire bounds only the wait, never the uncontended take.
- --provider-check-timeout defaults to 0 and derives min(15s, interval).
The flat 15s default contradicted the validation that the timeout may not
exceed the interval, so --provider-check-interval=12s alone failed to
start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A leader-only runnable checks every configured provider concurrently
under one timeout per round (first round at start, then every interval),
keeps the latest results for the metrics collector, logs targets going
down, changing reason or coming back, and publishes one ProviderHealth
object per provider: created with a managed-by label when missing, then
a status merge patch. Per-target lastTransitionTime survives rounds in
which ready does not flip. Objects of providers that are no longer
configured are deleted once at start; with checks turned off, all are.
Reads go through the API reader, so no cluster-wide informer is needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each configured host is checked concurrently over the pooled SSH+libvirt
connection: libvirt version and hostname prove the daemon answers, then
the configured pool must exist and be running, with its free space
reported. Dial failures are now tagged with their stage (SSH material,
TCP, SSH handshake, libvirt connect) by a dialError that keeps the
message and unwraps to the cause, so classification is unchanged and the
check can say why a host is unreachable. A pool verdict is decided after
the call returns so it never drops a working connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Checker is an optional interface next to Provider: Check returns one
verdict per target and never fails as a whole. The kubernetes provider
lists at most one managed Pod cluster-wide (the verb and scope GC relies
on); the GCP provider fetches a single page of at most one managed
instance through a new Probe seam method, since the SDK treats
MaxResults as a page size and the existing AggregatedList drains every
page.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collapse duplicated logic the review found in PR #5 without changing
behaviour: one releaseFinalizer helper for the four finalizer exits in
reconcileDelete, classify reusing isClassified and provider.Class, the
image URL check and format/firmware values shared from the provider
package, imageSource and the metadata parser embedding their config
structs instead of copying fields, and the SSH loader as a small struct
that keeps only the SSH fields and is seeded by New instead of loading
the key twice. Tracing builds its V(2)-only lists only at V(2).
Tests: drop a duplicate New test, fold near-identical tests into tables
(connection drops, metadata parsing, cached images), add mustSSHAuth,
and use wantClass where classes were checked by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging the bootstrap into each Proxy's cloud-config was the wrong
mechanism twice over. Detection only matched a literal #cloud-config
first line, so a jinja template or a MIME archive still fell back to
vendor-data, where cloud-init's default mergers drop it — the exact
failure the merge existed to prevent. And the YAML round-trip re-typed
unquoted scalars: version: 1.10 reached the guest as 1.1, comments were
dropped and keys reordered.
Neither is worth patching, because vendor-data delivered as a shell
script bypasses the merger entirely: cloud-init hands vendor-data to its
own handler chain, writes scripts to a separate directory and runs them
from cc_scripts_vendor, after the user's packages and before the user's
runcmd. The merge could not have been rescued anyway — merge_how in
user-data governs only the first source, and merge_how in vendor-data is
discarded before the document is stored.
The Proxy's cloud-init now reaches the seed byte for byte in any format.
installGuestAgent (default true) turns the script off; a configured
vendorData travels beside it as a MIME multipart, declared text/plain so
cloud-init re-sniffs it and this code never classifies user content.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The kvm-ssh Secret is mounted optional, but kvm.New read the key and
known_hosts eagerly and registry.Build aborts the whole provider set on a
constructor error, so a missing or misnamed Secret exited the manager
before any controller started and took the kubernetes and gcp providers
down with it.
lazyAuth loads the material on the first connection and caches it once it
works, so a Secret created after the manager starts heals the affected
proxies on the next reconcile with no restart. A missing file classifies
transient for that reason; a key that does not parse is permanent.
New still checks both files and logs a WARNING: line when they are
unreadable, keeping the startup diagnostic the eager load was there for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A host name is pasted into the providerID hosts/<name>/domains/<vm>, which
parseProviderID splits on "/", so a name containing a slash produced an ID
that never parsed again: Get, Delete and GC all failed permanently and the
VM leaked with the Proxy's finalizer stuck.
The image map key has the same defect and lands in a file name rather than
an ID: base-<key>.<format> in the host's storage pool. Both now have to
match a plain identifier pattern. Machine-type keys are left alone; they
are only ever compared, never embedded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
reconcileDelete asked Get first and dropped the finalizer only on
not-found, so a permanent Get retried forever: a KVM host removed from
the config, or a providerID the provider cannot resolve, left the Proxy
in Terminating with no way out. It now tries Delete anyway, which is
idempotent and can still finish where Get cannot converge, and releases
the object when that succeeds.
crawl.example.com/skip-cleanup abandons the instance for the cases that
stay stuck, such as revoked credentials. It is checked before the
provider lookup, so it also frees a Proxy whose provider is gone from
the config.
In the KVM provider, Delete over an unconfigured host is now a no-op
success, mirroring removeVolumes over a missing pool. Get keeps failing
permanently there on purpose: the live path reads not-found as "rebuild
it", so a typo in hosts must not look like a missing VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In the kind cluster two KVM Proxies with real cloud-configs booted fine
but never left Provisioning: the guest agent was not installed, so the
provider could not learn the IP. cloud-init merges vendor-data under
user-data with its default mergers, which keep the user-data's value
for any key both define and never append lists, so a cloud-config with
its own `packages:` silently dropped the vendor-data's packages and
runcmd. The integration test passed only because its user-data was
empty.
withGuestAgent now appends qemu-guest-agent to `packages` and the
enable command to `runcmd` of the Proxy's own cloud-config (skipping
entries already present) and re-renders it. Vendor-data is no longer
defaulted; `vendorData` is plain extra data. User-data that is not a
cloud-config document (script, #include, MIME) still gets the bootstrap
as vendor-data. A cloud-config that is not a YAML mapping, or whose
packages/runcmd is not a list, fails the Proxy as ErrPermanent.
Validated in-cluster: both Proxies Ready and healthy in two minutes, 204
through squid from the LAN. Docs, sample comment, CHANGELOG and the
plan execution log (Run 7) updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The manager runs with a read-only root filesystem and the seed ISO
writer stages files in a temp directory, so the first in-cluster Create
failed with "creating iso writer: read-only file system". Mount a 64 MiB
emptyDir on /tmp.
That error is classified ErrPermanent by writeSeed, yet hostPool.call
treated every non-libvirt error as a transport failure and dropped the
cached SSH connection, redialling on each retry. Errors that already
carry a provider sentinel are local verdicts, not evidence about the
link; leave the connection alone for those.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The known-gaps entry, decision log, sample comment and builder comment
blamed the host; the cause is trixie's GRUB config ("terminal_output
gfxterm serial" with a serial console and no video device), reported
and reproduced upstream and still open. UEFI remains the default for the
reason recorded; bookworm images are noted as unaffected.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 5 on the real host bisected the guest hang: the Debian 13 image
hangs after GRUB under SeaBIOS with or without our seed and with either
CPU model, and boots under OVMF. The image's legacy boot chain is the
problem, not the provider.
images.<key>.firmware is "uefi" (default) or "bios". The domain XML
emits <os firmware="efi"> with secure boot off, so libvirt selects the
OVMF binary and manages the NVRAM file, which the existing nvram undefine
flag removes. Direct-URL images default to UEFI too. OVMF joins the
hypervisor prerequisites; sample, config table, known gaps, decision log
and the execution log record the finding.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first live run showed the trace working but noisy: testr printed
ts/level/msg and the per-run constants on every line, the error text
appeared twice, host appeared twice on image-cache lines, and the Go
method name was less useful than the libvirt function it maps to.
The tracer now logs rpc=virDomainLookupByName (go-libvirt's names are the
public API names without the vir prefix), error plus libvirtError (code
name) and libvirtCode, and image.go stops re-adding host. The integration
test replaces testr with a compact logr sink: time, logger name, the rpc
as headline, then fields, with provider/host/name omitted since the first
line of the run states them. Docs and execution log updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A tracingAPI decorator wraps the 17-method libvirt seam and logs one
"libvirt call" line per RPC after it returns: method, the identifiers it
was given, curated results, duration, and on failure the error with its
libvirt code, name and message. At V(2) the same line carries the domain
and volume XML, metadata and addresses; upload bodies are never logged.
The pool applies it in call() when V(1) is enabled and logs connect,
dial-failure and drop events, taking the caller's logger explicitly like
every other helper in the package.
The integration test installs a testr logger at KVM_TEST_LOG_LEVEL
(default 1) so the trace interleaves with t.Log output, which also ends
the "log.SetLogger(...) was never called" warning. Docs gained a Logging
section and the test env var; the KVM provider execution log records run
4 (guest hang at GRUB) that prompted this.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 3 on the real host got past Create and failed in Get: libvirt 11.6
selects the <instance> element by our URI in DomainGetMetadata but hands
it back with no namespace declaration, and the parser required the URI
on the element. The lookup already guarantees the namespace, so match on
the local name only. The fake now strips the declaration the way libvirt
does, and a new test pins the bare form. Execution log updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first run against a real hypervisor (beelink, Ubuntu, libvirt 11.6)
failed at DomainCreate with "Could not open base image: Permission denied"
even with correct DAC ownership. libvirt's virt-aa-helper only grants QEMU
access to disk sources that carry a path; the <disk type="volume"> form
chosen in Step 2 gives it pool and volume names instead, so the overlay and
its backing image get no AppArmor rule on any stock Debian/Ubuntu host.
buildDomainXML now takes DiskPath/SeedPath and emits type="file" disks.
ensureOverlay and writeSeed return the volume handles and Create resolves
both paths with StorageVolGetPath before defining the domain. The domain
XML test rejects any volume/pool reference so this cannot drift back; the
Create test asserts the sources equal the paths libvirt reported.
Execution log: Step 2 note marked reverted, Step 7 entry opened with the
two failed runs and the diagnosis.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Behind -tags=integration and self-skipping unless KVM_TEST_* names a
host: one sequential lifecycle (image fetch, Create, idempotent
re-Create, Running with IP, ListByTag, Delete, pool state afterwards),
driven through the provider directly so it needs no Step 5 wiring.
Adds `make test-integration-kvm` and documents the run and the
deviation from the testcontainers convention in docs/testing.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Create looks the domain up first and only starts it when it exists;
otherwise it caches the image, reuses or creates the overlay, rebuilds
the seed ISO unconditionally (a cancelled upload can leave a truncated
seed that no later attempt would repair), defines and starts the domain.
Get maps libvirt state plus the first routable IPv4 from ARP or the
guest agent onto the provider states. Delete destroys, undefines and
removes the per-VM volumes by name, tolerating every already-gone case.
ListByTag keeps managed domains across all hosts, skipping domains it
cannot read and failing the sweep only for an unreachable host.
Also corrects the Step 2 seed determinism test: the iso9660 writer
stamps the build time into the volume descriptor, so builds differ by a
few bytes; the test now compares label, files and size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
resolveImage maps placement.image to a configured key or URL; imageCache
streams the download through StorageVolUpload with a sha256 tee, writes
a marker volume when the upload is verified, and re-fetches when the
marker is missing or the configured checksum changed. Errors carry
taxonomy sentinels that classify now honours. Adds the in-memory fake
libvirt API used from here on.
Step 3b of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Per-host connection pool over SSH with the libvirt unix socket forwarded
through the session, lazy dial and redial after disconnect, context
cancellation for go-libvirt's context-less RPCs, handshake deadline and
SSH keepalives. Error classification onto the provider taxonomy. Tested
with a fake dialer and an in-process SSH server that forwards unix sockets.
Step 3 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Introduces internal/provider/kvm with the side-effect-free pieces of the
provider: providerID format/parse, the namespaced <metadata> element that
carries the GC contract, q35 domain XML over pool volumes, volume XML for
overlays and raw seeds, the NoCloud seed ISO built with kdomanski/iso9660,
and config defaults. All unit-tested.
Step 2 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds the kvm block to the providers config (SSH access, images map,
machine-type profiles, hosts list) with validation and table tests.
validate() now rejects mismatched type-specific blocks generically via
setBlocks() instead of pairwise checks.
Step 1 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The option.WithLogger logger also reaches cloud.google.com/go/auth,
which logged its token exchange at Debug — JWT assertion and bearer
token included. wireLogger now allowlists only the compute client's
api request/response records at Debug (fail-closed for future SDK
additions); Warn/Error pass through. String fields over 1KiB (e.g.
Shielded-VM UEFI dbx blobs) are elided recursively by default; the new
--gcp-wire-log-full-payloads flag restores verbatim payloads.
Co-Authored-By: Claude <noreply@anthropic.com>
Inject an option.WithLogger slog logger bridged to the zap sink via
logr.ToSlogHandler with a V(1) shift, so the SDK's Debug-level
"api request"/"api response" records (URL, headers, full payloads)
appear only at --zap-log-level=5. Startup warning when active, since
raw insert payloads include cloud-init user-data. Note WithLogger
overrides GOOGLE_SDK_GO_LOGGING_LEVEL for this client.
Co-Authored-By: Claude <noreply@anthropic.com>
New internal/version package: ldflags-stamped Commit with a
debug.ReadBuildInfo VCS fallback for host builds. Startup log line
carries commit + Go version; --version prints the hash and exits.
Makefile computes GIT_COMMIT (12 chars, -dirty on any local change) and
passes it to docker-build/buildx; Dockerfile injects it via -ldflags and
an org.opencontainers.image.revision label. make build now uses ./cmd —
file-argument builds skip Go's automatic VCS stamp.
Co-Authored-By: Claude <noreply@anthropic.com>
One V(1) line per GCP API call (insert/get/delete/aggregatedList) with
outcome and operation name, V(2) request/per-instance detail, and raw
googleapi status+reasons logged before classify collapses them. Curated
fields only — cloud-init user-data never reaches logs (test-enforced).
Co-Authored-By: Claude <noreply@anthropic.com>
Create/Get/Delete/ListByTag against real corev1.Pod objects in the same
cluster the operator runs in, running an ubuntu/squid container -- picked
by actually checking Docker Hub metadata (Canonical-published, rebuilt
the same day this was decided, 50M+ pulls) rather than guessing an image
reference. It's a public image, so kind nodes pull it directly with no
build/load step.
providerID is "<namespace>/<podName>", parsed via
cache.SplitMetaNamespaceKey -- the same self-contained-providerID
reasoning the plan already calls for on the GCP provider's zone-qualified
IDs. Pod state maps to InstanceState with Succeeded/Failed/Unknown all
collapsing to Terminated, since the reconciler already treats Stopped and
Terminated identically; Running-without-PodIP maps to Provisioning so an
empty IP is never published.
The client is built internally via ctrl.GetConfig() (in-cluster or local
kubeconfig, whichever applies), not threaded through the registry
Constructor signature -- this is what lets `make run` against a local
kind cluster and running in-cluster share the exact same code path with
no provider-specific wiring in cmd/main.go. New() is deliberately
untested (0% coverage): it's the one function that must never run under
`go test`, since it would happily connect to whatever cluster the
developer's kubeconfig points at. Tests construct Provider via an
unexported newWithClient(client, cfg) instead.
ListByTag lists Pods across every namespace (orphan GC needs to find
every tagged Pod regardless of where it landed), which means this
provider's RBAC has to be a ClusterRole rather than namespace-scoped --
flagged now, wired in Step 10.
provider.Config gains KubernetesConfig (replacing MockConfig) and drops
the FailWith*/fault-injection surface entirely, since that need is now
served by a small in-test stub Provider for reconciler tests (Step 4),
not a config-driven mechanism on a real provider package.
Tests use sigs.k8s.io/controller-runtime/pkg/client/fake -- real Pod
objects, the real client.Client interface -- at 77.6% coverage.
make test green across the whole repo.
Co-Authored-By: Claude <noreply@anthropic.com>
The mock provider (state simulated via an injectable clock, a hand-rolled
shared/refcounted CONNECT-proxy listener per port to work around macOS's
loopback restrictions) worked, but the user felt it was too far removed
from the real system to build confidence in, and doesn't need the
automated test suite to stay fast enough to justify that complexity — a
kind-based verification pass "once in a while" is an acceptable trade for
tests that actually look like the final product.
Replacing it with a provider that creates real Pods in the same cluster,
running an actual Squid container. internal/provider/registry was already
designed to have zero dependency on any concrete provider package, so
removing this one required no changes anywhere else in the tree — go
build is clean with nothing implementing provider.Provider yet.
docs/plans/2026-08-07-1747-proxy-operator.md's Step 3 (and every other
reference to the mock provider throughout the plan) is updated in this
same commit to describe the replacement. Narrative on why and the
replacement's design lands in docs/plans-executions once it's built.
Co-Authored-By: Claude <noreply@anthropic.com>
An in-memory provider.Provider whose state (Provisioning -> Running ->
Terminated -> purged) is a pure function of an injectable clock, not
background timers, so it's deterministic under tests and correct under
real time with no goroutine lifecycle to leak.
Once an instance is observed Running, it lazily acquires a real HTTP
CONNECT proxy listener so the health engine's through-the-proxy probe
(later steps) genuinely tunnels a request end to end, instead of the
healthcheck being simulated or bypassed for local development.
Redesigned the listener sharing model from what the plan assumed: the
plan's "one loopback IP per instance" doesn't work on macOS (only
127.0.0.1 binds without a privileged ifconfig alias, unlike Linux where
the whole 127.0.0.0/8 routes to loopback by default), and there's no
channel for a provider to report a port back to the reconciler anyway
(EffectivePort() is spec-only). Instances now share one real listener
per port, reference-counted at the package level rather than per
Provider instance, since a bound TCP port is a genuinely process-global
OS resource -- two separately configured mock-typed provider entries
must not both try to bind the same default port.
Fault injection wired both ways: MockConfig.FailNextCreates/FailWith for
demos, InjectCreateFailures(n, class) for tests. Create is idempotent by
name.
Caught and fixed a real test flake (not a logic bug): the freePort test
helper asked the OS for a free port via bind-then-close, a TOCTOU race
under t.Parallel() that let two tests collide on the same "free" port.
Replaced it with a monotonic counter, since these tests only need
uniqueness within the test run.
internal/provider/mock at 91.1% coverage, including an end-to-end test
that opens real sockets: Create -> Get past provisionDelay -> a real
http.Client tunnelling a CONNECT through the mock to a real TLS origin.
make test green across the whole repo.
Co-Authored-By: Claude <noreply@anthropic.com>
kacerrandClaude Fable 5.1