In the kind cluster two KVM Proxies with real cloud-configs booted fine
but never left Provisioning: the guest agent was not installed, so the
provider could not learn the IP. cloud-init merges vendor-data under
user-data with its default mergers, which keep the user-data's value
for any key both define and never append lists, so a cloud-config with
its own `packages:` silently dropped the vendor-data's packages and
runcmd. The integration test passed only because its user-data was
empty.
withGuestAgent now appends qemu-guest-agent to `packages` and the
enable command to `runcmd` of the Proxy's own cloud-config (skipping
entries already present) and re-renders it. Vendor-data is no longer
defaulted; `vendorData` is plain extra data. User-data that is not a
cloud-config document (script, #include, MIME) still gets the bootstrap
as vendor-data. A cloud-config that is not a YAML mapping, or whose
packages/runcmd is not a list, fails the Proxy as ErrPermanent.
Validated in-cluster: both Proxies Ready and healthy in two minutes, 204
through squid from the LAN. Docs, sample comment, CHANGELOG and the
plan execution log (Run 7) updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The known-gaps entry, decision log, sample comment and builder comment
blamed the host; the cause is trixie's GRUB config ("terminal_output
gfxterm serial" with a serial console and no video device), reported
and reproduced upstream and still open. UEFI remains the default for the
reason recorded; bookworm images are noted as unaffected.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 5 on the real host bisected the guest hang: the Debian 13 image
hangs after GRUB under SeaBIOS with or without our seed and with either
CPU model, and boots under OVMF. The image's legacy boot chain is the
problem, not the provider.
images.<key>.firmware is "uefi" (default) or "bios". The domain XML
emits <os firmware="efi"> with secure boot off, so libvirt selects the
OVMF binary and manages the NVRAM file, which the existing nvram undefine
flag removes. Direct-URL images default to UEFI too. OVMF joins the
hypervisor prerequisites; sample, config table, known gaps, decision log
and the execution log record the finding.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first live run showed the trace working but noisy: testr printed
ts/level/msg and the per-run constants on every line, the error text
appeared twice, host appeared twice on image-cache lines, and the Go
method name was less useful than the libvirt function it maps to.
The tracer now logs rpc=virDomainLookupByName (go-libvirt's names are the
public API names without the vir prefix), error plus libvirtError (code
name) and libvirtCode, and image.go stops re-adding host. The integration
test replaces testr with a compact logr sink: time, logger name, the rpc
as headline, then fields, with provider/host/name omitted since the first
line of the run states them. Docs and execution log updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A tracingAPI decorator wraps the 17-method libvirt seam and logs one
"libvirt call" line per RPC after it returns: method, the identifiers it
was given, curated results, duration, and on failure the error with its
libvirt code, name and message. At V(2) the same line carries the domain
and volume XML, metadata and addresses; upload bodies are never logged.
The pool applies it in call() when V(1) is enabled and logs connect,
dial-failure and drop events, taking the caller's logger explicitly like
every other helper in the package.
The integration test installs a testr logger at KVM_TEST_LOG_LEVEL
(default 1) so the trace interleaves with t.Log output, which also ends
the "log.SetLogger(...) was never called" warning. Docs gained a Logging
section and the test env var; the KVM provider execution log records run
4 (guest hang at GRUB) that prompted this.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 3 on the real host got past Create and failed in Get: libvirt 11.6
selects the <instance> element by our URI in DomainGetMetadata but hands
it back with no namespace declaration, and the parser required the URI
on the element. The lookup already guarantees the namespace, so match on
the local name only. The fake now strips the declaration the way libvirt
does, and a new test pins the bare form. Execution log updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first run against a real hypervisor (beelink, Ubuntu, libvirt 11.6)
failed at DomainCreate with "Could not open base image: Permission denied"
even with correct DAC ownership. libvirt's virt-aa-helper only grants QEMU
access to disk sources that carry a path; the <disk type="volume"> form
chosen in Step 2 gives it pool and volume names instead, so the overlay and
its backing image get no AppArmor rule on any stock Debian/Ubuntu host.
buildDomainXML now takes DiskPath/SeedPath and emits type="file" disks.
ensureOverlay and writeSeed return the volume handles and Create resolves
both paths with StorageVolGetPath before defining the domain. The domain
XML test rejects any volume/pool reference so this cannot drift back; the
Create test asserts the sources equal the paths libvirt reported.
Execution log: Step 2 note marked reverted, Step 7 entry opened with the
two failed runs and the diagnosis.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Registers "kvm" in the provider builtins, mounts an optional kvm-ssh
Secret at /etc/proxy-operator/kvm, adds a kvm-home providers-config
sample and a proxy_kvm.yaml Proxy sample, documents the provider in
docs/kvm-provider.md (prerequisites, config reference, image cache and
its do-not-replace-in-use rule, known gaps), and updates the README
and architecture provider sections and decision log.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Behind -tags=integration and self-skipping unless KVM_TEST_* names a
host: one sequential lifecycle (image fetch, Create, idempotent
re-Create, Running with IP, ListByTag, Delete, pool state afterwards),
driven through the provider directly so it needs no Step 5 wiring.
Adds `make test-integration-kvm` and documents the run and the
deviation from the testcontainers convention in docs/testing.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Create looks the domain up first and only starts it when it exists;
otherwise it caches the image, reuses or creates the overlay, rebuilds
the seed ISO unconditionally (a cancelled upload can leave a truncated
seed that no later attempt would repair), defines and starts the domain.
Get maps libvirt state plus the first routable IPv4 from ARP or the
guest agent onto the provider states. Delete destroys, undefines and
removes the per-VM volumes by name, tolerating every already-gone case.
ListByTag keeps managed domains across all hosts, skipping domains it
cannot read and failing the sweep only for an unreachable host.
Also corrects the Step 2 seed determinism test: the iso9660 writer
stamps the build time into the volume descriptor, so builds differ by a
few bytes; the test now compares label, files and size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
resolveImage maps placement.image to a configured key or URL; imageCache
streams the download through StorageVolUpload with a sha256 tee, writes
a marker volume when the upload is verified, and re-fetches when the
marker is missing or the configured checksum changed. Errors carry
taxonomy sentinels that classify now honours. Adds the in-memory fake
libvirt API used from here on.
Step 3b of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Per-host connection pool over SSH with the libvirt unix socket forwarded
through the session, lazy dial and redial after disconnect, context
cancellation for go-libvirt's context-less RPCs, handshake deadline and
SSH keepalives. Error classification onto the provider taxonomy. Tested
with a fake dialer and an in-process SSH server that forwards unix sockets.
Step 3 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Introduces internal/provider/kvm with the side-effect-free pieces of the
provider: providerID format/parse, the namespaced <metadata> element that
carries the GC contract, q35 domain XML over pool volumes, volume XML for
overlays and raw seeds, the NoCloud seed ISO built with kdomanski/iso9660,
and config defaults. All unit-tested.
Step 2 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds the kvm block to the providers config (SSH access, images map,
machine-type profiles, hosts list) with validation and table tests.
validate() now rejects mismatched type-specific blocks generically via
setBlocks() instead of pairwise checks.
Step 1 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
run-demo.sh opens a 2x2 tmux grid: egress-IP table looping in a
netshoot pod, kubectl get px watch, and both create scripts running
with COUNT (default 4) proxies. show-egress-ips-table.sh is the
pane-sized one-line-per-proxy variant used by the driver.
Co-Authored-By: Claude <noreply@anthropic.com>
docs/demo/show-egress-ips.sh probes each healthy proxy from the
discovery API against an IP-echo site; create-kubernetes-proxies.sh and
create-gcp-proxies.sh bulk-create demo Proxies, the gcp one spreading
them across randomly picked EU zones.
Co-Authored-By: Claude <noreply@anthropic.com>
Full client-facing reference: auth, all routes with schemas and curl
examples, selection/cooldown semantics, caveats. README and
architecture.md link to it.
Co-Authored-By: Claude <noreply@anthropic.com>
Distilled from the house pattern across sibling projects: tag push +
workflow_dispatch triggers, REGISTRY_TOKEN login, raw docker build/push
to gitea.home.hrajfrisbee.cz. Adds a lightweight test gate, an immutable
sha-<12> tag, :latest only on real tag pushes, and a concurrency group.
Co-Authored-By: Claude <noreply@anthropic.com>
The option.WithLogger logger also reaches cloud.google.com/go/auth,
which logged its token exchange at Debug — JWT assertion and bearer
token included. wireLogger now allowlists only the compute client's
api request/response records at Debug (fail-closed for future SDK
additions); Warn/Error pass through. String fields over 1KiB (e.g.
Shielded-VM UEFI dbx blobs) are elided recursively by default; the new
--gcp-wire-log-full-payloads flag restores verbatim payloads.
Co-Authored-By: Claude <noreply@anthropic.com>
Inject an option.WithLogger slog logger bridged to the zap sink via
logr.ToSlogHandler with a V(1) shift, so the SDK's Debug-level
"api request"/"api response" records (URL, headers, full payloads)
appear only at --zap-log-level=5. Startup warning when active, since
raw insert payloads include cloud-init user-data. Note WithLogger
overrides GOOGLE_SDK_GO_LOGGING_LEVEL for this client.
Co-Authored-By: Claude <noreply@anthropic.com>
New internal/version package: ldflags-stamped Commit with a
debug.ReadBuildInfo VCS fallback for host builds. Startup log line
carries commit + Go version; --version prints the hash and exits.
Makefile computes GIT_COMMIT (12 chars, -dirty on any local change) and
passes it to docker-build/buildx; Dockerfile injects it via -ldflags and
an org.opencontainers.image.revision label. make build now uses ./cmd —
file-argument builds skip Go's automatic VCS stamp.
Co-Authored-By: Claude <noreply@anthropic.com>
One V(1) line per GCP API call (insert/get/delete/aggregatedList) with
outcome and operation name, V(2) request/per-instance detail, and raw
googleapi status+reasons logged before classify collapses them. Curated
fields only — cloud-init user-data never reaches logs (test-enforced).
Co-Authored-By: Claude <noreply@anthropic.com>
config/manager/kustomization.yaml: commit the images: stanza that
`kustomize edit set image` (run by make deploy, including inside make
test-e2e) writes into this tracked file. It showed up as unexplained
drift twice; committing it once ends that -- the edit is idempotent, so
future deploy/e2e runs produce no diff. The example.com image name is
the e2e suite's placeholder default and gets overridden by IMG= on any
real deploy.
Execution log: the Status checklist's Step 3 line still said "Mock
provider" from before the pivot; a fresh session resuming from the
checklist alone would have been misled.
Co-Authored-By: Claude <noreply@anthropic.com>
Follows the approved lean-down plan
(docs/plans/2026-08-08-1335-lean-scaffold-cleanup.md). Removes from the
application's deployed footprint:
- config/network-policy/ and its commented enable line -- the user does
not need network policies at the moment.
- The webhook-only halves of config/default/kustomization.yaml: the
commented ../webhook and ../certmanager resource lines, the
manager_webhook_patch.yaml reference, the serving-cert ->
Validating/Mutating WebhookConfiguration cainjection replacement
blocks, and the crdkustomizecainjection* scaffold markers -- anchors
only for `kubebuilder create webhook`, which is a permanent non-goal.
- The two commented [WEBHOOK] blocks in config/crd/kustomization.yaml
plus the now-empty patches: key; kept the one-line
crdkustomizeresource marker since `kubebuilder create api` could
legitimately run again.
- config/crd/kustomizeconfig.yaml, whose only consumer was the removed
configurations: block.
Explicitly kept per user direction: all of config/prometheus/, the
paired metrics-TLS-via-cert-manager plumbing (cert_metrics_manager_patch
+ the metrics-certs/ServiceMonitor replacement halves), all RBAC
manifests including the admin/editor/viewer helper roles, and all
developer tooling.
Also records in the execution log why the webhook machinery existed at
all: kubebuilder init emits it unconditionally, verified against the
v4.15.0 binary that no init flag can suppress it -- scaffold-then-prune
is the only supported path, and the pruning pass should have happened
at Step 0.
Verified: kustomize build clean on config/default and config/crd,
go build/vet clean with and without -tags=e2e, make test green with
coverage identical to pre-cleanup.
Co-Authored-By: Claude <noreply@anthropic.com>
kubebuilder's generic scaffold defensively wires up webhook TLS-cert
machinery and an unconditional cert-manager install in the e2e suite, in
case a project grows admission webhooks later. This one never will --
the spec's non-goals explicitly rule out admission webhooks and
cert-manager wiring -- so none of it does anything. Verified before
removing: no config/webhook/, no +kubebuilder:webhook markers anywhere,
and config/*/kustomization.yaml's [CERTMANAGER] blocks are all inert
(never uncommented).
cmd/main.go: drops the webhook import, the three webhook-cert-* flags,
and the WebhookServer wiring on ctrl.Options -- the manager now runs
with no webhook server, correctly, since nothing registers one. Left
the metrics-cert flags alone; those are unrelated to webhooks.
test/e2e/e2e_suite_test.go: drops the unconditional cert-manager
install/uninstall around the suite.
test/utils/utils.go: drops the now-dead InstallCertManager/
UninstallCertManager/IsCertManagerCRDsInstalled and their warnError
helper, plus UncommentCode -- unrelated to cert-manager, but found to
have zero callers even before this cleanup.
Left the inert commented-out [WEBHOOK]/[CERTMANAGER] kustomize blocks
and kubebuilder's scaffold marker comments alone: pure comments, no
runtime behavior, unlike the cert-manager install this actually removed.
Verified clean with both build tags (go build/vet, and -tags=e2e for
test/e2e). make test unchanged and green.
Co-Authored-By: Claude <noreply@anthropic.com>
Create/Get/Delete/ListByTag against real corev1.Pod objects in the same
cluster the operator runs in, running an ubuntu/squid container -- picked
by actually checking Docker Hub metadata (Canonical-published, rebuilt
the same day this was decided, 50M+ pulls) rather than guessing an image
reference. It's a public image, so kind nodes pull it directly with no
build/load step.
providerID is "<namespace>/<podName>", parsed via
cache.SplitMetaNamespaceKey -- the same self-contained-providerID
reasoning the plan already calls for on the GCP provider's zone-qualified
IDs. Pod state maps to InstanceState with Succeeded/Failed/Unknown all
collapsing to Terminated, since the reconciler already treats Stopped and
Terminated identically; Running-without-PodIP maps to Provisioning so an
empty IP is never published.
The client is built internally via ctrl.GetConfig() (in-cluster or local
kubeconfig, whichever applies), not threaded through the registry
Constructor signature -- this is what lets `make run` against a local
kind cluster and running in-cluster share the exact same code path with
no provider-specific wiring in cmd/main.go. New() is deliberately
untested (0% coverage): it's the one function that must never run under
`go test`, since it would happily connect to whatever cluster the
developer's kubeconfig points at. Tests construct Provider via an
unexported newWithClient(client, cfg) instead.
ListByTag lists Pods across every namespace (orphan GC needs to find
every tagged Pod regardless of where it landed), which means this
provider's RBAC has to be a ClusterRole rather than namespace-scoped --
flagged now, wired in Step 10.
provider.Config gains KubernetesConfig (replacing MockConfig) and drops
the FailWith*/fault-injection surface entirely, since that need is now
served by a small in-test stub Provider for reconciler tests (Step 4),
not a config-driven mechanism on a real provider package.
Tests use sigs.k8s.io/controller-runtime/pkg/client/fake -- real Pod
objects, the real client.Client interface -- at 77.6% coverage.
make test green across the whole repo.
Co-Authored-By: Claude <noreply@anthropic.com>
kacerrandClaude Fable 5.1