egress-proxies-operator/CHANGELOG.md

Changelog

2026-09-18 07:44 UTC — a kvm2 crawl-pool sample

Until now the only kvm2 manifest was the one-VM config/samples/proxy_kvm2.yaml, which left the multi-proxy shape — the one a crawler actually leases from — undocumented.

• Added config/samples/proxy_kvm2_crawl_pool.yaml: five Managed Proxies on the kvm2-home entry, four sharing one labelled cloud-init Secret and one inline (it runs squid on 8080, so it cannot share a Secret that pins http_port 3128). Covers the tier attribute the discovery API selects on, a medium/ubuntu-24.04 member with a raised maxLeases, and a member with a tightened healthCheck.
• Kept out of config/samples/kustomization.yaml on purpose so apply -k doesn't boot a whole pool; the exclusion is commented there next to the existing providers-config.yaml one. Validated with kubectl apply --dry-run=server.

2026-09-17 20:36 UTC — kvm2: the four high findings from the code review, and its first run on a real hypervisor

A scoped review of the kvm2 provider (review/REPORT.md: 52 findings, 4 high, 20 medium, 28 low) produced four fixes, one commit each, then the provider was deployed in-cluster and validated against beelink — the first time any of this code had run against real libvirt.

• H1 — a stalled write no longer delays a connection teardown. sshSocket.Close ran the blocking close before the unblocking one, so a transfer stuck in a TCP write pinned the teardown for as long as the kernel retransmitted (~15 min). link.markDead compounded it by killing inside a sync.Once before signalling dead, which made even cancelling the reconcile useless. With MaxConcurrentReconciles: 3, one silent host could stall every Proxy on every provider. internal/provider/kvm2/{ssh,backend}.go.
• H2 — a source error no longer reaches go-libvirt's stream sender. A failing HTTP body deadlocked StorageVolUpload outright and leaked two goroutines plus 4 MiB per occurrence, against a 128Mi pod limit. The defect is upstream (requestStream never drains an unbuffered outErr); the workaround wraps the reader so it can only ever end the stream cleanly, and moves the transfer deadline onto the HTTP source so a stalled mirror unwinds instead of leaking. A daemon-side failure mid-upload still leaks — documented in Known gaps with the two-character upstream patch. internal/provider/kvm2/{backend,image}.go.
• H3 — an unreadable address is no longer reported as a missing one. Any error from both address sources read as "no address yet", so a pooled connection killed by an unrelated reconcile demoted a healthy proxy to Provisioning, cleared its IP and made the health engine drop its counters. One source failing stays "no address yet"; every source failing, or a dead connection, is now Transient. internal/provider/kvm2/get.go.
• H4 — the base image cache is no longer shared between provider entries. kvm2 built byte-identical volume names to the kvm provider in the same pool, so two entries could share one file, a concurrent fetch could boot a guest from a half-written base, and a sha256 edit deleted the base under live overlays. Names now carry the entry (base-<entry>-<key>.<format>), and a digest mismatch is refused rather than resolved by deleting. internal/provider/kvm2/{volume,image}.go, internal/provider/config_kvm.go.
• klog and net/http errors now join the structured log stream. client-go's leader election wrote I0917 ... lines straight to stderr because nothing routed klog into logr, and the discovery server's http.Server had no ErrorLog. The deployment also gained --zap-encoder=json and --zap-time-encoding=rfc3339, so kubectl logs | jq works. cmd/main.go, internal/discovery/server.go, config/manager/manager.yaml.
• The kvm2-home provider entry is enabled in config/manager/providers_config.yaml. Validated end to end: health check Ready, a 433 MB image cached in 8.3 s under the new name beside kvm's untouched one, a second Proxy reusing the cache, Squid answering a real CONNECT probe, and Delete leaving the base image while removing the domain and its volumes.

2026-09-16 11:46 UTC — Provider blueprint split into contract, spec template and appendix

• docs/provider-contract.md is the generic provider specification: MUST/SHOULD/MAY rules with the caller behind each one, 19 behaviour-profile axes with stable codes (where providers may differ, such as blocking vs submit-and-return Create, or a backend with no provisioning state), error taxonomy, credential hygiene, testing obligations, the adding-a-provider checklist and the contract's known limits. It names no existing provider.
• docs/provider-spec-template.md is the fill-in questionnaire a new provider's design answers before implementation; entry numbers match the contract's profile table.
• docs/providers-existing.md maps kubernetes, gcp and kvm to every profile and keeps their per-provider reference notes, shared gaps and proposed features.
• docs/providers.md is now an index; README, CLAUDE.md and architecture.md point at the new files. This supersedes the single-file blueprint written earlier on the same branch.
• Two rules the callers enforce that were not written down before: orphan GC sweeps providers sequentially with no deadline, so a hung ListByTag stalls all collection; and the reconciler deletes an instance a backend reports as Stopped before start.

2026-09-16 08:45 UTC — Provider health checks: ProviderHealth CRD, per-provider checks, gauges

• Each configured provider is now checked on a schedule and reported in a cluster-scoped ProviderHealth object (kubectl get ph), one per provider, with a Ready condition, a reason, and per-target verdicts.
• Checks: kubernetes lists one object from the API server; GCP fetches a single instance page; KVM checks SSH/libvirt reachability and the storage pool on every host, in parallel. Failures map to fixed reasons (Unreachable, SSHHandshakeFailed, PoolNotFound, Timeout, …).
• The KVM check can't hurt reconciles: it uses hostPool.probe, which gives up on a slow RPC without dropping the shared connection, and waiting for a host slot can be interrupted.
• Metrics: proxy_operator_provider_target_up, _check_duration_seconds and _check_timestamp_seconds. The flags are --provider-check-interval and --provider-check-timeout (defaults to min(15s, interval)).
• Validated with make test, make test-e2e and the real-hypervisor KVM run against beelink. make lint is blocked by a Go 1.27 toolchain/staticcheck panic that has nothing to do with this change.
• Key files: api/v1alpha1/providerhealth_types.go, internal/provider/check.go, internal/provider/kvm/, internal/providerhealth/, docs/architecture.md.

2026-09-10 22:58 UTC — Code-review fixes on the KVM branch: deletion, validation, lazy SSH, guest agent

Five findings from a review of feat/kvm-provider, all fixed on the branch.

• A Proxy can always finish deleting. The delete path used to drop the finalizer only when Get answered not-found, so a permanent error there retried forever — a KVM host removed from the config was enough. It now calls Delete anyway, since that is idempotent and can finish where Get cannot, and crawl.example.com/skip-cleanup: "true" abandons the instance for whatever stays stuck. In the KVM provider, deleting over an unconfigured host is a no-op success; Get still fails permanently there so a typo cannot look like a missing VM.
• The guest agent ships as a vendor-data script, superseding the cloud-config merge in the entry below. Merging could never be made safe: cloud-init drops vendor-data keys the user-data defines, and the strategy cannot be overridden from either document. The Proxy's cloud-init now reaches the seed byte for byte in any format — Jinja, MIME, gzip, scripts — where the round-trip previously re-typed unquoted scalars (version: 1.10 arrived as 1.1), dropped comments and reordered keys. New installGuestAgent setting, default on.
• A missing kvm-ssh Secret no longer crash-loops the manager. The SSH material is read on first use and cached, so only KVM Proxies fail and they recover when the Secret appears. A missing file is transient, an unparseable key permanent, and startup still logs a warning.
• Host names and image keys are validated. Either can end up in a providerID or a volume file name, where a slash made the VM unaddressable for good.
• Verified end to end: unit suite, the integration test against a real hypervisor (22 s, agent reporting the IP), both in-cluster Proxies Ready in 2 min with 204 through squid, the guest showing qemu-guest-agent active with its cloud-config comments intact, and the manager staying up with the Secret pointed away.
• Key files: internal/provider/kvm/{seed.go,conn.go,kvm.go}, internal/controller/proxy_controller.go, internal/provider/config_kvm.go, api/v1alpha1/proxy_types.go.

2026-09-10 21:04 UTC — KVM provider passes in-cluster; guest agent merged into cloud-config

• Two KVM Proxies with real cloud-configs reach Ready from inside the kind cluster (2 min from apply, IP via the guest agent, 204 through squid from the LAN).
• The qemu-guest-agent bootstrap is now merged into each Proxy's cloud-config (packages + runcmd) instead of shipped as vendor-data: cloud-init's default merger drops vendor-data keys the user-data also defines, so any Proxy with its own packages: never got the agent and stayed Provisioning forever. vendorData is now plain extra vendor-data with no default; non-cloud-config user-data (scripts, #include) still gets the bootstrap as vendor-data. A cloud-config that is not a YAML mapping fails the Proxy permanently instead of booting without its config.
• The manager pod mounts a 64 MiB emptyDir on /tmp: the seed ISO writer stages files on disk and the root filesystem is read-only.
• Errors already classified with a provider sentinel no longer drop the cached libvirt SSH connection; only libvirt and transport errors do.
• Key files: internal/provider/kvm/{seed.go,conn.go,errors.go}, config/manager/manager.yaml, docs/kvm-provider.md.

2026-09-10 10:07 UTC — KVM provider validated on a real hypervisor; UEFI default; libvirt call tracing

• make test-integration-kvm passes end to end against a libvirt 11.6 / QEMU 10.1 host (Debian 13 image, UEFI, guest agent reporting the IP after ~20 s). Four defects found only on the real host, all fixed on the branch:
  • domain disks are now type="file" with resolved paths: pool-volume references get no AppArmor rule from virt-aa-helper, so QEMU was denied the disks;
  • domain metadata is parsed by local name: libvirt returns the element without its namespace declaration;
  • guests boot UEFI by default (images.<key>.firmware: uefi|bios, OVMF now a host prerequisite): Debian 13's image hangs in GRUB under SeaBIOS on a headless VM with a serial console (Debian bug #1111240, open); bookworm is unaffected;
  • the image cache's raw-upload path is confirmed to produce a byte-identical file.
• Every libvirt RPC is logged at --zap-log-level=debug as one libvirt call line (rpc=virDomainCreate, identifiers, duration, libvirtError on failure); level 2 adds XML, metadata and addresses; cloud-init is never logged. The integration test shows the trace in -v output (KVM_TEST_LOG_LEVEL, default 1).
• Key files: internal/provider/kvm/{domainxml.go,metadata.go,trace.go,conn.go}, internal/provider/config_kvm.go, docs/kvm-provider.md.

2026-09-08 11:00 UTC — KVM/libvirt provider

• New kvm provider type: Managed proxies as VMs on your own libvirt/KVM hosts. Each host is reached over SSH with the libvirt unix socket forwarded through the connection (pure Go, nothing to expose on the hypervisors beyond sshd); one provider instance can span several hosts, selected by placement.zone.
• Base images (images: map or a direct URL in placement.image) are downloaded by the operator into each host's storage pool on first use, verified by sha256 when configured, and cached there. Per-VM disks are qcow2 overlays; cloud-init arrives on a NoCloud seed ISO built in-process. GC identity lives in the domain's <metadata> element.
• Wiring: "kvm": kvm.New in cmd/main.go; optional kvm-ssh Secret mounted at /etc/proxy-operator/kvm; samples config/samples/proxy_kvm.yaml and a kvm-home block in providers-config.yaml; docs/kvm-provider.md; opt-in integration test (make test-integration-kvm) against a real hypervisor.
• Not validated on a real host at the time of this entry; see 2026-09-10 above.

2026-08-10 09:34 CEST — kind e2e verified; fix Squid OOM in containers; quickstart goes in-cluster

• Full end-to-end pass on a throwaway kind cluster: Squid pod Ready with a real CONNECT probe (89 ms), lease grant/report/cooldown-409/release through the discovery API, finalizer cleanup on delete.
• Fixed the kubernetes provider's generated squid.conf: max_filedescriptors 1024 (squid sizes FD tables from the container's effectively-unlimited RLIMIT_NOFILE and was OOM-killed at startup under kind/containerd) + cache_mem 16 MB.
• README quickstart now deploys the operator in-cluster: make run-dev on a laptop cannot reach kind pod IPs, so health probes fail by construction there (documented).

2026-08-09 17:27 CEST — Operator wired end to end: reconciler, health, leases, discovery, GC, two providers

• cmd/main.go is now the full composition root: --providers-config (required, fail-fast), --discovery-addr, --proxy-namespace, --health-workers, --gc-interval, --gc-min-age, --gc-allow-namespaced, --lease-cooldown, --max-lease-ttl; wires the kubernetes + gcp providers (metrics-instrumented), health engine, lease store, discovery API, orphan GC, and Prometheus metrics onto one manager.
• Deploy manifests: providers ConfigMap mount, optional DISCOVERY_TOKEN Secret env, discovery port 8090 + Service; pods RBAC for the kubernetes provider.
• Samples for all three proxy flavors + providers-config; make run-dev for local development.
• README rewritten (kind quickstart, GCP setup, the two load-bearing caveats); architecture doc completed with components table and the full decision log.