egress-proxies-operator/CHANGELOG.md

Changelog

2026-09-24 10:28 UTC — proxmox provider: reviewed, fixed, proven on a real Proxmox VE 9.2 host

The proxmox provider (VMs cloned from a template on Proxmox VE through its REST API) finished its last phase: a two-pass review, a real-host run on pv1-1, and captures of the three API answers the design had only guessed.

• Delete now removes the seed ISO before destroying the VM. Before, a seed delete that failed after the destroy was never retried (the next Get is NotFound), so the ISO leaked. A 403 for Datastore.Allocate now fails Delete on every path instead of being skipped.
• Create skips a probe VMID held by someone else's VM at once, instead of waiting up to 10 minutes on that VM's lock on every retry.
• The captures corrected two assumptions: a missing ISO storage is a 400 (the health check now says StorageNotFound, not APIError), and a hibernated VM can be destroyed (Delete destroys ours instead of latching the Proxy).
• TestProxmox_lifecycle and TestProxmoxCheck_realHost passed on PVE 9.2.20: Create 6.2 s, an address 50 s later. A kind pass took a sample Proxy to Ready and back.
• Review record: docs/reviews/2026-09-23-proxmox-provider-review/REPORT.md. Timings: docs/testing.md. Code: internal/provider/proxmox/{delete,create,check,errors}.go.

2026-09-18 07:52 UTC — docs/operating.md, starting with the libvirt-RPC log query

New home for day-to-day helpers against a running operator, seeded with the jq query that pulls timestamp + rpc out of the manager's JSON logs (both libvirt providers log the same "libvirt call" msg, so one filter covers kvm and kvm2), plus the fromjson? variant for streams carrying non-JSON lines. Linked from the README's doc index.

2026-09-18 07:44 UTC — a kvm2 crawl-pool sample

Until now the only kvm2 manifest was the one-VM config/samples/proxy_kvm2.yaml, which left the multi-proxy shape — the one a crawler actually leases from — undocumented.

• Added config/samples/proxy_kvm2_crawl_pool.yaml: five Managed Proxies on the kvm2-home entry, four sharing one labelled cloud-init Secret and one inline (it runs squid on 8080, so it cannot share a Secret that pins http_port 3128). Covers the tier attribute the discovery API selects on, a medium/ubuntu-24.04 member with a raised maxLeases, and a member with a tightened healthCheck.
• Kept out of config/samples/kustomization.yaml on purpose so apply -k doesn't boot a whole pool; the exclusion is commented there next to the existing providers-config.yaml one. Validated with kubectl apply --dry-run=server.

2026-09-17 20:36 UTC — kvm2: the four high findings from the code review, and its first run on a real hypervisor

A scoped review of the kvm2 provider (review/REPORT.md: 52 findings, 4 high, 20 medium, 28 low) produced four fixes, one commit each, then the provider was deployed in-cluster and validated against beelink — the first time any of this code had run against real libvirt.

• H1 — a stalled write no longer delays a connection teardown. sshSocket.Close ran the blocking close before the unblocking one, so a transfer stuck in a TCP write pinned the teardown for as long as the kernel retransmitted (~15 min). link.markDead compounded it by killing inside a sync.Once before signalling dead, which made even cancelling the reconcile useless. With MaxConcurrentReconciles: 3, one silent host could stall every Proxy on every provider. internal/provider/kvm2/{ssh,backend}.go.
• H2 — a source error no longer reaches go-libvirt's stream sender. A failing HTTP body deadlocked StorageVolUpload outright and leaked two goroutines plus 4 MiB per occurrence, against a 128Mi pod limit. The defect is upstream (requestStream never drains an unbuffered outErr); the workaround wraps the reader so it can only ever end the stream cleanly, and moves the transfer deadline onto the HTTP source so a stalled mirror unwinds instead of leaking. A daemon-side failure mid-upload still leaks — documented in Known gaps with the two-character upstream patch. internal/provider/kvm2/{backend,image}.go.
• H3 — an unreadable address is no longer reported as a missing one. Any error from both address sources read as "no address yet", so a pooled connection killed by an unrelated reconcile demoted a healthy proxy to Provisioning, cleared its IP and made the health engine drop its counters. One source failing stays "no address yet"; every source failing, or a dead connection, is now Transient. internal/provider/kvm2/get.go.
• H4 — the base image cache is no longer shared between provider entries. kvm2 built byte-identical volume names to the kvm provider in the same pool, so two entries could share one file, a concurrent fetch could boot a guest from a half-written base, and a sha256 edit deleted the base under live overlays. Names now carry the entry (base-<entry>-<key>.<format>), and a digest mismatch is refused rather than resolved by deleting. internal/provider/kvm2/{volume,image}.go, internal/provider/config_kvm.go.
• klog and net/http errors now join the structured log stream. client-go's leader election wrote I0917 ... lines straight to stderr because nothing routed klog into logr, and the discovery server's http.Server had no ErrorLog. The deployment also gained --zap-encoder=json and --zap-time-encoding=rfc3339, so kubectl logs | jq works. cmd/main.go, internal/discovery/server.go, config/manager/manager.yaml.
• The kvm2-home provider entry is enabled in config/manager/providers_config.yaml. Validated end to end: health check Ready, a 433 MB image cached in 8.3 s under the new name beside kvm's untouched one, a second Proxy reusing the cache, Squid answering a real CONNECT probe, and Delete leaving the base image while removing the domain and its volumes.

2026-09-16 11:46 UTC — Provider blueprint split into contract, spec template and appendix

• docs/provider-contract.md is the generic provider specification: MUST/SHOULD/MAY rules with the caller behind each one, 19 behaviour-profile axes with stable codes (where providers may differ, such as blocking vs submit-and-return Create, or a backend with no provisioning state), error taxonomy, credential hygiene, testing obligations, the adding-a-provider checklist and the contract's known limits. It names no existing provider.
• docs/provider-spec-template.md is the fill-in questionnaire a new provider's design answers before implementation; entry numbers match the contract's profile table.
• docs/providers-existing.md maps kubernetes, gcp and kvm to every profile and keeps their per-provider reference notes, shared gaps and proposed features.
• docs/providers.md is now an index; README, CLAUDE.md and architecture.md point at the new files. This supersedes the single-file blueprint written earlier on the same branch.
• Two rules the callers enforce that were not written down before: orphan GC sweeps providers sequentially with no deadline, so a hung ListByTag stalls all collection; and the reconciler deletes an instance a backend reports as Stopped before start.

2026-09-16 08:45 UTC — Provider health checks: ProviderHealth CRD, per-provider checks, gauges

• Each configured provider is now checked on a schedule and reported in a cluster-scoped ProviderHealth object (kubectl get ph), one per provider, with a Ready condition, a reason, and per-target verdicts.
• Checks: kubernetes lists one object from the API server; GCP fetches a single instance page; KVM checks SSH/libvirt reachability and the storage pool on every host, in parallel. Failures map to fixed reasons (Unreachable, SSHHandshakeFailed, PoolNotFound, Timeout, …).
• The KVM check can't hurt reconciles: it uses hostPool.probe, which gives up on a slow RPC without dropping the shared connection, and waiting for a host slot can be interrupted.
• Metrics: proxy_operator_provider_target_up, _check_duration_seconds and _check_timestamp_seconds. The flags are --provider-check-interval and --provider-check-timeout (defaults to min(15s, interval)).
• Validated with make test, make test-e2e and the real-hypervisor KVM run against beelink. make lint is blocked by a Go 1.27 toolchain/staticcheck panic that has nothing to do with this change.
• Key files: api/v1alpha1/providerhealth_types.go, internal/provider/check.go, internal/provider/kvm/, internal/providerhealth/, docs/architecture.md.

2026-09-10 22:58 UTC — Code-review fixes on the KVM branch: deletion, validation, lazy SSH, guest agent

Five findings from a review of feat/kvm-provider, all fixed on the branch.

• A Proxy can always finish deleting. The delete path used to drop the finalizer only when Get answered not-found, so a permanent error there retried forever — a KVM host removed from the config was enough. It now calls Delete anyway, since that is idempotent and can finish where Get cannot, and crawl.example.com/skip-cleanup: "true" abandons the instance for whatever stays stuck. In the KVM provider, deleting over an unconfigured host is a no-op success; Get still fails permanently there so a typo cannot look like a missing VM.
• The guest agent ships as a vendor-data script, superseding the cloud-config merge in the entry below. Merging could never be made safe: cloud-init drops vendor-data keys the user-data defines, and the strategy cannot be overridden from either document. The Proxy's cloud-init now reaches the seed byte for byte in any format — Jinja, MIME, gzip, scripts — where the round-trip previously re-typed unquoted scalars (version: 1.10 arrived as 1.1), dropped comments and reordered keys. New installGuestAgent setting, default on.
• A missing kvm-ssh Secret no longer crash-loops the manager. The SSH material is read on first use and cached, so only KVM Proxies fail and they recover when the Secret appears. A missing file is transient, an unparseable key permanent, and startup still logs a warning.
• Host names and image keys are validated. Either can end up in a providerID or a volume file name, where a slash made the VM unaddressable for good.
• Verified end to end: unit suite, the integration test against a real hypervisor (22 s, agent reporting the IP), both in-cluster Proxies Ready in 2 min with 204 through squid, the guest showing qemu-guest-agent active with its cloud-config comments intact, and the manager staying up with the Secret pointed away.
• Key files: internal/provider/kvm/{seed.go,conn.go,kvm.go}, internal/controller/proxy_controller.go, internal/provider/config_kvm.go, api/v1alpha1/proxy_types.go.

2026-09-10 21:04 UTC — KVM provider passes in-cluster; guest agent merged into cloud-config

• Two KVM Proxies with real cloud-configs reach Ready from inside the kind cluster (2 min from apply, IP via the guest agent, 204 through squid from the LAN).
• The qemu-guest-agent bootstrap is now merged into each Proxy's cloud-config (packages + runcmd) instead of shipped as vendor-data: cloud-init's default merger drops vendor-data keys the user-data also defines, so any Proxy with its own packages: never got the agent and stayed Provisioning forever. vendorData is now plain extra vendor-data with no default; non-cloud-config user-data (scripts, #include) still gets the bootstrap as vendor-data. A cloud-config that is not a YAML mapping fails the Proxy permanently instead of booting without its config.
• The manager pod mounts a 64 MiB emptyDir on /tmp: the seed ISO writer stages files on disk and the root filesystem is read-only.
• Errors already classified with a provider sentinel no longer drop the cached libvirt SSH connection; only libvirt and transport errors do.
• Key files: internal/provider/kvm/{seed.go,conn.go,errors.go}, config/manager/manager.yaml, docs/kvm-provider.md.

2026-09-10 10:07 UTC — KVM provider validated on a real hypervisor; UEFI default; libvirt call tracing

• make test-integration-kvm passes end to end against a libvirt 11.6 / QEMU 10.1 host (Debian 13 image, UEFI, guest agent reporting the IP after ~20 s). Four defects found only on the real host, all fixed on the branch:
  • domain disks are now type="file" with resolved paths: pool-volume references get no AppArmor rule from virt-aa-helper, so QEMU was denied the disks;
  • domain metadata is parsed by local name: libvirt returns the element without its namespace declaration;
  • guests boot UEFI by default (images.<key>.firmware: uefi|bios, OVMF now a host prerequisite): Debian 13's image hangs in GRUB under SeaBIOS on a headless VM with a serial console (Debian bug #1111240, open); bookworm is unaffected;
  • the image cache's raw-upload path is confirmed to produce a byte-identical file.
• Every libvirt RPC is logged at --zap-log-level=debug as one libvirt call line (rpc=virDomainCreate, identifiers, duration, libvirtError on failure); level 2 adds XML, metadata and addresses; cloud-init is never logged. The integration test shows the trace in -v output (KVM_TEST_LOG_LEVEL, default 1).
• Key files: internal/provider/kvm/{domainxml.go,metadata.go,trace.go,conn.go}, internal/provider/config_kvm.go, docs/kvm-provider.md.

2026-09-08 11:00 UTC — KVM/libvirt provider

• New kvm provider type: Managed proxies as VMs on your own libvirt/KVM hosts. Each host is reached over SSH with the libvirt unix socket forwarded through the connection (pure Go, nothing to expose on the hypervisors beyond sshd); one provider instance can span several hosts, selected by placement.zone.
• Base images (images: map or a direct URL in placement.image) are downloaded by the operator into each host's storage pool on first use, verified by sha256 when configured, and cached there. Per-VM disks are qcow2 overlays; cloud-init arrives on a NoCloud seed ISO built in-process. GC identity lives in the domain's <metadata> element.
• Wiring: "kvm": kvm.New in cmd/main.go; optional kvm-ssh Secret mounted at /etc/proxy-operator/kvm; samples config/samples/proxy_kvm.yaml and a kvm-home block in providers-config.yaml; docs/kvm-provider.md; opt-in integration test (make test-integration-kvm) against a real hypervisor.
• Not validated on a real host at the time of this entry; see 2026-09-10 above.

2026-08-10 09:34 CEST — kind e2e verified; fix Squid OOM in containers; quickstart goes in-cluster

• Full end-to-end pass on a throwaway kind cluster: Squid pod Ready with a real CONNECT probe (89 ms), lease grant/report/cooldown-409/release through the discovery API, finalizer cleanup on delete.
• Fixed the kubernetes provider's generated squid.conf: max_filedescriptors 1024 (squid sizes FD tables from the container's effectively-unlimited RLIMIT_NOFILE and was OOM-killed at startup under kind/containerd) + cache_mem 16 MB.
• README quickstart now deploys the operator in-cluster: make run-dev on a laptop cannot reach kind pod IPs, so health probes fail by construction there (documented).

2026-08-09 17:27 CEST — Operator wired end to end: reconciler, health, leases, discovery, GC, two providers

• cmd/main.go is now the full composition root: --providers-config (required, fail-fast), --discovery-addr, --proxy-namespace, --health-workers, --gc-interval, --gc-min-age, --gc-allow-namespaced, --lease-cooldown, --max-lease-ttl; wires the kubernetes + gcp providers (metrics-instrumented), health engine, lease store, discovery API, orphan GC, and Prometheus metrics onto one manager.
• Deploy manifests: providers ConfigMap mount, optional DISCOVERY_TOKEN Secret env, discovery port 8090 + Service; pods RBAC for the kubernetes provider.
• Samples for all three proxy flavors + providers-config; make run-dev for local development.
• README rewritten (kind quickstart, GCP setup, the two load-bearing caveats); architecture doc completed with components table and the full decision log.