egress-proxies-operator/CHANGELOG.md

# Changelog

## 2026-09-18 07:52 UTC — docs/operating.md, starting with the libvirt-RPC log query

New home for day-to-day helpers against a running operator, seeded with the `jq` query
that pulls timestamp + rpc out of the manager's JSON logs (both libvirt providers log the
same `"libvirt call"` msg, so one filter covers `kvm` and `kvm2`), plus the `fromjson?`
variant for streams carrying non-JSON lines. Linked from the README's doc index.

## 2026-09-18 07:44 UTC — a kvm2 crawl-pool sample

Until now the only kvm2 manifest was the one-VM `config/samples/proxy_kvm2.yaml`, which
left the multi-proxy shape — the one a crawler actually leases from — undocumented.

- Added `config/samples/proxy_kvm2_crawl_pool.yaml`: five Managed Proxies on the `kvm2-home`
  entry, four sharing one labelled cloud-init Secret and one inline (it runs squid on 8080,
  so it cannot share a Secret that pins `http_port 3128`). Covers the `tier` attribute the
  discovery API selects on, a `medium`/`ubuntu-24.04` member with a raised `maxLeases`, and
  a member with a tightened `healthCheck`.
- Kept out of `config/samples/kustomization.yaml` on purpose so `apply -k` doesn't boot a
  whole pool; the exclusion is commented there next to the existing `providers-config.yaml`
  one. Validated with `kubectl apply --dry-run=server`.

## 2026-09-17 20:36 UTC — kvm2: the four high findings from the code review, and its first run on a real hypervisor

A scoped review of the kvm2 provider (`review/REPORT.md`: 52 findings, 4 high, 20 medium,
28 low) produced four fixes, one commit each, then the provider was deployed in-cluster and
validated against beelink — the first time any of this code had run against real libvirt.

- **H1 — a stalled write no longer delays a connection teardown.** `sshSocket.Close` ran the
  blocking close before the unblocking one, so a transfer stuck in a TCP write pinned the
  teardown for as long as the kernel retransmitted (~15 min). `link.markDead` compounded it by
  killing inside a `sync.Once` before signalling dead, which made even cancelling the reconcile
  useless. With `MaxConcurrentReconciles: 3`, one silent host could stall every Proxy on every
  provider. `internal/provider/kvm2/{ssh,backend}.go`.
- **H2 — a source error no longer reaches go-libvirt's stream sender.** A failing HTTP body
  deadlocked `StorageVolUpload` outright and leaked two goroutines plus 4 MiB per occurrence,
  against a 128Mi pod limit. The defect is upstream (`requestStream` never drains an unbuffered
  `outErr`); the workaround wraps the reader so it can only ever end the stream cleanly, and
  moves the transfer deadline onto the HTTP source so a stalled mirror unwinds instead of
  leaking. A daemon-side failure mid-upload still leaks — documented in *Known gaps* with the
  two-character upstream patch. `internal/provider/kvm2/{backend,image}.go`.
- **H3 — an unreadable address is no longer reported as a missing one.** Any error from both
  address sources read as "no address yet", so a pooled connection killed by an unrelated
  reconcile demoted a healthy proxy to Provisioning, cleared its IP and made the health engine
  drop its counters. One source failing stays "no address yet"; every source failing, or a dead
  connection, is now Transient. `internal/provider/kvm2/get.go`.
- **H4 — the base image cache is no longer shared between provider entries.** kvm2 built
  byte-identical volume names to the `kvm` provider in the same pool, so two entries could share
  one file, a concurrent fetch could boot a guest from a half-written base, and a `sha256` edit
  deleted the base under live overlays. Names now carry the entry
  (`base-<entry>-<key>.<format>`), and a digest mismatch is refused rather than resolved by
  deleting. `internal/provider/kvm2/{volume,image}.go`, `internal/provider/config_kvm.go`.
- **klog and `net/http` errors now join the structured log stream.** client-go's leader election
  wrote `I0917 ...` lines straight to stderr because nothing routed klog into logr, and the
  discovery server's `http.Server` had no `ErrorLog`. The deployment also gained
  `--zap-encoder=json` and `--zap-time-encoding=rfc3339`, so `kubectl logs | jq` works.
  `cmd/main.go`, `internal/discovery/server.go`, `config/manager/manager.yaml`.
- **The `kvm2-home` provider entry is enabled** in `config/manager/providers_config.yaml`.
  Validated end to end: health check Ready, a 433 MB image cached in 8.3 s under the new name
  beside kvm's untouched one, a second Proxy reusing the cache, Squid answering a real CONNECT
  probe, and Delete leaving the base image while removing the domain and its volumes.

## 2026-09-16 11:46 UTC — Provider blueprint split into contract, spec template and appendix

- **`docs/provider-contract.md`** is the generic provider specification: MUST/SHOULD/MAY
  rules with the caller behind each one, 19 behaviour-profile axes with stable codes
  (where providers may differ, such as blocking vs submit-and-return Create, or a
  backend with no provisioning state), error taxonomy, credential hygiene, testing
  obligations, the adding-a-provider checklist and the contract's known limits. It
  names no existing provider.
- **`docs/provider-spec-template.md`** is the fill-in questionnaire a new provider's
  design answers before implementation; entry numbers match the contract's profile
  table.
- **`docs/providers-existing.md`** maps kubernetes, gcp and kvm to every profile and
  keeps their per-provider reference notes, shared gaps and proposed features.
- **`docs/providers.md`** is now an index; README, CLAUDE.md and architecture.md point
  at the new files. This supersedes the single-file blueprint written earlier on the
  same branch.
- Two rules the callers enforce that were not written down before: orphan GC sweeps
  providers sequentially with no deadline, so a hung `ListByTag` stalls all collection;
  and the reconciler deletes an instance a backend reports as Stopped before start.

## 2026-09-16 08:45 UTC — Provider health checks: ProviderHealth CRD, per-provider checks, gauges

- **Each configured provider is now checked on a schedule** and reported in a
  cluster-scoped `ProviderHealth` object (`kubectl get ph`), one per provider, with a
  `Ready` condition, a reason, and per-target verdicts.
- **Checks:** kubernetes lists one object from the API server; GCP fetches a single
  instance page; KVM checks SSH/libvirt reachability and the storage pool on every
  host, in parallel. Failures map to fixed reasons (`Unreachable`,
  `SSHHandshakeFailed`, `PoolNotFound`, `Timeout`, …).
- **The KVM check can't hurt reconciles:** it uses `hostPool.probe`, which gives up on
  a slow RPC without dropping the shared connection, and waiting for a host slot can
  be interrupted.
- **Metrics:** `proxy_operator_provider_target_up`, `_check_duration_seconds` and
  `_check_timestamp_seconds`. The flags are `--provider-check-interval` and
  `--provider-check-timeout` (defaults to `min(15s, interval)`).
- Validated with `make test`, `make test-e2e` and the real-hypervisor KVM run against
  beelink. `make lint` is blocked by a Go 1.27 toolchain/staticcheck panic that has
  nothing to do with this change.
- Key files: `api/v1alpha1/providerhealth_types.go`, `internal/provider/check.go`,
  `internal/provider/kvm/`, `internal/providerhealth/`, `docs/architecture.md`.

## 2026-09-10 22:58 UTC — Code-review fixes on the KVM branch: deletion, validation, lazy SSH, guest agent

Five findings from a review of `feat/kvm-provider`, all fixed on the branch.

- **A Proxy can always finish deleting.** The delete path used to drop the finalizer
  only when `Get` answered not-found, so a permanent error there retried forever —
  a KVM host removed from the config was enough. It now calls `Delete` anyway, since
  that is idempotent and can finish where `Get` cannot, and
  `crawl.example.com/skip-cleanup: "true"` abandons the instance for whatever stays
  stuck. In the KVM provider, deleting over an unconfigured host is a no-op success;
  `Get` still fails permanently there so a typo cannot look like a missing VM.
- **The guest agent ships as a vendor-data script**, superseding the cloud-config
  merge in the entry below. Merging could never be made safe: cloud-init drops
  vendor-data keys the user-data defines, and the strategy cannot be overridden from
  either document. The Proxy's cloud-init now reaches the seed byte for byte in any
  format — Jinja, MIME, gzip, scripts — where the round-trip previously re-typed
  unquoted scalars (`version: 1.10` arrived as `1.1`), dropped comments and reordered
  keys. New `installGuestAgent` setting, default on.
- **A missing `kvm-ssh` Secret no longer crash-loops the manager.** The SSH material
  is read on first use and cached, so only KVM Proxies fail and they recover when the
  Secret appears. A missing file is transient, an unparseable key permanent, and
  startup still logs a warning.
- **Host names and image keys are validated.** Either can end up in a providerID or a
  volume file name, where a slash made the VM unaddressable for good.
- Verified end to end: unit suite, the integration test against a real hypervisor
  (22 s, agent reporting the IP), both in-cluster Proxies Ready in 2 min with 204
  through squid, the guest showing `qemu-guest-agent` active with its cloud-config
  comments intact, and the manager staying up with the Secret pointed away.
- Key files: `internal/provider/kvm/{seed.go,conn.go,kvm.go}`,
  `internal/controller/proxy_controller.go`, `internal/provider/config_kvm.go`,
  `api/v1alpha1/proxy_types.go`.

## 2026-09-10 21:04 UTC — KVM provider passes in-cluster; guest agent merged into cloud-config

- Two KVM Proxies with real cloud-configs reach `Ready` from inside the kind cluster
  (2 min from apply, IP via the guest agent, 204 through squid from the LAN).
- The `qemu-guest-agent` bootstrap is now merged into each Proxy's cloud-config
  (`packages` + `runcmd`) instead of shipped as vendor-data: cloud-init's default
  merger drops vendor-data keys the user-data also defines, so any Proxy with its own
  `packages:` never got the agent and stayed `Provisioning` forever. `vendorData` is
  now plain extra vendor-data with no default; non-cloud-config user-data (scripts,
  `#include`) still gets the bootstrap as vendor-data. A cloud-config that is not a
  YAML mapping fails the Proxy permanently instead of booting without its config.
- The manager pod mounts a 64 MiB `emptyDir` on `/tmp`: the seed ISO writer stages
  files on disk and the root filesystem is read-only.
- Errors already classified with a provider sentinel no longer drop the cached
  libvirt SSH connection; only libvirt and transport errors do.
- Key files: `internal/provider/kvm/{seed.go,conn.go,errors.go}`,
  `config/manager/manager.yaml`, `docs/kvm-provider.md`.

## 2026-09-10 10:07 UTC — KVM provider validated on a real hypervisor; UEFI default; libvirt call tracing

- `make test-integration-kvm` passes end to end against a libvirt 11.6 / QEMU 10.1
  host (Debian 13 image, UEFI, guest agent reporting the IP after ~20 s). Four
  defects found only on the real host, all fixed on the branch:
  - domain disks are now `type="file"` with resolved paths: pool-volume references
    get no AppArmor rule from `virt-aa-helper`, so QEMU was denied the disks;
  - domain metadata is parsed by local name: libvirt returns the element without
    its namespace declaration;
  - guests boot **UEFI by default** (`images.<key>.firmware: uefi|bios`, OVMF now a
    host prerequisite): Debian 13's image hangs in GRUB under SeaBIOS on a headless
    VM with a serial console (Debian bug #1111240, open); bookworm is unaffected;
  - the image cache's raw-upload path is confirmed to produce a byte-identical file.
- Every libvirt RPC is logged at `--zap-log-level=debug` as one `libvirt call` line
  (`rpc=virDomainCreate`, identifiers, duration, `libvirtError` on failure); level 2
  adds XML, metadata and addresses; cloud-init is never logged. The integration test
  shows the trace in `-v` output (`KVM_TEST_LOG_LEVEL`, default 1).
- Key files: `internal/provider/kvm/{domainxml.go,metadata.go,trace.go,conn.go}`,
  `internal/provider/config_kvm.go`, `docs/kvm-provider.md`.

## 2026-09-08 11:00 UTC — KVM/libvirt provider

- New `kvm` provider type: Managed proxies as VMs on your own libvirt/KVM hosts. Each
  host is reached over SSH with the libvirt unix socket forwarded through the
  connection (pure Go, nothing to expose on the hypervisors beyond sshd); one provider
  instance can span several hosts, selected by `placement.zone`.
- Base images (`images:` map or a direct URL in `placement.image`) are downloaded by the
  operator into each host's storage pool on first use, verified by sha256 when configured,
  and cached there. Per-VM disks are qcow2 overlays; cloud-init arrives on a NoCloud seed
  ISO built in-process. GC identity lives in the domain's `<metadata>` element.
- Wiring: `"kvm": kvm.New` in `cmd/main.go`; optional `kvm-ssh` Secret mounted at
  `/etc/proxy-operator/kvm`; samples `config/samples/proxy_kvm.yaml` and a `kvm-home`
  block in `providers-config.yaml`; `docs/kvm-provider.md`; opt-in integration test
  (`make test-integration-kvm`) against a real hypervisor.
- Not validated on a real host at the time of this entry; see 2026-09-10 above.

## 2026-08-10 09:34 CEST — kind e2e verified; fix Squid OOM in containers; quickstart goes in-cluster

- Full end-to-end pass on a throwaway kind cluster: Squid pod Ready with a real CONNECT
  probe (89 ms), lease grant/report/cooldown-409/release through the discovery API,
  finalizer cleanup on delete.
- Fixed the kubernetes provider's generated squid.conf: `max_filedescriptors 1024`
  (squid sizes FD tables from the container's effectively-unlimited RLIMIT_NOFILE and
  was OOM-killed at startup under kind/containerd) + `cache_mem 16 MB`.
- README quickstart now deploys the operator in-cluster: `make run-dev` on a laptop
  cannot reach kind pod IPs, so health probes fail by construction there (documented).

## 2026-08-09 17:27 CEST — Operator wired end to end: reconciler, health, leases, discovery, GC, two providers

- `cmd/main.go` is now the full composition root: `--providers-config` (required, fail-fast),
  `--discovery-addr`, `--proxy-namespace`, `--health-workers`, `--gc-interval`, `--gc-min-age`,
  `--gc-allow-namespaced`, `--lease-cooldown`, `--max-lease-ttl`; wires the kubernetes + gcp
  providers (metrics-instrumented), health engine, lease store, discovery API, orphan GC, and
  Prometheus metrics onto one manager.
- Deploy manifests: providers ConfigMap mount, optional `DISCOVERY_TOKEN` Secret env,
  discovery port 8090 + Service; pods RBAC for the kubernetes provider.
- Samples for all three proxy flavors + providers-config; `make run-dev` for local development.
- README rewritten (kind quickstart, GCP setup, the two load-bearing caveats); architecture doc
  completed with components table and the full decision log.