egress-proxies-operator/CHANGELOG.md

# Changelog

## 2026-09-16 08:45 UTC — Provider health checks: ProviderHealth CRD, per-provider checks, gauges

- **Each configured provider is now checked on a schedule** and reported in a
  cluster-scoped `ProviderHealth` object (`kubectl get ph`), one per provider, with a
  `Ready` condition, a reason, and per-target verdicts.
- **Checks:** kubernetes lists one object from the API server; GCP fetches a single
  instance page; KVM checks SSH/libvirt reachability and the storage pool on every
  host, in parallel. Failures map to fixed reasons (`Unreachable`,
  `SSHHandshakeFailed`, `PoolNotFound`, `Timeout`, …).
- **The KVM check can't hurt reconciles:** it uses `hostPool.probe`, which gives up on
  a slow RPC without dropping the shared connection, and waiting for a host slot can
  be interrupted.
- **Metrics:** `proxy_operator_provider_target_up`, `_check_duration_seconds` and
  `_check_timestamp_seconds`. The flags are `--provider-check-interval` and
  `--provider-check-timeout` (defaults to `min(15s, interval)`).
- Validated with `make test`, `make test-e2e` and the real-hypervisor KVM run against
  beelink. `make lint` is blocked by a Go 1.27 toolchain/staticcheck panic that has
  nothing to do with this change.
- Key files: `api/v1alpha1/providerhealth_types.go`, `internal/provider/check.go`,
  `internal/provider/kvm/`, `internal/providerhealth/`, `docs/architecture.md`.

## 2026-09-10 22:58 UTC — Code-review fixes on the KVM branch: deletion, validation, lazy SSH, guest agent

Five findings from a review of `feat/kvm-provider`, all fixed on the branch.

- **A Proxy can always finish deleting.** The delete path used to drop the finalizer
  only when `Get` answered not-found, so a permanent error there retried forever —
  a KVM host removed from the config was enough. It now calls `Delete` anyway, since
  that is idempotent and can finish where `Get` cannot, and
  `crawl.example.com/skip-cleanup: "true"` abandons the instance for whatever stays
  stuck. In the KVM provider, deleting over an unconfigured host is a no-op success;
  `Get` still fails permanently there so a typo cannot look like a missing VM.
- **The guest agent ships as a vendor-data script**, superseding the cloud-config
  merge in the entry below. Merging could never be made safe: cloud-init drops
  vendor-data keys the user-data defines, and the strategy cannot be overridden from
  either document. The Proxy's cloud-init now reaches the seed byte for byte in any
  format — Jinja, MIME, gzip, scripts — where the round-trip previously re-typed
  unquoted scalars (`version: 1.10` arrived as `1.1`), dropped comments and reordered
  keys. New `installGuestAgent` setting, default on.
- **A missing `kvm-ssh` Secret no longer crash-loops the manager.** The SSH material
  is read on first use and cached, so only KVM Proxies fail and they recover when the
  Secret appears. A missing file is transient, an unparseable key permanent, and
  startup still logs a warning.
- **Host names and image keys are validated.** Either can end up in a providerID or a
  volume file name, where a slash made the VM unaddressable for good.
- Verified end to end: unit suite, the integration test against a real hypervisor
  (22 s, agent reporting the IP), both in-cluster Proxies Ready in 2 min with 204
  through squid, the guest showing `qemu-guest-agent` active with its cloud-config
  comments intact, and the manager staying up with the Secret pointed away.
- Key files: `internal/provider/kvm/{seed.go,conn.go,kvm.go}`,
  `internal/controller/proxy_controller.go`, `internal/provider/config_kvm.go`,
  `api/v1alpha1/proxy_types.go`.

## 2026-09-10 21:04 UTC — KVM provider passes in-cluster; guest agent merged into cloud-config

- Two KVM Proxies with real cloud-configs reach `Ready` from inside the kind cluster
  (2 min from apply, IP via the guest agent, 204 through squid from the LAN).
- The `qemu-guest-agent` bootstrap is now merged into each Proxy's cloud-config
  (`packages` + `runcmd`) instead of shipped as vendor-data: cloud-init's default
  merger drops vendor-data keys the user-data also defines, so any Proxy with its own
  `packages:` never got the agent and stayed `Provisioning` forever. `vendorData` is
  now plain extra vendor-data with no default; non-cloud-config user-data (scripts,
  `#include`) still gets the bootstrap as vendor-data. A cloud-config that is not a
  YAML mapping fails the Proxy permanently instead of booting without its config.
- The manager pod mounts a 64 MiB `emptyDir` on `/tmp`: the seed ISO writer stages
  files on disk and the root filesystem is read-only.
- Errors already classified with a provider sentinel no longer drop the cached
  libvirt SSH connection; only libvirt and transport errors do.
- Key files: `internal/provider/kvm/{seed.go,conn.go,errors.go}`,
  `config/manager/manager.yaml`, `docs/kvm-provider.md`.

## 2026-09-10 10:07 UTC — KVM provider validated on a real hypervisor; UEFI default; libvirt call tracing

- `make test-integration-kvm` passes end to end against a libvirt 11.6 / QEMU 10.1
  host (Debian 13 image, UEFI, guest agent reporting the IP after ~20 s). Four
  defects found only on the real host, all fixed on the branch:
  - domain disks are now `type="file"` with resolved paths: pool-volume references
    get no AppArmor rule from `virt-aa-helper`, so QEMU was denied the disks;
  - domain metadata is parsed by local name: libvirt returns the element without
    its namespace declaration;
  - guests boot **UEFI by default** (`images.<key>.firmware: uefi|bios`, OVMF now a
    host prerequisite): Debian 13's image hangs in GRUB under SeaBIOS on a headless
    VM with a serial console (Debian bug #1111240, open); bookworm is unaffected;
  - the image cache's raw-upload path is confirmed to produce a byte-identical file.
- Every libvirt RPC is logged at `--zap-log-level=debug` as one `libvirt call` line
  (`rpc=virDomainCreate`, identifiers, duration, `libvirtError` on failure); level 2
  adds XML, metadata and addresses; cloud-init is never logged. The integration test
  shows the trace in `-v` output (`KVM_TEST_LOG_LEVEL`, default 1).
- Key files: `internal/provider/kvm/{domainxml.go,metadata.go,trace.go,conn.go}`,
  `internal/provider/config_kvm.go`, `docs/kvm-provider.md`.

## 2026-09-08 11:00 UTC — KVM/libvirt provider

- New `kvm` provider type: Managed proxies as VMs on your own libvirt/KVM hosts. Each
  host is reached over SSH with the libvirt unix socket forwarded through the
  connection (pure Go, nothing to expose on the hypervisors beyond sshd); one provider
  instance can span several hosts, selected by `placement.zone`.
- Base images (`images:` map or a direct URL in `placement.image`) are downloaded by the
  operator into each host's storage pool on first use, verified by sha256 when configured,
  and cached there. Per-VM disks are qcow2 overlays; cloud-init arrives on a NoCloud seed
  ISO built in-process. GC identity lives in the domain's `<metadata>` element.
- Wiring: `"kvm": kvm.New` in `cmd/main.go`; optional `kvm-ssh` Secret mounted at
  `/etc/proxy-operator/kvm`; samples `config/samples/proxy_kvm.yaml` and a `kvm-home`
  block in `providers-config.yaml`; `docs/kvm-provider.md`; opt-in integration test
  (`make test-integration-kvm`) against a real hypervisor.
- Not validated on a real host at the time of this entry; see 2026-09-10 above.

## 2026-08-10 09:34 CEST — kind e2e verified; fix Squid OOM in containers; quickstart goes in-cluster

- Full end-to-end pass on a throwaway kind cluster: Squid pod Ready with a real CONNECT
  probe (89 ms), lease grant/report/cooldown-409/release through the discovery API,
  finalizer cleanup on delete.
- Fixed the kubernetes provider's generated squid.conf: `max_filedescriptors 1024`
  (squid sizes FD tables from the container's effectively-unlimited RLIMIT_NOFILE and
  was OOM-killed at startup under kind/containerd) + `cache_mem 16 MB`.
- README quickstart now deploys the operator in-cluster: `make run-dev` on a laptop
  cannot reach kind pod IPs, so health probes fail by construction there (documented).

## 2026-08-09 17:27 CEST — Operator wired end to end: reconciler, health, leases, discovery, GC, two providers

- `cmd/main.go` is now the full composition root: `--providers-config` (required, fail-fast),
  `--discovery-addr`, `--proxy-namespace`, `--health-workers`, `--gc-interval`, `--gc-min-age`,
  `--gc-allow-namespaced`, `--lease-cooldown`, `--max-lease-ttl`; wires the kubernetes + gcp
  providers (metrics-instrumented), health engine, lease store, discovery API, orphan GC, and
  Prometheus metrics onto one manager.
- Deploy manifests: providers ConfigMap mount, optional `DISCOVERY_TOKEN` Secret env,
  discovery port 8090 + Service; pods RBAC for the kubernetes provider.
- Samples for all three proxy flavors + providers-config; `make run-dev` for local development.
- README rewritten (kind quickstart, GCP setup, the two load-bearing caveats); architecture doc
  completed with components table and the full decision log.