Verify end-to-end on kind: fix Squid FD-table OOM, make the quickstart in-cluster

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-08-10 09:34:42 +02:00
parent d595a93d36
commit 0fe62ef314
6 changed files with 129 additions and 11 deletions

View File

@@ -16,7 +16,7 @@ Pairs with [docs/plans/2026-08-07-1747-proxy-operator.md](../plans/2026-08-07-17
- [x] Step 9 — Orphan GC + metrics
- [x] Step 10 — Wiring, config, docs
- [x] Step 11 — Tests
- [ ] Verification (vet/test/kind e2e) + commit, push, open MR
- [x] Verification (vet/test/kind e2e) + commit, push, open MR
## Step 0 — Branch and scaffold
@@ -1061,3 +1061,71 @@ test flips mode and adds an endpoint in the same update to isolate the
immutability rules as the thing that rejects. The plan's remaining
checklist item is Verification: the throwaway-kind-cluster run of the
README quickstart, then push + MR.
## Verification — kind end-to-end
Static checks first (`go vet ./...`, `make build`, full `make test` with
`-race`): all green. Then the real thing, per the spec's §13 "run the kind
quickstart yourself and fix what breaks" — and two things broke, both now
fixed.
**Finding 1 — `make run-dev` cannot produce a Ready proxy on kind.** The
operator on the host provisions the pod fine (Provisioned=True, IP
published), but the health probe originates on the host, and kind pod IPs
(10.244.x.x) are not host-routable — every probe fails by construction
and the proxy latches `Unhealthy`:
```text
Healthy=False: Get "https://www.gstatic.com/generate_204":
proxyconnect tcp: dial tcp 10.244.0.5:3128: connect: connection refused
```
Everything around the failure worked exactly as designed (thresholds,
condition, phase, and the finalizer delete ran clean from the host). Fix:
the README quickstart now deploys the operator **in-cluster**
(docker-build → kind load → deploy → port-forward 8090), with the
run-dev limitation documented in both the quickstart and the Development
section.
**Finding 2 — Squid was OOM-killed at startup in-cluster.** With the
operator deployed in-cluster the pod crash-looped (`OOMKilled`, empty
logs). Root cause: squid sizes its file-descriptor tables from
`RLIMIT_NOFILE`, and containerd under kind sets that effectively
unlimited (~10^9) — squid allocates gigabytes before it ever listens.
Fix in the generated config (`internal/provider/kubernetes/pod.go`):
`max_filedescriptors 1024` (the load-bearing line) plus `cache_mem 16 MB`
(a crawling forward proxy gains nothing from squid's 256 MB default),
with a regression assertion added to `pod_test.go`.
**With both fixes, the full pass:**
```bash
kind create cluster --name proxy-operator-demo
make install
make docker-build IMG=egress-proxies-operator:dev
kind load docker-image egress-proxies-operator:dev --name proxy-operator-demo
make deploy IMG=egress-proxies-operator:dev
kubectl apply -f config/samples/proxy_kubernetes.yaml
# → Ready 10.244.0.9 lat=89ms Provisioned=True Healthy=True (~30 s)
kubectl -n egress-proxies-operator-system port-forward svc/...-discovery-service 8090:8090 &
curl -s 'localhost:8090/v1/proxies?healthy=true' # count:1, latencyMillis:89
curl -s -XPOST localhost:8090/v1/leases -d '{"selector":{"geo":"local"},"ttlSeconds":300}'
# → 201 {leaseID, proxy(activeLeases:1), expiresAt, ttlSeconds:300}
curl -XPOST .../report -d '{"result":"rate_limited","target":"example.com"}' # 204
curl -XPOST /v1/leases -d '{...,"target":"example.com"}' # 409 {inCooldown:1} ✓
curl -XDELETE /v1/leases/<id> # 204, and 204 again ✓
kubectl delete -f config/samples/proxy_kubernetes.yaml # finalizer: pod Terminating, CR gone
kind delete cluster --name proxy-operator-demo
```
A real Squid pod went Ready through a real CONNECT probe, a lease was
held on it, the cooldown machinery answered a 409 with correct
arithmetic, and the finalizer cleaned up — the plan's success bar, met
with the actual product.
Worth noting: `make deploy` runs `kustomize edit set image` and mutates
`config/manager/kustomization.yaml` in the working tree — reverted before
committing (the repo keeps the pinned stanza). The health probe's ~89 ms
latency is gstatic-through-squid from a kind pod on this machine;
metrics-side observations were not separately checked in-cluster (covered
by unit tests).