Verify end-to-end on kind: fix Squid FD-table OOM, make the quickstart in-cluster
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -16,7 +16,7 @@ Pairs with [docs/plans/2026-08-07-1747-proxy-operator.md](../plans/2026-08-07-17
|
||||
- [x] Step 9 — Orphan GC + metrics
|
||||
- [x] Step 10 — Wiring, config, docs
|
||||
- [x] Step 11 — Tests
|
||||
- [ ] Verification (vet/test/kind e2e) + commit, push, open MR
|
||||
- [x] Verification (vet/test/kind e2e) + commit, push, open MR
|
||||
|
||||
## Step 0 — Branch and scaffold
|
||||
|
||||
@@ -1061,3 +1061,71 @@ test flips mode and adds an endpoint in the same update to isolate the
|
||||
immutability rules as the thing that rejects. The plan's remaining
|
||||
checklist item is Verification: the throwaway-kind-cluster run of the
|
||||
README quickstart, then push + MR.
|
||||
|
||||
## Verification — kind end-to-end
|
||||
|
||||
Static checks first (`go vet ./...`, `make build`, full `make test` with
|
||||
`-race`): all green. Then the real thing, per the spec's §13 "run the kind
|
||||
quickstart yourself and fix what breaks" — and two things broke, both now
|
||||
fixed.
|
||||
|
||||
**Finding 1 — `make run-dev` cannot produce a Ready proxy on kind.** The
|
||||
operator on the host provisions the pod fine (Provisioned=True, IP
|
||||
published), but the health probe originates on the host, and kind pod IPs
|
||||
(10.244.x.x) are not host-routable — every probe fails by construction
|
||||
and the proxy latches `Unhealthy`:
|
||||
|
||||
```text
|
||||
Healthy=False: Get "https://www.gstatic.com/generate_204":
|
||||
proxyconnect tcp: dial tcp 10.244.0.5:3128: connect: connection refused
|
||||
```
|
||||
|
||||
Everything around the failure worked exactly as designed (thresholds,
|
||||
condition, phase, and the finalizer delete ran clean from the host). Fix:
|
||||
the README quickstart now deploys the operator **in-cluster**
|
||||
(docker-build → kind load → deploy → port-forward 8090), with the
|
||||
run-dev limitation documented in both the quickstart and the Development
|
||||
section.
|
||||
|
||||
**Finding 2 — Squid was OOM-killed at startup in-cluster.** With the
|
||||
operator deployed in-cluster the pod crash-looped (`OOMKilled`, empty
|
||||
logs). Root cause: squid sizes its file-descriptor tables from
|
||||
`RLIMIT_NOFILE`, and containerd under kind sets that effectively
|
||||
unlimited (~10^9) — squid allocates gigabytes before it ever listens.
|
||||
Fix in the generated config (`internal/provider/kubernetes/pod.go`):
|
||||
`max_filedescriptors 1024` (the load-bearing line) plus `cache_mem 16 MB`
|
||||
(a crawling forward proxy gains nothing from squid's 256 MB default),
|
||||
with a regression assertion added to `pod_test.go`.
|
||||
|
||||
**With both fixes, the full pass:**
|
||||
|
||||
```bash
|
||||
kind create cluster --name proxy-operator-demo
|
||||
make install
|
||||
make docker-build IMG=egress-proxies-operator:dev
|
||||
kind load docker-image egress-proxies-operator:dev --name proxy-operator-demo
|
||||
make deploy IMG=egress-proxies-operator:dev
|
||||
kubectl apply -f config/samples/proxy_kubernetes.yaml
|
||||
# → Ready 10.244.0.9 lat=89ms Provisioned=True Healthy=True (~30 s)
|
||||
kubectl -n egress-proxies-operator-system port-forward svc/...-discovery-service 8090:8090 &
|
||||
curl -s 'localhost:8090/v1/proxies?healthy=true' # count:1, latencyMillis:89
|
||||
curl -s -XPOST localhost:8090/v1/leases -d '{"selector":{"geo":"local"},"ttlSeconds":300}'
|
||||
# → 201 {leaseID, proxy(activeLeases:1), expiresAt, ttlSeconds:300}
|
||||
curl -XPOST .../report -d '{"result":"rate_limited","target":"example.com"}' # 204
|
||||
curl -XPOST /v1/leases -d '{...,"target":"example.com"}' # 409 {inCooldown:1} ✓
|
||||
curl -XDELETE /v1/leases/<id> # 204, and 204 again ✓
|
||||
kubectl delete -f config/samples/proxy_kubernetes.yaml # finalizer: pod Terminating, CR gone
|
||||
kind delete cluster --name proxy-operator-demo
|
||||
```
|
||||
|
||||
A real Squid pod went Ready through a real CONNECT probe, a lease was
|
||||
held on it, the cooldown machinery answered a 409 with correct
|
||||
arithmetic, and the finalizer cleaned up — the plan's success bar, met
|
||||
with the actual product.
|
||||
|
||||
Worth noting: `make deploy` runs `kustomize edit set image` and mutates
|
||||
`config/manager/kustomization.yaml` in the working tree — reverted before
|
||||
committing (the repo keeps the pinned stanza). The health probe's ~89 ms
|
||||
latency is gstatic-through-squid from a kind pod on this machine;
|
||||
metrics-side observations were not separately checked in-cluster (covered
|
||||
by unit tests).
|
||||
|
||||
Reference in New Issue
Block a user