kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Add the mock provider with a real CONNECT proxy per port (Step 3)

Browse Source 
An in-memory provider.Provider whose state (Provisioning -> Running ->
Terminated -> purged) is a pure function of an injectable clock, not
background timers, so it's deterministic under tests and correct under
real time with no goroutine lifecycle to leak.

Once an instance is observed Running, it lazily acquires a real HTTP
CONNECT proxy listener so the health engine's through-the-proxy probe
(later steps) genuinely tunnels a request end to end, instead of the
healthcheck being simulated or bypassed for local development.

Redesigned the listener sharing model from what the plan assumed: the
plan's "one loopback IP per instance" doesn't work on macOS (only
127.0.0.1 binds without a privileged ifconfig alias, unlike Linux where
the whole 127.0.0.0/8 routes to loopback by default), and there's no
channel for a provider to report a port back to the reconciler anyway
(EffectivePort() is spec-only). Instances now share one real listener
per port, reference-counted at the package level rather than per
Provider instance, since a bound TCP port is a genuinely process-global
OS resource -- two separately configured mock-typed provider entries
must not both try to bind the same default port.

Fault injection wired both ways: MockConfig.FailNextCreates/FailWith for
demos, InjectCreateFailures(n, class) for tests. Create is idempotent by
name.

Caught and fixed a real test flake (not a logic bug): the freePort test
helper asked the OS for a free port via bind-then-close, a TOCTOU race
under t.Parallel() that let two tests collide on the same "free" port.
Replaced it with a monotonic counter, since these tests only need
uniqueness within the test run.

internal/provider/mock at 91.1% coverage, including an end-to-end test
that opens real sockets: Create -> Get past provisionDelay -> a real
http.Client tunnelling a CONNECT through the mock to a real TLS origin.
make test green across the whole repo.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Jan Novak
2026-08-07 23:26:34 +02:00
parent a4a483acbc
commit ef1387dc01
6 changed files with 1016 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

106
docs/plans-executions/2026-08-07-1747-proxy-operator.md
View File

@@ -7,7 +7,7 @@ Pairs with [docs/plans/2026-08-07-1747-proxy-operator.md](../plans/2026-08-07-17
- [x] Step 0 — Branch and scaffold
- [x] Step 1 — API types (`api/v1alpha1/proxy_types.go`)
- [x] Step 2 — Provider contract (`internal/provider/`)
- [ ] Step 3 — Mock provider (`internal/provider/mock/`)
- [x] Step 3 — Mock provider (`internal/provider/mock/`)
- [ ] Step 4 — Reconciler (`internal/controller/`)
- [ ] Step 5 — Health engine (`internal/health/`)
- [ ] Step 6 — Lease store (`internal/lease/`)
@@ -199,3 +199,107 @@ built in (rejects unknown fields, which is what "fail fast on unknown type"
in the plan actually needs) and was already pulled in transitively by the
k8s.io toolchain, so no new dependency was added — `go mod tidy` just
promoted it from indirect to direct.
## Step 3 — Mock provider (`internal/provider/mock/`)
Started from the plan's design (state as a pure function of an injectable
clock, no background timers) but had to redesign the "real proxy" part
before writing any code, once a load-bearing assumption turned out false.
The plan (and the earlier decision to make the mock run a real proxy)
assumed each instance could get its own loopback address —
`127.0.0.1:<port>` per instance, "a fake IP from a private range." Verified
that assumption directly before committing to it:
```bash
cat <<'EOF' > /tmp/loopbacktest.go
package main
import ("fmt"; "net")
func main() {
for _, addr := range []string{"127.0.0.2:0", "127.0.0.55:0", "127.1.2.3:0"} {
l, err := net.Listen("tcp", addr)
if err != nil { fmt.Printf("%s: FAIL: %v\n", addr, err); continue }
fmt.Printf("%s: OK\n", addr)
l.Close()
}
}
EOF
go run /tmp/loopbacktest.go
# 127.0.0.2:0: FAIL: listen tcp 127.0.0.2:0: bind: can't assign requested address
# 127.0.0.55:0: FAIL: listen tcp 127.0.0.55:0: bind: can't assign requested address
# 127.1.2.3:0: FAIL: listen tcp 127.1.2.3:0: bind: can't assign requested address
```
Only `127.0.0.1` binds on macOS without `sudo ifconfig lo0 alias ... up` —
Linux routes the whole `127.0.0.0/8` block to loopback by default, macOS
doesn't. That's not something the operator can or should do at runtime, so
per-instance loopback IPs were out. There's also a second problem the
per-instance-IP design didn't solve anyway: `EffectivePort()`
(`api/v1alpha1/helpers.go`, Step 1) is computed purely from `spec.port`
with no channel for a provider to report back a different *port* — so
whatever a mock instance actually listens on has to be the literal port the
reconciler will pass through `CreateRequest.Port`, not an OS-assigned
ephemeral one.
Redesigned around one real listener **per port**, shared and
reference-counted across every instance that uses it, rather than one
listener per instance (`proxy.go`, `sharedProxies`). This fixes both
problems at once: every instance binds the same `127.0.0.1` (no OS issue),
and any number of instances can share a port without conflict since it's
the exact same underlying listener. Deliberately made the refcounting
**package-level**, not a field on `mock.Provider`, because a bound TCP port
is a genuinely process-global OS resource — two separately configured
mock-typed provider entries (e.g. two named `"mock"` instances in
`providers-config.yaml`) would otherwise both try to bind the same default
port and the second one would just fail. This is a case where a package
global is the correct model, not a shortcut: it mirrors an OS-level
singleton, not application state.
Instance lifecycle otherwise follows the plan exactly: `Get`/`ListByTag`
derive `Provisioning → Running → Terminated → purged (ErrNotFound)` from
`createdAt`/`deletedAt` compared against an injectable clock, with the real
proxy listener acquired lazily on the first observed `Running` and released
on `Delete` (or lazily on purge, so orphaned records can't leak a
reference). `Create` is idempotent by name. Fault injection is wired both
ways per the plan: `MockConfig.FailNextCreates`/`FailWith` for the demo
config, `InjectCreateFailures(n, class)` for tests.
Tests initially had a real flake, caught by running with `-race -count=3`
rather than trusting one green run:
```bash
go test -race -v ./internal/provider/mock/... 2>&1 | tail -5
# --- FAIL: TestAcquireProxy_sharedAcrossAcquires
# proxy_test.go:37: port not released after last reference: bind: address already in use
```
Root cause wasn't the refcounting logic — it was the test helper. `freePort`
asked the OS for a free port by binding to `:0` and immediately closing it,
which is a classic TOCTOU race under `t.Parallel()`: two tests can be handed
the same "free" port before either actually claims it, since nothing holds
it open in between. Fixed by replacing the OS-asks approach with a
monotonic counter (`20000 + atomic.Int32`) — these tests only need a port
unique *within this test run*, not one verified free by the OS at an
instant in time, so guaranteeing uniqueness outright is both simpler and
correct where the "ask and hope" approach wasn't. Reran `-race -count=3`
clean afterward.
Added tests beyond the plan's list to close real coverage gaps rather than
stopping at "green": config-override branches in `New`, all four
`failClassFromString` branches, `ListByTag`'s purge-on-list and
Running/IP-inclusion paths, and a plain-`http://` forwarding test
(`handleForward`) alongside the CONNECT one, since a `probeURL` override
could use either scheme. Landed at 91.1% coverage; the remainder is
OS-failure branches (bind errors, hijack failures) not worth simulating for
a prototype.
The `TestProvider_realProxyTunnelsConnect` test is the one that actually
matters most here: it opens real sockets end to end — mock `Create` →
`Get` past `provisionDelay` → real `http.Client` with
`Transport.Proxy` dialing through the mock's CONNECT tunnel to a real
`httptest.NewTLSServer` — and gets a real `204` back. That's the concrete
proof the "mock runs a real proxy" decision actually delivers a genuine
end-to-end healthcheck, not a simulated one.
`internal/provider/mock` at 91.1% coverage. `make test` green across the
whole repo.