egress-proxies-operator/docs/plans/2026-09-11-1526-provider-health-checks.md

Plan: Provider health checks — see whether the cluster can reach each provider

Created: 2026-09-11 15:26 UTC

Context

Today the operator can't say whether a provider's backend is reachable. The Provider interface (provider.go:89-109) only has Create/Get/Delete/ListByTag, and /healthz and /readyz are bare healthz.Ping. A broken KVM path only shows up later: as a Proxy stuck in Provisioning, or as an orphan-GC sweep logging an error. Several failures stay invisible until a Proxy needs that host:

• a missing kvm-ssh Secret;
• a rotated host key;
• a Tailscale route that is down;
• a storage pool that is inactive or full.

The goal is a periodic sanity check of every configured provider from inside the cluster, with results you can see through kubectl and Prometheus.

Decisions made by the user during planning:

1. Surface: a new operator-owned, read-only, cluster-scoped ProviderHealth CRD, with one object per configured provider, a Ready condition and per-target detail. Also Prometheus gauges, and a log line whenever a target goes up or down. No EventRecorder, matching the earlier decision.
2. Scope: real checks for all three providers.
   • kvm: checked per host.
   • kubernetes: confirms the API server and RBAC work.
   • gcp: confirms credentials and the API work.
3. KVM depth: reachability plus the storage pool.
   • Over the existing pooled SSH+libvirt connection, call the libvirt version and hostname RPCs.
   • Confirm the configured pool is running and report its free space.
   • No bridge check.

Not affected: the Proxy reconcile behaviour, and the manager's /healthz and /readyz. A dead hypervisor must never make the manager unready.

Design

Contract: internal/provider/check.go

type TargetStatus struct {
    Name, Address   string
    Ready           bool
    Reason, Message string
    Latency         time.Duration
    Details         map[string]string
}
// Check never errors; each target carries its own verdict. Must return promptly once ctx ends.
type Checker interface{ Check(ctx context.Context) []TargetStatus }

• Shared reasons: Reachable, Timeout, Unreachable, Unauthorized, Forbidden, APIError.
• Compile-time checks: each provider's check.go has var _ provider.Checker = (*Provider)(nil).
• Bypass the wrappers. cmd/main.go:197-201 wraps every provider in WithTracing(WithMetrics(...)), so the Checkers are collected from the raw providers map, before that loop. As a result, checks don't inflate provider_requests_total.

The checks

kvm (internal/provider/kvm/check.go)

• Concurrency: one goroutine per cfg.Hosts entry (wg.Go), writing to out[i] so results stay in config order.

• Inside p.hosts.call: ConnectGetLibVersion, ConnectGetHostname, StoragePoolLookupByName(host.Pool), then StoragePoolGetInfo.

• Rules for fn:

  • It returns only raw RPC errors. call drops the shared connection on any other error (conn.go:160), so "pool not running" is decided after call returns, never returned from fn.
  • It fills a captured result struct only after all four calls succeed. That struct is read only when call returned nil, which avoids a race with an abandoned goroutine.

• Details: hostname, libvirtVersion (decoded as major.minor.patch), pool, poolState, poolCapacity and poolAvailable (in GiB).

• New RPCs: add the three to libvirtAPI (conn.go:29-47), tracingAPI (trace.go) and fakeAPI (fake_test.go). fakeAPI gets a hostname, a fixed version, and a per-pool state map that defaults to Running.

• Telling failure stages apart without matching strings: add type dialError struct{ stage dialStage; err error } in conn.go, with Error() returning the inner message unchanged and Unwrap(). It wraps at four places:

  • lazySSH.dial, when loading the key material fails (ssh-material)
  • dialSSH, at the dial step (tcp)
  • dialSSH, at the handshake (ssh-handshake)
  • connectLibvirt (libvirt-connect)

  Because Unwrap is kept, classify and isClassified still see ErrPermanent and fs.ErrNotExist, so reconcile behaviour doesn't change.

• checkReason(err), tested in order:

  Condition	Reason
  stage ssh-material	SSHMaterialUnavailable
  stage tcp	Unreachable
  stage ssh-handshake	SSHHandshakeFailed
  stage libvirt-connect	LibvirtConnectFailed
  ctx deadline or cancel	Timeout
  ErrNoStoragePool	PoolNotFound
  any other libvirt error	LibvirtError
  anything else	Unreachable
  success, pool not running	PoolNotActive
  success	Reachable

• Accepted behaviour:

  • A check that hits its timeout drops the shared connection. That already happens for any cancelled reconcile context, and it only fires when the host isn't answering.
  • The leader now keeps one SSH connection per host open from startup, with keepalives every 15s. Before, the connection only opened with the first KVM reconcile.
  • No forced periodic redial. lazySSH caches the key material forever, so a redial would not pick up a rotated Secret anyway; that gap goes into the docs under "Known gaps".

kubernetes (internal/provider/kubernetes/check.go)

• Call: List pods with the managed-label selector and client.Limit(1), cluster-wide. It uses the same uncached client, verb and RBAC as ListByTag and GC. There is no namespace setting in the kubernetes config.
• Target: one target named apiserver, with the address from a new apiHost field set in NewWithTransportWrapper.
• Reasons:
  • Timeout: the context ended, or the error is IsTimeout/IsServerTimeout.
  • Unauthorized, Forbidden.
  • APIError: any other API status.
  • Unreachable: anything else.

gcp (internal/provider/gcp/check.go, plus a seam change in gcp.go)

• Why a new seam method: the existing AggregatedList seam reads every page (gcp.go:63-78), and MaxResults is only the page size. So add Probe(ctx, *computepb.AggregatedListInstancesRequest) error, which makes one it.Next() call and treats iterator.Done as nil.
• Request: Project, MaxResults: 1, ReturnPartialSuccess: true, and the same managed-label filter ListByTag uses.
• Target: one per project, Name: cfg.Project, Address: compute.googleapis.com. GCP config has no zones, and reading them from Proxy placements would tie the check to the Proxy cache.
• Reasons:
  • Timeout: the context ended.
  • For *googleapi.Error: 401 is Unauthorized, 403 is Forbidden (this also covers a disabled API), anything else is APIError.
  • Unreachable: anything else, including a failure to get a token.

The ProviderHealth CRD: api/v1alpha1/providerhealth_types.go

• Written by hand, since kubebuilder isn't installed; PROJECT is edited by hand too.
• Scope and markers: cluster-scoped, shortName=ph, status subresource, no spec.
• Scheme registration: add the type to the existing AddKnownTypes call in proxy_types.go:362, not a second init().
• Status:
  • type
  • conditions (Ready; ConditionReady constant)
  • targets[], as a list-map keyed by name, each with: name, address, ready, reason, message (at most 1024 characters), latencyMillis, lastTransitionTime, details
  • targetsReady (for example "1/2", used by a printer column)
  • lastCheckTime
• Printer columns: Type, Ready, Targets, Last Check, Age.
• Viewer role: config/rbac/providerhealth_viewer_role.yaml. No editor role, since the operator owns the object.

Monitor: internal/providerhealth

• Fields: Monitor{Client, Reader, Targets []Target{Name, Type, Checker}, Interval, Timeout}.
• Scheduling:
  • NeedLeaderElection() true.
  • Checks once immediately, then on a ticker. Copy the ticker loop from gc.go:54-80.
• Each cycle:
  • Runs every provider concurrently under context.WithTimeout(Timeout), inside one tracing.Start span.
  • Stores the snapshot under a mutex; Snapshot() returns a deep copy.
  • Logs changes. Going down logs WARNING: provider target not ready with provider, target, address, reason and message. Coming back logs Provider target ready. A target that is ready on first sight logs at V(1) only.
• Writes:
  • Get through mgr.GetAPIReader(), not the cache. That avoids a cluster-wide informer and the watch permission it would need, whatever --proxy-namespace is set to.
  • Create the object if it is missing, labelled app.kubernetes.io/managed-by=egress-proxies-operator.
  • Then Status().Patch(MergeFrom(base)) when the status changed, following status.go:114-121.
  • This means one write per provider per interval, because lastCheckTime moves each time. That's acceptable given the 10s minimum interval.
• buildStatus(prev, result, now) is a pure function:
  • A target keeps its lastTransitionTime unless ready flipped.
  • The Ready condition is set through apimeta.SetStatusCondition:
    • True/AllTargetsReady when every target is ready
    • False/TargetsNotReady otherwise, with the failing targets in the message
    • Unknown/ChecksNotImplemented when the provider has no Checker
    • Unknown/NoTargets when there are zero targets
• Cleanup, once at Start: delete labelled objects whose name isn't in the config. The config is loaded once per process, and unlabelled objects are never touched.
• Interval 0 (disabled): run the cleanup with nothing kept, then return nil, so no object is left showing a stale Ready=True.
• Invalid provider names: provider names aren't validated as DNS-1123 names. For an invalid name, warn once and skip the object writes; metrics are still emitted.
• ValidateSchedule(interval, timeout) is called in main.go before the manager starts. Interval must be 0 or at least 10s; timeout must be in (0, interval].
• RBAC markers on Monitor:
  • providerhealths: get, list, create, delete
  • providerhealths/status: get, patch
  • make manifests regenerates config/rbac/role.yaml.

Metrics and wiring

• New function: metrics.RegisterProviderTargets(reg, read func() []ProviderTarget) error. Register's signature is left alone.
• Gauges: a scrape-time collector, in the style of constCollector, reads the Monitor snapshot and emits three gauges labelled {provider,target}:
  • proxy_operator_provider_target_up
  • proxy_operator_provider_check_duration_seconds
  • proxy_operator_provider_check_timestamp_seconds
• Conversion closure: the one that turns the Monitor snapshot into gauges lives in main.go, next to proxyPhaseCounts. So providerhealth doesn't import metrics.
• No pool-bytes gauge: that value is in status, and node-exporter is the right source for disk alerts.
• cmd/main.go:
  • New flags --provider-check-interval (1m) and --provider-check-timeout (15s).
  • Build the Targets from cfg.Providers in config order, with c, _ := raw[pc.Name].(provider.Checker).
  • mgr.Add(&providerhealth.Monitor{Client: mgr.GetClient(), Reader: mgr.GetAPIReader(), ...}), then register the collector.

Steps (one commit each, execution-log entry per step)

Step 0 — Save the plan.

• Create branch feat/provider-health off main.
• Copy this plan to docs/plans/2026-09-11-1526-provider-health-checks.md and commit it alone.
• The uncommitted .claude/settings.json change stays out of every commit.

Step 1 — ProviderHealth API and CRD.

• api/v1alpha1/providerhealth_types.go
• zz_generated.deepcopy.go, from make generate
• config/crd/bases/crawl.example.com_providerhealths.yaml, from make manifests
• the CRD and RBAC kustomization.yaml files, plus the viewer role
• PROJECT

Step 2 — Contract plus the kubernetes and GCP checks.

• internal/provider/check.go
• internal/provider/kubernetes/{kubernetes.go,check.go,check_test.go}
• internal/provider/gcp/{gcp.go,check.go,check_test.go,gcp_test.go}

Step 3 — KVM check.

• internal/provider/kvm/{conn.go,trace.go,check.go,check_test.go,fake_test.go,conn_test.go,trace_test.go,kvm_integration_test.go}

Step 4 — Monitor.

• internal/providerhealth/{monitor.go,status.go,monitor_test.go,status_test.go,envtest_test.go}
• config/rbac/role.yaml, from make manifests

Step 5 — Metrics and wiring.

• internal/metrics/{metrics.go,metrics_test.go}
• cmd/main.go
• config/manager/manager.yaml: a comment for the new flags
• test/e2e/e2e_test.go

Step 6 — Docs.

• docs/architecture.md: a component row plus a provider-health section
• docs/kvm-provider.md: the reason table, the shared-connection behaviour, and the Secret-rotation gap under "Known gaps"
• docs/testing.md: the new tests and the commands
• CLAUDE.md: its Architecture and Gotchas sections still say "empty scaffold"; point them at docs/architecture.md

Step 7 — Validate, record, push, open the PR.

• Run go vet ./..., go test -short ./..., make test and make test-e2e.
• Hand the user the KVM commands below, and record their results in the execution log.
• Add a CHANGELOG entry once the user confirms.
• Push, using the token-file credential helper from memory.
• Open the PR against main through the Gitea API (tea isn't installed).
• Don't merge.

Tests

Provider checks (table-driven, Test<Func>_<scenario>):

• kubernetes:
  • reachable
  • list options: Limit==1, the label selector, no namespace
  • failure reasons: forbidden, unauthorized, timeout, cancelled, internal, and a network *url.Error
• gcp:
  • reachable, asserting the request fields
  • failure reasons: 401, 403, 500, deadline, network
  • TestRealInstances_Probe_fetchesOnePage against httptest: exactly one request with maxResults=1, even though the page has a nextPageToken
• kvm (using hostFakes/newTestProvider from kvm_test.go):
  • reachable, with details filled in
  • per-host verdicts, order preserved
  • pool inactive, pool missing
  • hosts checked concurrently (a sequential check would deadlock)
  • a pool verdict keeps the connection (the dial count stays 1)
  • a checkReason table
  • TestSSHAuth_dial_stages, using the in-process SSH server in conn_test.go
  • three new rows in the tracing table

Monitor:

• Setup: the fake client is built with a status subresource and a RESTMapper that registers ProviderHealth as cluster-scoped.
• Tests:
  • buildStatus table
  • objects are created with the label, and existing ones are patched
  • the timeout bounds a checker that blocks
  • a nil checker gives Unknown
  • only transitions are logged (funcr logger)
  • cleanup deletes only labelled objects that aren't configured
  • the disabled path cleans up and returns
  • an invalid name skips the writes
  • ValidateSchedule table
• An envtest round-trip, skipped under -short, which proves the generated schema is accepted by a real apiserver.

Metrics: TestRegisterProviderTargets_scrape, using a fresh registry and GatherAndCompare; an empty snapshot produces no series.

e2e (kind): providerhealth kubernetes reaches Ready=True, which proves the generated RBAC works end to end.

Verification

Automated (Claude runs): go vet ./..., go test -short ./..., make test (envtest with -race) and make test-e2e.

Against the real hypervisor (handed to the user, not run by Claude), with beelink as the host:

KVM_TEST_HOST=beelink.tail8c69c.ts.net:22 KVM_TEST_USER=novakj KVM_TEST_SSH_KEY=~/.ssh/id_ed25519 \
KVM_TEST_KNOWN_HOSTS=~/.ssh/known_hosts KVM_TEST_BRIDGE=<bridge> KVM_TEST_IMAGE_URL=<image-url> \
  go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/

make install && go run ./cmd/main.go --providers-config <scratch copy of kvm-home config, local key paths> \
  --metrics-bind-address :8080 --metrics-secure=false --provider-check-interval=15s
kubectl get ph -w
kubectl get ph kvm-home -o yaml
curl -s localhost:8080/metrics | grep -E 'proxy_operator_provider_(target_up|check_)'

Each negative case is a one-line edit to the scratch config; nothing changes on the host.

Edit	Expected reason
pool: nope	PoolNotFound
address: …:2	Unreachable
knownHostsFile: /dev/null or user: nobody	SSHHandshakeFailed
sshKeyFile: /nonexistent	SSHMaterialUnavailable
socket: /nonexistent	LibvirtConnectFailed

Out of scope

• Gating Proxy creates on a host that is known to be down.
• Tying /readyz to provider health.
• Bridge checks.
• SelfSubjectAccessReview checks of create/delete rights.
• Hot-reloading the providers config.
• Picking up a rotated kvm-ssh Secret without a restart. This is an existing gap; it gets documented.