Log the provider docs split in the changelog

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

CHANGELOG.md

@@ -1,5 +1,46 @@
 # Changelog
 
+## 2026-09-16 11:46 UTC — Provider blueprint split into contract, spec template and appendix
+
+- **`docs/provider-contract.md`** is the generic provider specification: MUST/SHOULD/MAY
+  rules with the caller behind each one, 19 behaviour-profile axes with stable codes
+  (where providers may differ, such as blocking vs submit-and-return Create, or a
+  backend with no provisioning state), error taxonomy, credential hygiene, testing
+  obligations, the adding-a-provider checklist and the contract's known limits. It
+  names no existing provider.
+- **`docs/provider-spec-template.md`** is the fill-in questionnaire a new provider's
+  design answers before implementation; entry numbers match the contract's profile
+  table.
+- **`docs/providers-existing.md`** maps kubernetes, gcp and kvm to every profile and
+  keeps their per-provider reference notes, shared gaps and proposed features.
+- **`docs/providers.md`** is now an index; README, CLAUDE.md and architecture.md point
+  at the new files. This supersedes the single-file blueprint written earlier on the
+  same branch.
+- Two rules the callers enforce that were not written down before: orphan GC sweeps
+  providers sequentially with no deadline, so a hung `ListByTag` stalls all collection;
+  and the reconciler deletes an instance a backend reports as Stopped before start.
+
+## 2026-09-16 08:45 UTC — Provider health checks: ProviderHealth CRD, per-provider checks, gauges
+
+- **Each configured provider is now checked on a schedule** and reported in a
+  cluster-scoped `ProviderHealth` object (`kubectl get ph`), one per provider, with a
+  `Ready` condition, a reason, and per-target verdicts.
+- **Checks:** kubernetes lists one object from the API server; GCP fetches a single
+  instance page; KVM checks SSH/libvirt reachability and the storage pool on every
+  host, in parallel. Failures map to fixed reasons (`Unreachable`,
+  `SSHHandshakeFailed`, `PoolNotFound`, `Timeout`, …).
+- **The KVM check can't hurt reconciles:** it uses `hostPool.probe`, which gives up on
+  a slow RPC without dropping the shared connection, and waiting for a host slot can
+  be interrupted.
+- **Metrics:** `proxy_operator_provider_target_up`, `_check_duration_seconds` and
+  `_check_timestamp_seconds`. The flags are `--provider-check-interval` and
+  `--provider-check-timeout` (defaults to `min(15s, interval)`).
+- Validated with `make test`, `make test-e2e` and the real-hypervisor KVM run against
+  beelink. `make lint` is blocked by a Go 1.27 toolchain/staticcheck panic that has
+  nothing to do with this change.
+- Key files: `api/v1alpha1/providerhealth_types.go`, `internal/provider/check.go`,
+  `internal/provider/kvm/`, `internal/providerhealth/`, `docs/architecture.md`.
+
 ## 2026-09-10 22:58 UTC — Code-review fixes on the KVM branch: deletion, validation, lazy SSH, guest agent
 
 Five findings from a review of `feat/kvm-provider`, all fixed on the branch.

CLAUDE.md

@@ -25,10 +25,16 @@ entrypoint yet, so there's nothing to run locally.
 
 ## Architecture
 
-Not yet built. TODO: once the controller-runtime manager and reconcile loop exist,
-document the data-flow diagram and package/role table here, following the standard
-reconcile-loop shape: fetch → resolve → list related → compute desired state → merge
-→ patch only on diff.
+Documented in [docs/architecture.md](docs/architecture.md). It holds the component
+table, the reconcile flow, and one section per runnable: health engine, discovery and
+lease API, orphan GC, metrics, tracing, and provider health checks. The generic
+provider contract (MUST/SHOULD rules, behaviour profiles, checklist for adding a
+provider, known limits) is in [docs/provider-contract.md](docs/provider-contract.md);
+a new provider starts from [docs/provider-spec-template.md](docs/provider-spec-template.md);
+how the existing providers meet the contract, with their gaps, is in
+[docs/providers-existing.md](docs/providers-existing.md); [docs/providers.md](docs/providers.md)
+indexes them. Provider-specific detail is in `docs/kvm-provider.md` and the GCP
+docs; the test layout is in `docs/testing.md`.
 
 ## Conventions
 
@@ -214,7 +220,9 @@ A lightweight `check` job (`go vet` / `go build` / `go test -short`) gates the b
 
 ## Gotchas
 
-- This repo currently contains only `go.mod` (module
-  `gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator`) — no controller code,
-  `main.go`, Dockerfile, k8s manifests, Makefile, or CI config exist yet. Nothing in
-  this file describes working software; it's the convention scaffold to build against.
+- The "What this is" and "Commands" sections above predate the code and are stale.
+  The Makefile targets (`make test`, `make manifests generate`, `make test-e2e`,
+  `make test-integration-kvm`) and `docs/testing.md` are authoritative.
+- `+kubebuilder:rbac` markers are package-level. Keep them in their own comment block,
+  never inside a type's or function's doc comment: controller-gen silently ignores
+  them there. After `make manifests`, check `git diff config/rbac/role.yaml`.

PROJECT

@@ -18,4 +18,11 @@ resources:
   kind: Proxy
   path: gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/api/v1alpha1
   version: v1alpha1
+- api:
+    crdVersion: v1
+  domain: example.com
+  group: crawl
+  kind: ProviderHealth
+  path: gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/api/v1alpha1
+  version: v1alpha1
 version: "3"

README.md

@@ -104,7 +104,8 @@ make deploy IMG=<registry>/egress-proxies-operator:dev
 
 - Provider config comes from the `providers-config` ConfigMap
   ([config/manager/providers_config.yaml](config/manager/providers_config.yaml));
-  the default ships only the kubernetes provider.
+  the default ships the kubernetes provider plus a `kvm-home` entry that needs
+  the `kvm-ssh` Secret (without it the manager starts and warns).
 - The discovery API is exposed by the
   `controller-manager-discovery-service` Service on port 8090.
 - Auth: create the token Secret, or the API serves **unauthenticated**
@@ -185,46 +186,19 @@ can be two differently-configured instances of the same `gcp` type (see
 | --- | --- | --- | --- |
 | `kubernetes` | A real Squid pod (`ubuntu/squid:6.6-24.04_edge` by default) in the same cluster the operator runs in | `image` (optional) | Needs no cloud account — local dev, CI, and the kind quickstart. Pods share the cluster's egress IP, so it exercises the full lifecycle but not distinct egress paths. |
 | `gcp` | A Compute Engine VM with an ephemeral external IP | `project` (required), `network`, `networkTag`, `diskSizeGb` | The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: `Create` returns as soon as the operation is submitted and the reconciler discovers progress by polling `Get`. Auth is Application Default Credentials — workload identity in-cluster, `gcloud` ADC locally; no key-file plumbing. |
+| `kvm` | A libvirt VM on one of your own hypervisors, reached over SSH | `sshKeyFile`, `knownHostsFile` (unless `insecureIgnoreHostKey`), `machineTypes`, `hosts` (required); `images`, `diskSizeGb`, `vendorData`, `installGuestAgent` | Home-lab or on-prem egress. The base image is downloaded into each host's pool on first use; `Create` waits for the VM to start. See [docs/kvm-provider.md](docs/kvm-provider.md). |
 
 ### Adding a provider
 
-A new backend (Hetzner, AWS, ...) is four pieces; the contract lives in
-[internal/provider/provider.go](internal/provider/provider.go):
-
-1. **Implement the 4-method `Provider` interface** in a new
-   `internal/provider/<type>/` package:
-   - `Create` submits and returns — it never blocks until the VM runs, and
-     must be idempotent keyed on `req.Name` (a deterministic name derived
-     from the Proxy's UID), so a repeat call after a crash finds the
-     existing instance instead of duplicating it.
-   - `Get` returns `provider.ErrNotFound` as a *normal* outcome — the
-     reconciler branches on it for replacement and adoption, so don't
-     treat it as exceptional.
-   - `Delete` is idempotent: deleting an already-gone instance is not an
-     error.
-   - `ListByTag` returns every instance the operator ever tagged, for
-     orphan GC.
-2. **Tag every created resource** with `LabelManaged=true` and
-   `LabelUID=<Proxy UID>`, and report `CreatedAt` — orphan GC relies on
-   all three to find owned resources and skip in-flight creates.
-3. **Classify every returned error** with `provider.Wrap` into the
-   four-sentinel taxonomy in
-   [internal/provider/errors.go](internal/provider/errors.go)
-   (`ErrNotFound` / `ErrQuotaExceeded` / `ErrTransient` / `ErrPermanent`)
-   — the reconciler decides retry, slow backoff, or latching `Failed`
-   purely from that classification, never from provider-specific types.
-4. **Wire it up**: add a type-specific config block in
-   [internal/provider/config.go](internal/provider/config.go), and
-   register the constructor in the builtins map in
-   [cmd/main.go](cmd/main.go) (`"<type>": <pkg>.New`). The registry
-   ([internal/provider/registry](internal/provider/registry/registry.go))
-   handles named instances, and the metrics wrapper is applied
-   automatically.
-
-Test against a fake API seam rather than the real cloud — see the
-`instancesAPI` seam in
-[internal/provider/gcp/gcp.go](internal/provider/gcp/gcp.go) for the
-pattern.
+The generic contract is [docs/provider-contract.md](docs/provider-contract.md),
+the questionnaire a new provider's design fills in is
+[docs/provider-spec-template.md](docs/provider-spec-template.md), and how the
+existing providers meet the contract is
+[docs/providers-existing.md](docs/providers-existing.md);
+[docs/providers.md](docs/providers.md) indexes them. Start there: several rules (a
+self-contained `providerID`, `Get` returning `ErrNotFound` only when the
+instance is really gone, `CreatedAt` read back for orphan GC) are enforced by
+callers, not by the interface.
 
 ## Tracing
 

api/v1alpha1/providerhealth_types.go

@@ -0,0 +1,127 @@
+/*
+Copyright 2026.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha1
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// ConditionReady summarizes a ProviderHealth's targets: True when every
+// target passed its last check.
+const ConditionReady = "Ready"
+
+// ProviderTarget is the latest check verdict for one thing a provider
+// depends on: a KVM hypervisor, the Kubernetes API server, a GCP project.
+type ProviderTarget struct {
+	// name identifies the target within its provider: the KVM host name,
+	// "apiserver", or the GCP project.
+	// +kubebuilder:validation:MinLength=1
+	// +required
+	Name string `json:"name"`
+
+	// address is where the check connected to.
+	// +optional
+	Address string `json:"address,omitempty"`
+
+	// ready is true when the last check succeeded.
+	// +required
+	Ready bool `json:"ready"`
+
+	// reason is a CamelCase verdict such as Reachable, Unreachable,
+	// Forbidden, SSHHandshakeFailed or PoolNotActive.
+	// +optional
+	Reason string `json:"reason,omitempty"`
+
+	// message is human-readable detail, usually the error of a failed check.
+	// +kubebuilder:validation:MaxLength=1024
+	// +optional
+	Message string `json:"message,omitempty"`
+
+	// latencyMillis is how long the last check took.
+	// +optional
+	LatencyMillis int64 `json:"latencyMillis,omitempty"`
+
+	// lastTransitionTime is when ready last changed.
+	// +required
+	LastTransitionTime metav1.Time `json:"lastTransitionTime"`
+
+	// details are provider-specific facts from the last check, e.g. the
+	// libvirt version and free storage-pool space of a KVM host.
+	// +optional
+	Details map[string]string `json:"details,omitempty"`
+}
+
+// ProviderHealthStatus is the result of the latest check of one provider.
+type ProviderHealthStatus struct {
+	// type is the provider's type from the providers config: kubernetes,
+	// gcp or kvm.
+	// +optional
+	Type string `json:"type,omitempty"`
+
+	// conditions hold Ready, which summarizes the targets.
+	// +listType=map
+	// +listMapKey=type
+	// +optional
+	Conditions []metav1.Condition `json:"conditions,omitempty"`
+
+	// targets hold one verdict per checked target.
+	// +listType=map
+	// +listMapKey=name
+	// +optional
+	Targets []ProviderTarget `json:"targets,omitempty"`
+
+	// targetsReady counts the ready targets as "ready/total".
+	// +optional
+	TargetsReady string `json:"targetsReady,omitempty"`
+
+	// lastCheckTime is when the latest check finished.
+	// +optional
+	LastCheckTime *metav1.Time `json:"lastCheckTime,omitempty"`
+}
+
+// +kubebuilder:object:root=true
+// +kubebuilder:subresource:status
+// +kubebuilder:resource:scope=Cluster,shortName=ph
+// +kubebuilder:printcolumn:name="Type",type=string,JSONPath=`.status.type`
+// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status`
+// +kubebuilder:printcolumn:name="Targets",type=string,JSONPath=`.status.targetsReady`
+// +kubebuilder:printcolumn:name="Last Check",type=date,JSONPath=`.status.lastCheckTime`
+// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
+
+// ProviderHealth reports the periodic health check of one configured
+// provider. It is named after the provider, and the operator creates,
+// updates and deletes it itself; there is no spec to edit.
+type ProviderHealth struct {
+	metav1.TypeMeta `json:",inline"`
+
+	// metadata is a standard object metadata
+	// +optional
+	metav1.ObjectMeta `json:"metadata,omitzero"`
+
+	// status is the result of the latest check
+	// +optional
+	Status ProviderHealthStatus `json:"status,omitzero"`
+}
+
+// +kubebuilder:object:root=true
+
+// ProviderHealthList contains a list of ProviderHealth
+type ProviderHealthList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata,omitzero"`
+	Items           []ProviderHealth `json:"items"`
+}

api/v1alpha1/proxy_types.go

@@ -361,7 +361,7 @@ type ProxyList struct {
 
 func init() {
 	SchemeBuilder.Register(func(s *runtime.Scheme) error {
-		s.AddKnownTypes(SchemeGroupVersion, &Proxy{}, &ProxyList{})
+		s.AddKnownTypes(SchemeGroupVersion, &Proxy{}, &ProxyList{}, &ProviderHealth{}, &ProviderHealthList{})
 		return nil
 	})
 }

api/v1alpha1/zz_generated.deepcopy.go

@@ -95,6 +95,120 @@ func (in *PlacementSpec) DeepCopy() *PlacementSpec {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ProviderHealth) DeepCopyInto(out *ProviderHealth) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Status.DeepCopyInto(&out.Status)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ProviderHealth.
+func (in *ProviderHealth) DeepCopy() *ProviderHealth {
+	if in == nil {
+		return nil
+	}
+	out := new(ProviderHealth)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *ProviderHealth) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ProviderHealthList) DeepCopyInto(out *ProviderHealthList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]ProviderHealth, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ProviderHealthList.
+func (in *ProviderHealthList) DeepCopy() *ProviderHealthList {
+	if in == nil {
+		return nil
+	}
+	out := new(ProviderHealthList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *ProviderHealthList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ProviderHealthStatus) DeepCopyInto(out *ProviderHealthStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]v1.Condition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	if in.Targets != nil {
+		in, out := &in.Targets, &out.Targets
+		*out = make([]ProviderTarget, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	if in.LastCheckTime != nil {
+		in, out := &in.LastCheckTime, &out.LastCheckTime
+		*out = (*in).DeepCopy()
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ProviderHealthStatus.
+func (in *ProviderHealthStatus) DeepCopy() *ProviderHealthStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(ProviderHealthStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ProviderTarget) DeepCopyInto(out *ProviderTarget) {
+	*out = *in
+	in.LastTransitionTime.DeepCopyInto(&out.LastTransitionTime)
+	if in.Details != nil {
+		in, out := &in.Details, &out.Details
+		*out = make(map[string]string, len(*in))
+		for key, val := range *in {
+			(*out)[key] = val
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ProviderTarget.
+func (in *ProviderTarget) DeepCopy() *ProviderTarget {
+	if in == nil {
+		return nil
+	}
+	out := new(ProviderTarget)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *Proxy) DeepCopyInto(out *Proxy) {
 	*out = *in

cmd/main.go

@@ -16,8 +16,8 @@ limitations under the License.
 
 // Package main is the composition root: it loads the provider config (fail
 // fast), assembles the provider registry, and wires the reconciler, health
-// engine, lease store, discovery API, orphan GC, and metrics onto one
-// controller-runtime manager.
+// engine, lease store, discovery API, orphan GC, provider health checks and
+// metrics onto one controller-runtime manager.
 package main
 
 import (
@@ -60,6 +60,7 @@ import (
 	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kubernetes"
 	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kvm"
 	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/registry"
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/providerhealth"
 	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
 	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/version"
 	// +kubebuilder:scaffold:imports
@@ -97,6 +98,7 @@ func main() {
 	var showVersion bool
 	var gcpWireFullPayloads bool
 	var traceHealthProbes bool
+	var providerCheckInterval, providerCheckTimeout time.Duration
 
 	flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+
 		"Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.")
@@ -140,6 +142,13 @@ func main() {
 	flag.BoolVar(&traceHealthProbes, "trace-health-probes", false,
 		"Emit one trace span per health probe. Off by default: probes run about "+
 			"once per second per proxy and would dominate trace volume.")
+	flag.DurationVar(&providerCheckInterval, "provider-check-interval", time.Minute,
+		"Interval between provider health checks (can the cluster reach each hypervisor, API "+
+			"and account), published as ProviderHealth objects and metrics. 0 turns them off; "+
+			"otherwise at least 10s.")
+	flag.DurationVar(&providerCheckTimeout, "provider-check-timeout", 0,
+		"Time limit for one round of provider health checks: 15s, or the interval when that is "+
+			"shorter. If set, at most the interval.")
 
 	opts := zap.Options{
 		Development: true,
@@ -174,6 +183,10 @@ func main() {
 		setupLog.Error(nil, "--providers-config is required")
 		os.Exit(1)
 	}
+	if err := providerhealth.ValidateSchedule(providerCheckInterval, providerCheckTimeout); err != nil {
+		setupLog.Error(err, "Invalid provider check flags")
+		os.Exit(1)
+	}
 	cfg, err := provider.LoadConfigFile(providersConfig)
 	if err != nil {
 		setupLog.Error(err, "Failed to load providers config", "path", providersConfig)
@@ -195,6 +208,14 @@ func main() {
 		os.Exit(1)
 	}
 	m := metrics.New()
+	// Checkers come from the unwrapped providers, in config order: the
+	// metrics and tracing wrappers implement only Provider, and a check is
+	// not a provider request to be counted anyway.
+	checkTargets := make([]providerhealth.Target, 0, len(cfg.Providers))
+	for _, pc := range cfg.Providers {
+		checker, _ := providers[pc.Name].(provider.Checker)
+		checkTargets = append(checkTargets, providerhealth.Target{Name: pc.Name, Type: pc.Type, Checker: checker})
+	}
 	for name, p := range providers {
 		// Tracing outermost: the span covers the metrics recording too.
 		providers[name] = provider.WithTracing(name, provider.WithMetrics(name, p, m))
@@ -314,6 +335,18 @@ func main() {
 		os.Exit(1)
 	}
 
+	monitor := &providerhealth.Monitor{
+		Client:   mgr.GetClient(),
+		Reader:   mgr.GetAPIReader(),
+		Targets:  checkTargets,
+		Interval: providerCheckInterval,
+		Timeout:  providerCheckTimeout,
+	}
+	if err := mgr.Add(monitor); err != nil {
+		setupLog.Error(err, "Failed to add provider health monitor")
+		os.Exit(1)
+	}
+
 	if err := m.Register(ctrlmetrics.Registry,
 		proxyPhaseCounts(mgr.GetClient()),
 		func() int {
@@ -327,6 +360,10 @@ func main() {
 		setupLog.Error(err, "Failed to register metrics")
 		os.Exit(1)
 	}
+	if err := metrics.RegisterProviderTargets(ctrlmetrics.Registry, providerTargetSamples(monitor)); err != nil {
+		setupLog.Error(err, "Failed to register provider check metrics")
+		os.Exit(1)
+	}
 
 	if err := (&controller.ProxyReconciler{
 		Client:       mgr.GetClient(),
@@ -389,3 +426,23 @@ func proxyPhaseCounts(c client.Reader) func() map[string]int {
 		return counts
 	}
 }
+
+// providerTargetSamples flattens the monitor's latest results for the
+// scrape-time provider-check gauges.
+func providerTargetSamples(mon *providerhealth.Monitor) func() []metrics.ProviderTarget {
+	return func() []metrics.ProviderTarget {
+		var out []metrics.ProviderTarget
+		for _, r := range mon.Snapshot() {
+			for _, ts := range r.Targets {
+				out = append(out, metrics.ProviderTarget{
+					Provider:  r.Provider,
+					Target:    ts.Name,
+					Up:        ts.Ready,
+					Duration:  ts.Latency,
+					CheckedAt: r.CheckedAt,
+				})
+			}
+		}
+		return out
+	}
+}

config/crd/bases/crawl.example.com_providerhealths.yaml

@@ -0,0 +1,193 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.21.0
+  name: providerhealths.crawl.example.com
+spec:
+  group: crawl.example.com
+  names:
+    kind: ProviderHealth
+    listKind: ProviderHealthList
+    plural: providerhealths
+    shortNames:
+    - ph
+    singular: providerhealth
+  scope: Cluster
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .status.type
+      name: Type
+      type: string
+    - jsonPath: .status.conditions[?(@.type=="Ready")].status
+      name: Ready
+      type: string
+    - jsonPath: .status.targetsReady
+      name: Targets
+      type: string
+    - jsonPath: .status.lastCheckTime
+      name: Last Check
+      type: date
+    - jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: |-
+          ProviderHealth reports the periodic health check of one configured
+          provider. It is named after the provider, and the operator creates,
+          updates and deletes it itself; there is no spec to edit.
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          status:
+            description: status is the result of the latest check
+            properties:
+              conditions:
+                description: conditions hold Ready, which summarizes the targets.
+                items:
+                  description: Condition contains details for one aspect of the current
+                    state of this API Resource.
+                  properties:
+                    lastTransitionTime:
+                      description: |-
+                        lastTransitionTime is the last time the condition transitioned from one status to another.
+                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: |-
+                        message is a human readable message indicating details about the transition.
+                        This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: |-
+                        observedGeneration represents the .metadata.generation that the condition was set based upon.
+                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+                        with respect to the current state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: |-
+                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
+                        Producers of specific condition types may define expected values and meanings for this field,
+                        and whether the values are considered a guaranteed API.
+                        The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+                x-kubernetes-list-map-keys:
+                - type
+                x-kubernetes-list-type: map
+              lastCheckTime:
+                description: lastCheckTime is when the latest check finished.
+                format: date-time
+                type: string
+              targets:
+                description: targets hold one verdict per checked target.
+                items:
+                  description: |-
+                    ProviderTarget is the latest check verdict for one thing a provider
+                    depends on: a KVM hypervisor, the Kubernetes API server, a GCP project.
+                  properties:
+                    address:
+                      description: address is where the check connected to.
+                      type: string
+                    details:
+                      additionalProperties:
+                        type: string
+                      description: |-
+                        details are provider-specific facts from the last check, e.g. the
+                        libvirt version and free storage-pool space of a KVM host.
+                      type: object
+                    lastTransitionTime:
+                      description: lastTransitionTime is when ready last changed.
+                      format: date-time
+                      type: string
+                    latencyMillis:
+                      description: latencyMillis is how long the last check took.
+                      format: int64
+                      type: integer
+                    message:
+                      description: message is human-readable detail, usually the error
+                        of a failed check.
+                      maxLength: 1024
+                      type: string
+                    name:
+                      description: |-
+                        name identifies the target within its provider: the KVM host name,
+                        "apiserver", or the GCP project.
+                      minLength: 1
+                      type: string
+                    ready:
+                      description: ready is true when the last check succeeded.
+                      type: boolean
+                    reason:
+                      description: |-
+                        reason is a CamelCase verdict such as Reachable, Unreachable,
+                        Forbidden, SSHHandshakeFailed or PoolNotActive.
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - name
+                  - ready
+                  type: object
+                type: array
+                x-kubernetes-list-map-keys:
+                - name
+                x-kubernetes-list-type: map
+              targetsReady:
+                description: targetsReady counts the ready targets as "ready/total".
+                type: string
+              type:
+                description: |-
+                  type is the provider's type from the providers config: kubernetes,
+                  gcp or kvm.
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}

config/crd/kustomization.yaml

@@ -3,4 +3,5 @@
 # It should be run by config/default
 resources:
 - bases/crawl.example.com_proxies.yaml
+- bases/crawl.example.com_providerhealths.yaml
 # +kubebuilder:scaffold:crdkustomizeresource

config/manager/manager.yaml

@@ -66,6 +66,10 @@ spec:
           - --providers-config=/etc/proxy-operator/providers.yaml
           # Add --trace-health-probes to emit one span per health probe
           # (high volume: ~1/s per proxy).
+          # Provider health checks (kubectl get providerhealth) run every
+          # minute; tune with --provider-check-interval (0 turns them off,
+          # otherwise at least 10s) and --provider-check-timeout (15s, or
+          # the interval if that is shorter).
         env:
           # Bearer token for the discovery API. Optional: without the
           # Secret the API serves unauthenticated (with a loud warning).

config/rbac/kustomization.yaml

@@ -25,4 +25,7 @@ resources:
 - proxy_admin_role.yaml
 - proxy_editor_role.yaml
 - proxy_viewer_role.yaml
+# ProviderHealth objects are written only by the operator, so they get a
+# viewer role and nothing else.
+- providerhealth_viewer_role.yaml
 

config/rbac/providerhealth_viewer_role.yaml

@@ -0,0 +1,29 @@
+# This rule is not used by the project egress-proxies-operator itself.
+# It is provided to allow the cluster admin to help manage permissions for users.
+#
+# Grants read-only access to ProviderHealth objects, the results of the
+# operator's periodic provider checks. The operator creates, updates and
+# deletes them itself, so no admin or editor role is provided.
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    app.kubernetes.io/name: egress-proxies-operator
+    app.kubernetes.io/managed-by: kustomize
+  name: providerhealth-viewer-role
+rules:
+- apiGroups:
+  - crawl.example.com
+  resources:
+  - providerhealths
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - crawl.example.com
+  resources:
+  - providerhealths/status
+  verbs:
+  - get

config/rbac/role.yaml

@@ -22,6 +22,22 @@ rules:
   - get
   - list
   - watch
+- apiGroups:
+  - crawl.example.com
+  resources:
+  - providerhealths
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+- apiGroups:
+  - crawl.example.com
+  resources:
+  - providerhealths/status
+  verbs:
+  - get
+  - patch
 - apiGroups:
   - crawl.example.com
   resources:

docs/architecture.md

@@ -4,9 +4,9 @@
 
 | Component | Package | Runs as | Leader-elected | Role |
 |---|---|---|---|---|
-| Proxy CRD + helpers | `api/v1alpha1` | types | — | `Proxy` spec/status, CEL validation, defaulting, pure helpers |
+| Proxy + ProviderHealth CRDs | `api/v1alpha1` | types | — | `Proxy` spec/status, CEL validation, defaulting, pure helpers; the status-only, cluster-scoped `ProviderHealth` |
 | Reconciler | `internal/controller` | controller | yes (with the manager) | the state machine: provision, replace, delete, represent health |
-| Provider contract | `internal/provider` | library | — | `Provider` interface, error taxonomy, deterministic naming, config, metrics decorator |
+| Provider contract | `internal/provider` | library | — | `Provider` interface, optional `Checker`, error taxonomy, deterministic naming, config, metrics decorator; rules in [provider-contract.md](provider-contract.md), per-provider profile table in [providers-existing.md](providers-existing.md) |
 | kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) |
 | gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
 | kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds |
@@ -14,6 +14,7 @@
 | Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
 | Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
 | Orphan GC | `internal/gc` | Runnable | yes | deletes tagged instances whose CR is gone |
+| Provider health monitor | `internal/providerhealth` | Runnable | yes | periodic provider checks → one `ProviderHealth` per provider, target gauges, transition logs |
 | Metrics | `internal/metrics` | library | — | explicit registration, scrape-time collectors |
 | Composition root | `cmd/main.go` | binary | — | flags, provider registry, wires everything onto one manager |
 
@@ -306,6 +307,8 @@ registries). Two kinds:
 - **Scrape-time collectors** — `proxy_operator_proxies{phase}` and
   `proxy_operator_leases_active` read the cache / lease store at every
   scrape; reconcile-incremented gauges inevitably drift and leak series.
+  The provider-check gauges (§11) read the health monitor's snapshot the
+  same way.
 - **Fed vectors** — `healthcheck_duration_seconds{proxy}` and
   `healthcheck_failures_total{proxy}` observe EVERY probe (status writes
   are transition-only; metrics carry the high-frequency signal), and the
@@ -348,7 +351,8 @@ incoming trace context):
 - Discovery API: one server span per request (named from the route
   pattern), incoming W3C `traceparent` honored so clients' traces continue
   into the operator; `/healthz` excluded.
-- `gc.sweep` per GC pass; `health.probe` per probe **only** with
+- `gc.sweep` per GC pass; `providerhealth.check` per round of provider
+  checks; `health.probe` per probe **only** with
   `--trace-health-probes` (default off — probes run ~1/s per proxy).
 
 Log correlation: logr sinks never see a context, so trace IDs ride on the
@@ -359,6 +363,63 @@ Re-deriving from the base rather than layering keeps zap from emitting
 duplicate keys on nested spans. GCP V(5) wire logs get the same keys from
 the slog handler's ctx.
 
+### 11. Provider health checks (`internal/providerhealth/`)
+
+Answers "can this cluster reach each provider's backend right now?"
+without waiting for a Proxy to fail. A provider opts in by implementing
+`provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per
+target and never fails as a whole. All three providers do:
+
+| Provider | Target | What one check does |
+|---|---|---|
+| kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) |
+| kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on |
+| gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` |
+
+```text
+Monitor.Start(ctx)   ── leader-only; --provider-check-interval (1m; 0 = off,
+  │                     otherwise ≥ 10s), --provider-check-timeout (15s, or
+  │                     the interval if that is shorter)
+  ├─ once: delete managed-by-labelled ProviderHealth objects whose provider
+  │   is not configured (all of them when checks are off) — the providers
+  │   config is read once per process
+  └─ at start, then every Interval: checkOnce
+       │ every provider's Check concurrently, under one context.WithTimeout
+       │ snapshot → the scrape-time gauges
+       │ log transitions: down or reason changed → "WARNING: provider target
+       │   not ready"; back up → "Provider target ready"; steady → silent
+       └ per provider: APIReader.Get → Create if missing → status merge patch
+```
+
+- **One status write per provider per round**, since `lastCheckTime`
+  moves every round. At the 10s minimum interval that is still far below
+  leader-election renewals. Proxy status avoids writes like this only
+  because its probes run once a second per proxy.
+- **Reads go through the API reader**, not the cache. The objects are
+  cluster-scoped; a cached Get would start a cluster-wide informer (and
+  need `watch`) even with `--proxy-namespace`.
+- **Checks bypass the provider decorators.** `cmd/main.go` collects the
+  Checkers before wrapping the providers in `WithTracing(WithMetrics(...))`,
+  so a check never counts as a provider request. KVM RPCs still appear in
+  the V(1) `libvirt call` trace, and each round is one
+  `providerhealth.check` span.
+- **Not wired to `/readyz`.** A dead hypervisor must not make the manager
+  unready while the other providers keep working.
+- **Gauges** `proxy_operator_provider_target_up`,
+  `proxy_operator_provider_check_duration_seconds` and
+  `proxy_operator_provider_check_timestamp_seconds`, each labelled
+  `{provider,target}`. Only the leader exports them.
+
+Illustrative output:
+
+```sh
+$ kubectl get providerhealth          # short name: ph
+NAME         TYPE         READY   TARGETS   LAST CHECK   AGE
+kubernetes   kubernetes   True    1/1       12s          3d
+kvm-home     kvm          False   0/1       12s          3d
+$ kubectl get ph kvm-home -o yaml     # per target: reason, message, details
+```
+
 ## Decisions
 
 Judgment calls the spec left open, and deliberate deviations — recorded so
@@ -528,3 +589,27 @@ they read as choices, not accidents. Chronological by build step.
   external sites. The kube-apiserver ignores incoming `traceparent` by
   design (public endpoint), so k8s client spans are leaves — in-process
   traces, not cross-process ones.
+- **Provider health is a status-only, cluster-scoped CRD** (user
+  decision). `kubectl get providerhealth` works without Prometheus, which
+  the deployment does not ship (its ServiceMonitor is commented out). There
+  is no spec: the providers config file stays the one place providers are
+  configured, and the object only reports on it. The gauges carry the same
+  verdicts for alerting.
+- **KVM checks reuse the pooled connection instead of dialing fresh.** A
+  fresh dial would not pick up a rotated Secret anyway, since the SSH
+  material is cached. It would also open an sshd session per host per
+  minute. And a working cached connection is exactly what reconciles use.
+  Sharing must not cost the reconciles anything, so the check calls
+  through `hostPool.probe`. A check that runs out of time abandons its
+  RPCs instead of closing the connection, which would kill an upload or a
+  Create running on it. At most one abandoned check per host is left
+  waiting, and until it returns, the next check reports `Timeout` at once.
+  Waiting for a host's connection slot also ends with the caller's
+  deadline, so a reconcile redialing a dead host cannot stall the round.
+- **Provider transitions are logs, not Events**, in line with the
+  EventRecorder decision above. They log at Info with a `WARNING:` prefix,
+  and each target's `lastTransitionTime` in the object records when it
+  flipped.
+- **A provider name that is not a valid object name** gets metrics but no
+  `ProviderHealth` object, with a warning at start. Tightening config
+  validation instead would crash-loop existing deployments on upgrade.

docs/kvm-provider.md

@@ -265,12 +265,65 @@ The provider logs through controller-runtime's logger, named `kvm`, with
   only `length` and `duration`. There is no raw wire tier like the GCP
   provider's V(5); go-libvirt exposes no hook for it.
 
+## Health check
+
+The operator checks every configured host every `--provider-check-interval`
+(default 1m). It publishes the result in two places: the `ProviderHealth`
+object named after the provider (`kubectl get ph kvm-home -o yaml`), and the
+`proxy_operator_provider_target_up{provider,target}` gauge.
+
+All hosts are checked concurrently, each over the same pooled connection
+that Create and Get use:
+
+1. The SSH and libvirt handshakes. These only happen on the first check, or
+   after the link dropped.
+2. `virConnectGetLibVersion` and `virConnectGetHostname`.
+3. `virStoragePoolLookupByName` and `virStoragePoolGetInfo` on the host's
+   `pool`.
+
+A passing target's `details` carry `hostname`, `libvirtVersion`, `pool`,
+`poolState`, `poolCapacity` and `poolAvailable` (in GiB). The free space is
+the closest thing this provider has to a capacity signal. Every target also
+shows its SSH `user`. A failing target says where the check broke:
+
+| Reason | Meaning | Look at |
+|---|---|---|
+| `SSHMaterialUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*) |
+| `Unreachable` | the TCP connect failed, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address` |
+| `SSHHandshakeFailed` | the host key is missing from known_hosts or has changed, or `user` rejected the key | the known_hosts entry; `user`'s `authorized_keys` |
+| `LibvirtConnectFailed` | SSH worked, the libvirt socket did not | `user` in the `libvirt` group, `libvirtd` running, the `socket` path |
+| `Timeout` | no answer within `--provider-check-timeout` (default 15s), or the previous check of the host is still waiting | a hung `libvirtd`, a saturated link, or a reconcile stuck redialing the host |
+| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
+| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
+| `LibvirtError` | any other libvirt error | the message carries libvirt's own text |
+
+Sharing the connection has two consequences:
+
+- The leader now keeps one SSH connection per host open from startup, with a
+  keepalive every 15s. Before, the first connection opened with the first KVM
+  reconcile.
+- A check that runs out of time gives up on its RPCs but leaves the
+  connection alone, so an image upload or a Create running on it is not cut
+  off. At most one abandoned check per host keeps waiting; until it
+  returns, the next check reports `Timeout` straight away. A link that is
+  really dead is closed by the keepalive within about two keepalive
+  intervals.
+- A check never waits longer than its timeout for the host's connection,
+  even while a reconcile holds it to redial a host that is down.
+
 ## Known gaps
 
 - **No quota or capacity detection.** libvirt reports an out-of-space
   pool or an over-committed host as an ordinary error; the reconciler
   treats it as transient and retries with backoff rather than the slower
-  quota backoff.
+  quota backoff. The health check does report each pool's free space
+  (`poolAvailable`), so a pool that is filling up shows before Creates
+  start failing.
+- **A rotated `kvm-ssh` Secret needs a manager restart.** The key and
+  known_hosts files are read on first use and cached for the life of the
+  process. Replacing them in the Secret therefore changes nothing for
+  later connections, health checks included. After a host key change, the
+  check keeps reporting `SSHHandshakeFailed` until the manager restarts.
 - **Orphan volumes without a domain are invisible to GC.** A crash
   between creating the volumes and defining the domain leaves an overlay
   and seed that `ListByTag` cannot see (it lists domains). They are
@@ -315,4 +368,6 @@ Unit tests run over an in-memory fake of the libvirt API. The optional
 integration test against a real host is described in
 [testing.md](testing.md#the-kvm-integration-test-real-hypervisor); its
 `-v` output includes the `libvirt call` trace above, at the verbosity
-`KVM_TEST_LOG_LEVEL` selects (default 1).
+`KVM_TEST_LOG_LEVEL` selects (default 1). `TestKVMCheck_realHost` runs the
+health check against the same host and creates nothing:
+`go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/`.

docs/plans-executions/2026-09-11-1526-provider-health-checks.md

@@ -0,0 +1,378 @@
+# Execution: provider health checks
+Plan: [docs/plans/2026-09-11-1526-provider-health-checks.md](../plans/2026-09-11-1526-provider-health-checks.md)
+
+- [x] Step 0 — Save the plan
+- [x] Step 1 — ProviderHealth API and CRD
+- [x] Step 2 — Contract plus the kubernetes and GCP checks
+- [x] Step 3 — KVM check
+- [x] Step 4 — Monitor
+- [x] Step 5 — Metrics and wiring
+- [x] Step 6 — Docs
+- [x] Step 7 — Validate, record, push, open the PR
+
+## Step 0 — Save the plan
+
+Branched off `main` (at the PR #5 merge, `eb8f075`). The plan was copied out of
+`~/.claude/plans/` and committed on its own, before any code, as the repo's plan rules
+require:
+
+```bash
+git checkout -b feat/provider-health main
+cp ~/.claude/plans/now-i-would-like-flickering-wall.md docs/plans/2026-09-11-1526-provider-health-checks.md
+```
+
+The uncommitted `.claude/settings.json` change was left unstaged and stays out of this
+branch.
+
+## Step 1 — ProviderHealth API and CRD
+
+`kubebuilder` isn't installed here (only `bin/controller-gen`, `bin/kustomize` and
+`bin/setup-envtest` are), so `api/v1alpha1/providerhealth_types.go` and the `PROJECT`
+entry were written by hand instead of with `kubebuilder create api`. For a cluster-scoped
+resource `PROJECT` simply omits `namespaced`, which is what kubebuilder itself writes.
+
+The new types are registered in the existing `AddKnownTypes` call in `proxy_types.go`,
+not in a second `init()`.
+
+The deepcopy code, the CRD and the RBAC come from the usual generators:
+
+```bash
+make generate manifests
+go build ./... && go vet ./api/... && go test -short ./api/...
+```
+
+To confirm the new CRD and the viewer role actually reach the rendered install, I built
+the kustomization and grepped for them:
+
+```bash
+bin/kustomize build config/default | grep -n providerhealth
+# providerhealths.crawl.example.com (scope: Cluster, shortName ph)
+# egress-proxies-operator-providerhealth-viewer-role
+```
+
+Worth noting:
+- The only role added is the viewer role. The operator is the object's only writer, so
+  the admin and editor roles kubebuilder scaffolds for Proxy were not added.
+- The type file has no unit test, since it has no logic. The schema, including the
+  required `ready`/`lastTransitionTime` and the list-map keys, is proven by the envtest
+  round-trip in Step 4.
+- `config/rbac/role.yaml` is unchanged so far. The manager's own permissions arrive in
+  Step 4 with the Monitor's RBAC markers.
+
+## Step 2 — Contract plus the kubernetes and GCP checks
+
+`provider.Checker` is an optional interface in `internal/provider/check.go`, kept out of
+`Provider`. Each implementation has a compile-time assertion, because the monitor's
+type assertion would otherwise fail silently.
+
+- **kubernetes:** lists at most one managed Pod cluster-wide, using the same client, verb
+  and scope as `ListByTag`. The target is `apiserver`, at the `restCfg.Host` now stored
+  in the new `apiHost` field.
+- **GCP:** calls a new seam method, `Probe`. `realInstances.Probe` lives in `check.go`,
+  next to its only caller. `ListByTag`'s `Sprintf` filter became the shared
+  `managedFilter` constant, so the check and the GC sweep can't drift apart.
+
+Before relying on "MaxResults is only a page size", I checked the SDK's REST
+`AggregatedList` in the module cache:
+
+```bash
+grep -n "func (c \*instancesRESTClient) AggregatedList" -A80 \
+  "$(go list -m -f '{{.Dir}}' cloud.google.com/go/compute)"/apiv1/instances_client.go
+# InternalFetch: req.MaxResults = pageSize ... return elems, resp.GetNextPageToken(), nil
+```
+
+```bash
+go build ./... && go vet ./internal/provider/... && go test -race -count=1 ./internal/provider/...
+```
+
+Worth noting:
+- The planned `TestRealInstances_Probe_fetchesOnePage` became a three-case
+  `TestRealInstances_Probe` table.
+  - It keeps the first page's `nextPageToken`, and asserts exactly one request with
+    `maxResults=1` for each case.
+  - "empty project" covers `iterator.Done`.
+  - "api error keeps its status" proves the real client returns a `*googleapi.Error`, so
+    a disabled Compute API is reported as `Forbidden` rather than `Unreachable`.
+- Once a context has ended, both checks report `Timeout` whatever the error says. The
+  monitor's timeout is the usual cause, and the underlying error only repeats it.
+
+## Step 3 — KVM check
+
+`conn.go` now wraps each of the four dial stages in a `dialError`: ssh-material, tcp,
+ssh-handshake and libvirt-connect. `Error()` returns the inner message unchanged and
+`Unwrap()` returns the cause, so `classify`/`isClassified` and every existing
+message-matching test behave as before. `TestDialError_keepsClassification` pins this
+down.
+
+The three new RPCs went into `libvirtAPI`, `tracingAPI` and `fakeAPI`. The method-count
+guard in `TestTracingAPI_logsEveryMethod` fails until a traced row exists for each new
+method, which is what forced the three new rows.
+
+`Check` runs one goroutine per host, over the shared connection, through
+`hostPool.call`. The pool verdict is decided only after `call` returns, because
+returning it as an error from inside the callback would make `call` drop a perfectly
+good connection.
+
+Build, vet (also with the integration tag, since that file changed) and race tests:
+
+```bash
+go build ./internal/provider/... && go vet ./internal/provider/kvm/ \
+  && go vet -tags=integration ./internal/provider/kvm/ \
+  && go test -race -count=1 ./internal/provider/kvm/
+```
+
+Worth noting:
+- **Pool state wording:** `poolState` uses virsh's words (`running`, `inactive`, …)
+  rather than go-libvirt's constant names, so it reads like `virsh pool-list` output.
+- **Details:** each host also reports its SSH `user`. That is the first thing to check
+  when the reason is `SSHHandshakeFailed`.
+- **Tests renamed or merged:**
+  - The planned `TestSSHAuth_dial_stages` is `TestLazySSH_dial_tagsTheFailingStage`. It
+    dials through `lazySSH`, so the ssh-material stage is covered as well as the three
+    stages the in-process SSH server can trigger.
+  - The pool-inactive and pool-missing cases are merged into `TestCheck_libvirtVerdicts`,
+    which also covers a daemon error. Each case runs two checks and asserts a single
+    dial.
+- **Tests added beyond the plan:**
+  - `TestCheck_timeout`: a daemon that stops answering gives `Timeout`.
+  - `TestDialError_keepsClassification`.
+- **Not run yet:** `TestKVMCheck_realHost` in the integration file only compiles here.
+  It needs beelink, so it goes on the Step 7 checklist for the user to run.
+
+## Step 4 — Monitor
+
+`internal/providerhealth` has the `Monitor` runnable plus the pure `buildStatus`.
+
+- **Scheduling:** the Monitor is leader-only. It cleans up once at start, checks
+  immediately, then checks on a ticker.
+- **Each round:** providers are checked concurrently under one timeout, then logged and
+  published one after another. A failed write is logged, and the next round writes
+  again.
+- **Reads:** objects are read through the reader it is given. `main.go` passes the
+  manager's API reader (Step 5).
+
+The first `make manifests` left `config/rbac/role.yaml` unchanged, with no error.
+I had put the `+kubebuilder:rbac` markers inside `Monitor`'s doc comment, and
+controller-gen only reads RBAC markers from package-level comment groups. So they were
+silently attached to the type and ignored. An empty `git diff config/rbac/role.yaml` was
+the only sign. Moving them into their own comment block fixed it:
+
+```bash
+make manifests && git diff config/rbac/role.yaml
+# + providerhealths: create, delete, get, list
+# + providerhealths/status: get, patch
+```
+
+Vet, race tests and the envtest round-trip (using the `bin/k8s/1.36.2-linux-amd64`
+binaries `make setup-envtest` had already fetched):
+
+```bash
+go vet ./internal/providerhealth/ && go test -race -count=1 -v ./internal/providerhealth/
+# --- PASS: TestMonitor_envtestRoundTrip (6.45s) ... ok
+```
+
+Worth noting:
+- **Fake client scope:** the fake client needs a RESTMapper that registers
+  ProviderHealth as `RESTScopeRoot`. Its default mapper treats every kind as namespaced.
+- **Why the envtest calls `publish` directly:** `checkOnce` only logs write errors, so a
+  schema rejection would look like a missing object. Calling `publish` makes the API
+  server's answer the test failure.
+  - Its first round writes a 2000-character message; the stored message is truncated to
+    1024.
+  - Its second round proves that only the target that flipped gets a new
+    `lastTransitionTime`.
+- **No guard against a hung checker:** the Monitor trusts `Checker`'s contract to return
+  once the context ends, and doesn't abandon a checker that ignores it. All three
+  checkers stop when the context ends: KVM through `awaitOrCancel`, and kubernetes and
+  GCP through their clients.
+- **Test renamed:** the planned `TestMonitor_timeoutBoundsBlockingChecker` is
+  `TestMonitor_checkOnce_timeoutBoundsEachRound`. It also asserts that a fast provider
+  still publishes next to a hung one.
+
+## Step 5 — Metrics and wiring
+
+`metrics.RegisterProviderTargets` registers a scrape-time collector for three gauges,
+each labelled `{provider,target}`: `up`, check duration and check timestamp.
+`Register`'s signature is unchanged.
+
+`cmd/main.go`:
+- gains `--provider-check-interval` (1m) and `--provider-check-timeout` (15s);
+- validates them before even loading the providers config;
+- collects the `Checker`s from the unwrapped providers in config order, just before the
+  metrics and tracing wrap loop;
+- adds the Monitor with the manager's API reader;
+- registers the collector through a small `providerTargetSamples` closure next to
+  `proxyPhaseCounts`.
+
+Build, vet and metrics tests:
+
+```bash
+go build ./... && go vet ./cmd/... ./internal/metrics/ && go test -race -count=1 ./internal/metrics/
+```
+
+To confirm bad flags fail fast, before anything touches the cluster, I ran the binary
+with a 5s interval. zap's development mode prints a stack trace after the error, hence
+the grep:
+
+```bash
+go run ./cmd --providers-config=/dev/null --provider-check-interval=5s 2>&1 | grep -m1 "Invalid provider check flags"
+# ERROR setup Invalid provider check flags {"error": "--provider-check-interval must be 0 (off) or at least 10s, got 5s"}
+go vet -tags=e2e ./test/e2e/
+```
+
+The e2e suite gets a new ordered spec, "should publish provider health". It waits for
+`providerhealth kubernetes` to report `Ready=True`. It runs before the existing metrics
+spec, which now also asserts
+`proxy_operator_provider_target_up{provider="kubernetes",target="apiserver"} 1`. By then
+the first round is known to be done, so the series is certain to exist.
+
+Worth noting:
+- **Main-only wiring:** `providerTargetSamples` is an 8-line loop in `package main`,
+  which has no test file. The e2e metrics assertion is its only coverage.
+- **The KVM entry isn't asserted in e2e:** the default providers config also ships
+  `kvm-home` (beelink), which would report `SSHMaterialUnavailable` in kind. That entry
+  is the user's own host, and asserting it would break the suite as soon as the default
+  config changes.
+- **Manifest:** `config/manager/manager.yaml` only gained a comment about the two flags.
+  The defaults need no args.
+
+## Step 6 — Docs
+
+- **`docs/architecture.md`:**
+  - component rows for the CRD, the `Checker` contract and the monitor;
+  - a new §11 covering the provider table, the loop diagram, the write, read and
+    decorator choices, the gauges, and sample `kubectl get ph` output;
+  - cross-references from the metrics (§9) and tracing (§10) sections;
+  - four Decisions entries.
+- **`docs/kvm-provider.md`:**
+  - a "Health check" section with the reason → meaning → where-to-look table and the
+    two consequences of sharing the connection;
+  - under Known gaps, the capacity bullet now points at `poolAvailable`, and a new
+    bullet says a rotated `kvm-ssh` Secret needs a restart;
+  - the Testing section names `TestKVMCheck_realHost`.
+- **`docs/testing.md`:**
+  - suite rows for the three `check_test.go` files, `internal/providerhealth` (unit and
+    envtest) and the metrics gauges;
+  - "17 libvirt calls" is now 20;
+  - the integration and e2e sections describe the new tests.
+- **`CLAUDE.md`:**
+  - the Architecture TODO now points at `docs/architecture.md`;
+  - the stale "only `go.mod` exists" gotcha is replaced by a note that the "What this
+    is" and "Commands" sections predate the code, plus Step 4's RBAC-marker gotcha.
+
+Worth noting:
+- `docs/api.md` is untouched. It documents the discovery HTTP API, which this change
+  doesn't alter.
+- `CLAUDE.md`'s "What this is" and "Commands" sections were left stale on purpose.
+  Rewriting them is outside this plan, so the new gotcha flags them instead of leaving
+  them silently wrong.
+
+## Step 7 — Validate, record, push, open the PR
+
+The full suite runs codegen, `go fmt` and `go vet`, then `go test -race` on every
+package, with both envtest suites included:
+
+```bash
+make test
+# every package ok; internal/providerhealth 92.5 %, internal/provider/kvm 92.0 % coverage
+```
+
+`go fmt` rewrote nothing, and `git status` showed only the Step 6 docs.
+
+`make lint` could not run, for a reason unrelated to this branch. The Makefile rebuilt
+the custom golangci-lint, which reports "built with go1.27.0", and its bundled
+staticcheck (honnef.co/go/tools v0.7.0) panics on the standard library before it
+reaches any repo package:
+
+```bash
+make lint
+# buildir: package "poll" (isInitialPkg: false, needAnalyzeSource: true): unexpected expr: *ast.KeyValueExpr
+# make: *** [Makefile:107: lint] Error 3
+```
+
+The local toolchain is go1.27.0 while `go.mod` says 1.26.0; the otel-tracing plan's
+log shows `make lint` running normally before the toolchain moved. Retrying with
+`GOTOOLCHAIN=go1.26.0` would have meant downloading a toolchain, and the user declined
+that. `go vet` is clean in all three configurations: plain inside `make test`, plus
+`-tags=integration` and `-tags=e2e`.
+
+The kind e2e suite creates and deletes its own `egress-proxies-operator-test-e2e`
+cluster. I checked `KIND_CLUSTER` first, because the two clusters already on the
+machine (`kind`, `proxy-operator-demo`) must not be torn down by `cleanup-test-e2e`:
+
+```bash
+make test-e2e > <scratch>/e2e.log 2>&1
+# Ran 3 of 5 Specs in 86.210 seconds
+# SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 2 Skipped
+```
+
+In the deployed manager, the `kubernetes` ProviderHealth reported `Ready=True` about a
+second after the pod was up (20:30:05.6 → 20:30:06.7). The metrics scrape that followed
+contained `proxy_operator_provider_target_up{provider="kubernetes",target="apiserver"} 1`.
+That exercises the generated RBAC, the API-reader path and the `main.go` gauge wiring
+end to end. The two skipped specs are the Tempo-gated tracing specs; `TEMPO_URL` was not
+set.
+
+### Follow-up — the check must not cost the reconciles their connection
+
+Reviewing the shared-connection decision after the automated validation turned up two
+ways the health check could hurt the work it shares the link with, both of which the
+plan's "a check against a hung host drops the shared connection" line accepted too
+easily:
+
+- A check that timed out went through `hostPool.call`, whose cancel path drops the
+  connection. A `libvirtd` slow to answer one `ConnectGetLibVersion` would therefore
+  cut off an image upload or a `Create` running on the same link.
+- `hostEntry` was guarded by a `sync.Mutex`, so waiting for a host's slot could not be
+  interrupted. A reconcile redialing a host that is down holds the slot for as long as
+  the SSH dial takes — minutes on a blackholed address — and the whole check round
+  would wait behind it, past its own timeout.
+
+So the check now calls `hostPool.probe` instead of `call`: on `ctx` ending it abandons
+the RPC and leaves the connection alone (a link that is really dead is closed by the
+keepalive anyway, which ends the abandoned `fn` too). One abandoned probe per host is
+allowed to stay waiting — `hostEntry.probing` is an `atomic.Bool`, not a counter, so a
+daemon that never answers cannot pile up goroutines; further probes get the new
+`errProbeBusy`, which `checkReason` maps to `Timeout`. `hostEntry.mu` became a one-slot
+channel with an `acquire(ctx)` that only bounds *waiting*: an uncontended caller takes a
+free slot even with an expired context, so `call`'s behaviour is unchanged where nothing
+is contended.
+
+The second change came out of the same pass: `--provider-check-timeout` defaulted to a
+flat 15s while the flag validation rejected a timeout longer than the interval, so
+`--provider-check-interval=12s` alone failed to start. The flag now defaults to 0 and
+`Monitor.setDefaults` derives `min(15s, Interval)`; `ValidateSchedule` accepts 0 and
+rejects only negatives and timeouts past the interval.
+
+Re-verified with the same commands as above:
+
+```bash
+go vet ./...
+go test -race -count=1 ./internal/provider/kvm/ ./internal/providerhealth/
+make test
+# every package ok; internal/provider/kvm 92.2 %, internal/providerhealth 93.7 % coverage
+```
+
+Worth noting:
+- `make lint` still cannot run, for the toolchain reason recorded above; unchanged by
+  this follow-up.
+- The two new `kvm` tests are timing-shaped by necessity — they assert that a caller
+  with a 50ms deadline does not wait out a dial that is blocked on a channel. The
+  assertions are one-sided (`> 1s` fails) so a slow machine does not make them flaky.
+- `docs/architecture.md`, `docs/kvm-provider.md` and `docs/testing.md` were corrected in
+  the same commit: the old text promised the drop-on-timeout behaviour that is now gone.
+- The real-hypervisor commands from the plan's Verification section are still the user's
+  to run; nothing here replaces them.
+
+### Real-hypervisor validation and PR
+
+The user ran the plan's real-hypervisor checks against beelink (the `TestKVMCheck`
+integration test plus the manual `ProviderHealth` and metrics checks) and confirmed
+they passed. After that the CHANGELOG entry went in, and the branch was pushed with
+the token-file credential helper. The PR against `main` was opened through the Gitea
+API because `tea` isn't installed.
+
+Worth noting:
+- The branch also carries `docs/plans/2026-09-16-0841-provider-blueprint.md`, on
+  purpose: that plan reaches `main` with this merge, and the blueprint work starts on a
+  new branch afterwards.

docs/plans-executions/2026-09-16-0841-provider-blueprint.md

@@ -0,0 +1,177 @@
+# Execution: provider blueprint
+Plan: [docs/plans/2026-09-16-0841-provider-blueprint.md](../plans/2026-09-16-0841-provider-blueprint.md)
+
+- [x] Step 0 — Save the plan; wait for the merge
+- [x] Step 1 — Collect evidence (read-only, in parallel)
+- [x] Step 2 — Write `docs/providers.md`
+- [x] Step 3 — Verify, then open the MR
+
+## Step 0 — Save the plan; wait for the merge
+
+The plan was committed on `feat/provider-health` (`80b2e9c`) and reached `main`
+through PR #6. After the user confirmed the merge, the branch was cut from the
+updated `main`:
+
+```bash
+git switch main && git pull --ff-only
+git switch -c feat/provider-blueprint
+```
+
+Worth noting: `.claude/settings.json` has local, uncommitted permission edits
+that carried over through the switch. They are deliberately left out of this
+branch.
+
+## Step 1 — Collect evidence (read-only, in parallel)
+
+Three Explore agents answered the plan's 12-item questionnaire, one each for
+kubernetes, gcp and kvm. Meanwhile I read the shared layer:
+`internal/provider/*.go`, `registry.go`, the provider wiring in `cmd/main.go`,
+`reconcileManaged`/`replaceInstance`/`reconcileDelete` in the controller,
+`internal/gc/gc.go`, and how the monitor uses `Checker`.
+
+Before relying on the agents' claims, I spot-checked them against the code. One
+was wrong and did not go into the doc: the kvm agent said `Delete` skips volume
+removal when `DomainDestroy` returns `ErrNoDomain`. In fact `removeDomain`
+returns nil there, and `Delete` still calls `removeVolumes` afterwards
+(`kvm.go:477-482`).
+
+Rules that exist only in the callers, and not in any comment, were found by
+reading the controller and GC. They went into the doc as MUST rules:
+- A zero `CreatedAt` makes an instance immediately collectable (`gc.go:120`).
+- `Get` returning `ErrPermanent` on the deletion path makes the reconciler call
+  `Delete` anyway.
+- Replacement waits for `Get` to return NotFound before creating an instance
+  with the same name.
+- `Stopped` and `Terminated` both lead to delete-and-recreate.
+
+Worth noting, found outside this plan's scope and not changed here:
+- `docs/gcp-vm-validation.md` filters on `labels.proxy-operator-managed=yes`,
+  but the value is `true`.
+- According to the gcp agent, `docs/gcp-in-specific-project.md` holds a real
+  project ID, IAM output and an egress IP. The user was told.
+
+## Step 2 — Write `docs/providers.md`
+
+Wrote the eight sections the plan lists. Linked the new doc from the Provider
+contract row in `docs/architecture.md` and from the CLAUDE.md "Architecture"
+section.
+
+Judgment calls:
+- The file:line evidence sits in per-provider notes under the matrix, as
+  planned. The contract rules in §2 cite the reconciler and GC lines each rule
+  protects.
+- The gap list includes documentation drift (kvm-provider.md's ARP order and
+  log keys, gcp-vm-validation.md's label value and its claim that async errors
+  reach status), as well as code gaps. Both would mislead the next
+  provider author in the same way.
+- "Proposed features" stays a flat list with no design, as the user decided.
+
+Worth noting: the line numbers in the doc are a snapshot. The doc says so and
+asks readers to look for the named function when a line has moved.
+
+## Step 3 — Verify, then open the MR
+
+An Explore agent re-checked the doc against the code: about 150 claims,
+including every matrix cell. It found four mismatches, and all four were fixed:
+- **Startup dial:** I had listed kvm-provider.md's "connection kept open from
+  startup" as doc drift. It isn't: the pool dials lazily, but the provider
+  health monitor runs a check as soon as the leader starts
+  (`monitor.go:147`, default interval 1m), and that check opens the connection.
+  The bullet was removed and the kvm notes now say this.
+- **Checker wording:** §5 said the monitor type-asserts `Checker`. It is
+  `cmd/main.go:216`; the monitor only sees nil or not.
+- **Reasons, not conditions:** §3 called `QuotaExceeded` and `PermanentError`
+  conditions. They are reasons on `Provisioned`.
+- **Latch lines:** the line range is `:133-139`, not `:130-135`.
+
+It also noted that none of the providers assert `var _ provider.Provider`, and
+that the kubernetes provider's RBAC markers live in the controller package. The
+§7 checklist now says both.
+
+Link check (every relative `](…)` target and anchor in the new and edited docs
+resolves), then the sanity build:
+
+```bash
+go build ./... && go test -short ./...
+# all packages ok
+```
+
+Worth noting: no CHANGELOG entry yet. Per CLAUDE.md it goes in after the user
+confirms the doc.
+
+### Follow-up — tracing was missing from the blueprint
+
+The user pointed out, after the PR was opened, that the blueprint said almost
+nothing about tracing. The plan's questionnaire item 10 had collected the
+evidence, but the doc reduced it to a matrix row about logging and a checklist
+line telling authors not to open spans. That line even suggests tracing needs
+no work. A provider author needs the opposite message: the operation span comes
+free, and child spans for backend calls are theirs to add.
+
+Added a new §6, "Observability: tracing, logs and metrics", and renumbered the
+matrix, checklist and gaps sections to §7–§9. It covers:
+- **Three layers:** decorator operation spans; child spans per backend call,
+  with parent-gated HTTP transports or a span decorator over the SDK seam; and
+  logs.
+- **The ctx and logger rules** that keep `traceID` on log lines.
+- **A per-provider status table:**
+  - kubernetes: parent-gated `RestConfigWrapper` spans.
+  - gcp: the Google SDK's own `otelhttp`, never disabled by the provider.
+  - kvm: no spans. `trace.go` is a log decorator.
+
+Also added two tracing rows to the §7 matrix, rewrote checklist item 8 as
+"Tracing and logging" with an in-memory span-recorder test, and added gaps (kvm
+has no child spans; only kubernetes is checked against Tempo, and only at
+operation level). A proposal notes that the kvm `libvirtAPI` seam takes no
+`ctx`, so per-RPC spans need that plumbed in.
+
+The links were re-checked, including the new `architecture.md#10-tracing-internaltracing` anchor.
+
+### Follow-up — a completeness sweep
+
+The user asked for one more pass over everything provider-related. After the
+tracing miss, this pass read beyond the provider packages:
+- the Proxy API types and the spec hash;
+- the health engine and discovery call sites that consume `Instance.IP`;
+- the manager Deployment, the providers ConfigMap and the samples;
+- README, testing.md, kvm-provider.md and the GCP docs;
+- the initial prompt, the proxy-operator and kvm plans, and the 2026-08-10 PR
+  review;
+- the CHANGELOG.
+
+What the blueprint was missing, now added:
+- **What the instance is for.** The IP and port must be reachable from the
+  operator's health probes and from discovery clients. Providers don't know
+  which proxy runs; the proxy comes from cloud-init, except for kubernetes.
+- **Concurrency.** Three reconciles, GC and checks can call a provider at once,
+  so it MUST be safe for concurrent use.
+- **The runtime envelope:** read-only root filesystem, a 64Mi `/tmp`, a 128Mi
+  memory limit, credentials as optional mounts.
+- **Create:** `placement` can't be validated by the CRD, so error messages must
+  name the field. Catalogue edits in config aren't in the spec hash.
+- **Delete** is re-called every `DeletionPoll` while a delete is in progress.
+- **Provider names** must be DNS-1123 for `ProviderHealth`, and `spec.provider`
+  is immutable.
+- **Metric names** in §6.
+- **Three matrix rows:** proxy software, IP reachability, distinct egress.
+- **A deployment checklist item:** credentials mount, config entries, sample
+  Proxy, Makefile helper.
+- **Gaps:**
+  - still open from the PR review: GC has no installation identity; an
+    unconfigured provider latches Proxies; kubernetes Pod RBAC always installed
+    and no `OwnerReference` on Pods;
+  - the duplicated type list;
+  - samples that can't work: `proxy_gcp.yaml` uses a debian-cloud image without
+    cloud-init, and `proxy_kvm.yaml` places on `nas1` where configs define
+    `beelink`;
+  - the `"mock"` example in the `spec.provider` doc.
+
+Outside the plan's "leave other docs alone" line, README.md changed:
+- Its "Adding a provider" section restated the contract, with the old "Create
+  never blocks" rule that kvm contradicts. It now points to `docs/providers.md`
+  instead of keeping a second copy that disagrees.
+- The "Implemented providers" table gained the missing kvm row.
+- The deploy note no longer claims the default ConfigMap ships only the
+  kubernetes provider.
+
+The links were re-checked for providers.md and README.md.

docs/plans-executions/2026-09-16-1108-provider-docs-split.md

@@ -0,0 +1,228 @@
+# Execution: Split the provider blueprint into contract, appendix, template and index
+
+Plan: [docs/plans/2026-09-16-1108-provider-docs-split.md](../plans/2026-09-16-1108-provider-docs-split.md)
+
+- [x] Step 0 — Save and commit the plan
+- [x] Step 1 — Write `docs/provider-contract.md`
+- [x] Step 2 — Write `docs/provider-spec-template.md`
+- [x] Step 3 — Write `docs/providers-existing.md`
+- [x] Step 4 — Rewrite `docs/providers.md` as the index; link edits
+- [x] Step 5 — Verification sweep
+- [x] Step 6 — Final summary, CHANGELOG after confirmation
+
+## Step 0 — Save and commit the plan
+
+Committed as `cf89ac4` on the existing `feat/provider-blueprint` branch, on
+its own, before any document work. The pre-existing local modification of
+`.claude/settings.json` was left out of the commit.
+
+## Step 1 — Write `docs/provider-contract.md`
+
+2026-09-16 11:17. Written from scratch rather than edited down from the old
+blueprint, so that nothing provider-specific could survive by accident. Every
+MUST names its caller by function (`reconcileManaged`, `replaceInstance`,
+`reconcileDelete`, `providerFailure`, `deletionFailure`, `Sweeper.sweep`,
+`Monitor.checkOnce`, `Monitor.run`, `registry.Build`) and states what breaks.
+
+**Axis codes frozen** (19, §4.1 to §4.19): `readiness`, `create-return`,
+`delete-return`, `ip-source`, `gc-mark`, `created-at`, `list-partial`,
+`quota`, `not-found`, `primary-key`, `provisioning-state`, `async-ops`,
+`secondary-resources`, `credentials`, `client-kind`, `timeouts`, `logging`,
+`child-spans`, `backend-test`. The appendix table and the template are keyed
+on these strings; the §4.20 summary table's last column gives the template
+entry number each axis maps to.
+
+**Two rules that were not in the old blueprint**, both found by reading the
+callers while verifying:
+
+- `Sweeper.sweep` loops providers sequentially in one goroutine, and no
+  caller puts a deadline on the context (only `Monitor.checkOnce` uses
+  `WithTimeout`). So a hung `ListByTag` stalls orphan GC for every provider.
+  Recorded in §1.5 and the `timeouts` axis.
+- The Stopped branch of `reconcileManaged` calls `Delete` and requeues
+  without clearing `ProviderID`, then deletes again on every poll until
+  NotFound. A backend that reports a freshly created, never started instance
+  as stopped would have it deleted right after creation. Recorded as a MUST
+  in §2.2 and as the `provisioning-state` axis with three allowed profiles.
+
+Checks run on the draft (all clean):
+
+```sh
+grep -niE "kubernetes|gcp|kvm|libvirt|squid|google|compute engine" docs/provider-contract.md
+grep -n "—" docs/provider-contract.md
+grep -nE '\.go:[0-9]+|:[0-9]+-[0-9]+' docs/provider-contract.md
+grep -ohE "(internal|cmd|config|docs|api|hack|test)/[A-Za-z0-9_./-]+" docs/provider-contract.md | sort -u | while read p; do test -e "$p" || echo "MISSING $p"; done
+```
+
+The path check reports three expected misses: `config/samples/proxy_<type>.yaml`
+is a pattern, `docs/provider-spec-template.md` is written in Step 2, and
+`internal/provider/providertest` is a proposal named in §13.
+
+Worth noting: the contract refers to "the operator pod" and "the cluster's
+own API" rather than naming the platform, so that the abstractness grep stays
+meaningful. The `hack/providers-dev.yaml` rule is phrased as "only if the type
+needs no credentials" because that file exists for `make run-dev` without any
+Secret.
+
+## Step 2 — Write `docs/provider-spec-template.md`
+
+2026-09-16 11:28. Twenty-four entries in a fixed shape (Question, Allowed
+answers, Obligations, Answer, Evidence). Thirteen of them carry an axis code
+and their numbers are the ones the contract's §4.20 table points at; the
+other eleven (identity, placement, image, cloud-init, machine type, config
+block, Checker, concurrency, deployment, docs, accepted limits) are decisions
+the contract leaves open without being a profile axis. Entries 9, 13 and 15
+carry an empty table to fill in (state mapping, error classification, config
+fields); entry 21 carries the contract §11 test list as checkboxes so a
+filled spec doubles as the test plan.
+
+Check that every axis code appears in the template:
+
+```sh
+for c in $(grep -oE '^### 4\.[0-9]+ `[a-z-]+`' docs/provider-contract.md | grep -oE '`[a-z-]+`' | tr -d '`'); do
+  grep -q "\`$c\`" docs/provider-spec-template.md || echo "template missing $c"
+done
+```
+
+Clean. Worth noting: the preamble allows "verify against the target version"
+only on an Evidence line, never as an Answer, because the whole point of the
+file is that the implementer does not guess.
+
+## Step 3 — Write `docs/providers-existing.md`
+
+2026-09-16 11:35. The profile table has exactly one row per contract axis
+code, checked by the same loop as Step 2 with a row count. Everything else
+comes from the old blueprint's §6, §7 and §9 plus a fresh read of the three
+packages, with every `file:line` replaced by a symbol name.
+
+Corrections to earlier notes applied while writing: kvm's Checker `Details`
+also carry `user` and `pool`; the kubernetes provider has no seam interface
+(`newWithClient` injects a `client.Client`, the fake client stands in); kvm's
+`classify` has an `isClassified` pass-through. Timeouts were checked per
+provider for the `timeouts` row: kubernetes sets no `rest.Config.Timeout`,
+gcp relies on SDK defaults, kvm bounds only the SSH handshake
+(`defaultHandshakeTimeout` in `dialSSH`), so all three are weaker than the
+contract's SHOULD and the table says so.
+
+Symbol check on the appendix (every backticked identifier grepped for in
+Go, Makefile and YAML sources):
+
+```sh
+grep -ohE '`[A-Za-z_][A-Za-z0-9_.]*`' docs/providers-existing.md | tr -d '`' | sort -u | while read s; do
+  grep -rqwF --include='*.go' --include='Makefile' --include='*.yaml' -- "$s" . || echo "$s"
+done
+```
+
+The residue is file names, dotted `Type.method` forms, the `timeouts` axis
+code, libvirt's `nostate`, and the API names `OwnerReference` and
+`rest.Config.Timeout`; nothing unexpected.
+
+Worth noting: matrix rows that the profile table now carries (Create
+non-blocking, GC marks, IP source, quota detection, Checker presence,
+credentials, logging, tracing, tests) were dropped from "Other differences"
+rather than kept twice.
+
+## Step 4 — Rewrite `docs/providers.md` as the index; link edits
+
+2026-09-16 11:40. `docs/providers.md` was rewritten in place (same path, so
+the inbound links from `README.md`, `CLAUDE.md`, `docs/architecture.md` and
+the historical plan files still resolve, and the pre-split blueprint stays
+reachable in the path's git history). The three inbound sentences were
+updated to name the new files:
+
+- `README.md` "### Adding a provider": the first sentence now names the
+  contract, the template and the appendix, with `docs/providers.md` as the
+  index; the "Start there" sentence is unchanged.
+- `CLAUDE.md` "## Architecture": the providers.md sentence now names all
+  three files and the index.
+- `docs/architecture.md` components row "Provider contract": "rules in
+  provider-contract.md, per-provider profile table in providers-existing.md".
+
+The historical files under `docs/plans/` and `docs/plans-executions/` keep
+their `providers.md` links on purpose.
+
+## Step 5 — Verification sweep
+
+2026-09-16 11:45. All ten checks from the plan, run against the working tree
+after Step 4; nothing needed fixing.
+
+Links (file targets and anchors) across the four new files plus `README.md`,
+`CLAUDE.md` and `docs/architecture.md`, with anchors compared against
+slugified headings of the target file:
+
+```sh
+python3 - <<'PY'
+import re, pathlib
+files = ["docs/provider-contract.md","docs/provider-spec-template.md","docs/providers-existing.md","docs/providers.md","README.md","CLAUDE.md","docs/architecture.md"]
+def slug(h):
+    h = re.sub(r'`','',h).strip().lower(); h = re.sub(r'[^\w\s-]','',h)
+    return re.sub(r'\s+','-',h).strip('-')
+def headings(p):
+    return {slug(m.group(2)) for m in (re.match(r'^(#{1,6})\s+(.*)$', l) for l in open(p)) if m}
+bad = 0
+for f in files:
+    for m in re.finditer(r'\]\(([^)\s]+)\)', open(f).read()):
+        t = m.group(1)
+        if t.startswith(('http://','https://','mailto:')): continue
+        path, _, anchor = t.partition('#')
+        target = pathlib.Path(f) if path == '' else pathlib.Path(f).parent/path
+        if not target.exists(): print(f, "missing", t); bad += 1; continue
+        if anchor and target.suffix == '.md' and anchor not in headings(target): print(f, "bad anchor", t); bad += 1
+print("link problems:", bad)
+PY
+```
+
+Output: `link problems: 0`.
+
+Em-dashes, line references and the contract's abstractness:
+
+```sh
+grep -n "—" docs/provider-contract.md docs/provider-spec-template.md docs/providers-existing.md docs/providers.md
+git diff 2d959af -- README.md CLAUDE.md docs/architecture.md | grep '^+' | grep "—"
+grep -nE '\.go:[0-9]+|:[0-9]+-[0-9]+' docs/provider-contract.md docs/provider-spec-template.md docs/providers-existing.md docs/providers.md
+grep -niE "kubernetes|gcp|kvm|libvirt|squid|google|compute engine" docs/provider-contract.md
+```
+
+The only em-dash hit is the pre-existing "Leader-elected" cell of the
+`architecture.md` components row, which shows as an added line because the
+row's last cell was edited; no new em-dash was written. The other three
+greps are empty.
+
+Nothing lost from the old blueprint's §9: the bold leads of every gap and
+proposal bullet in `git show 0bb699b:docs/providers.md` were looked up in
+the new files. The literal-prefix match reported 22 misses, all caused by
+rewording ("isn't" to "is not", bold removed), so each topic was re-checked
+by a distinctive phrase and all 22 are present in `providers-existing.md`
+or contract §13.
+
+Paths and symbols: every `path/...` token in the four files exists except
+the `config/samples/proxy_<type>.yaml` pattern and the
+`internal/provider/providertest` proposal; every backticked identifier in
+the contract and template resolves in Go, Makefile or YAML sources except
+file names, dotted `Type.method` forms, profile names and the `NewWith...`
+placeholder.
+
+Axis alignment: the 19 axis codes from contract §4 each appear once as a
+row in the appendix table (six cells per row) and at least once in the
+template.
+
+`go build ./... && go test -short ./...`: all packages `ok`.
+
+## Step 6 — Final summary
+
+The split is complete on `feat/provider-blueprint`: five commits after the
+plan commit, one per step. The CHANGELOG entry is deliberately not written
+yet; per `CLAUDE.md` it is added once the user confirms the documents read
+the way they want. Nothing has been pushed.
+
+Worth noting for whoever picks this up:
+
+- The contract's §4.20 table maps axis codes to template entry numbers; if
+  entries are renumbered, that column must follow.
+- The `CHANGELOG.md` heading format in `CLAUDE.md` uses an em-dash; the
+  "no em-dashes" rule applied to the four new documents only.
+- The stale `mock` comments in `internal/provider/provider.go`,
+  `internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
+  are still listed as gaps, not fixed here.
+- The next planned use of these files is a Proxmox provider: copy the
+  template to `docs/proxmox-provider-spec.md` and fill it in before any code.

docs/plans/2026-09-11-1526-provider-health-checks.md

@@ -0,0 +1,354 @@
+# Plan: Provider health checks — see whether the cluster can reach each provider
+**Created:** 2026-09-11 15:26 UTC
+
+## Context
+
+Today the operator can't say whether a provider's backend is reachable. The `Provider`
+interface ([provider.go:89-109](internal/provider/provider.go#L89-L109)) only has
+Create/Get/Delete/ListByTag, and `/healthz` and `/readyz` are bare `healthz.Ping`. A
+broken KVM path only shows up later: as a Proxy stuck in Provisioning, or as an orphan-GC
+sweep logging an error. Several failures stay invisible until a Proxy needs that host:
+
+- a missing `kvm-ssh` Secret;
+- a rotated host key;
+- a Tailscale route that is down;
+- a storage pool that is inactive or full.
+
+The goal is a periodic sanity check of every configured provider from inside the cluster,
+with results you can see through `kubectl` and Prometheus.
+
+Decisions made by the user during planning:
+
+1. **Surface:** a new operator-owned, read-only, cluster-scoped **`ProviderHealth`** CRD,
+   with one object per configured provider, a `Ready` condition and per-target detail.
+   Also Prometheus gauges, and a log line whenever a target goes up or down. No
+   EventRecorder, matching the earlier decision.
+2. **Scope:** real checks for all three providers.
+   - **kvm:** checked per host.
+   - **kubernetes:** confirms the API server and RBAC work.
+   - **gcp:** confirms credentials and the API work.
+3. **KVM depth:** reachability plus the storage pool.
+   - Over the existing pooled SSH+libvirt connection, call the libvirt version and
+     hostname RPCs.
+   - Confirm the configured pool is running and report its free space.
+   - No bridge check.
+
+Not affected: the Proxy reconcile behaviour, and the manager's `/healthz` and `/readyz`.
+A dead hypervisor must never make the manager unready.
+
+## Design
+
+### Contract: `internal/provider/check.go`
+
+```go
+type TargetStatus struct {
+    Name, Address   string
+    Ready           bool
+    Reason, Message string
+    Latency         time.Duration
+    Details         map[string]string
+}
+// Check never errors; each target carries its own verdict. Must return promptly once ctx ends.
+type Checker interface{ Check(ctx context.Context) []TargetStatus }
+```
+
+- **Shared reasons:** `Reachable`, `Timeout`, `Unreachable`, `Unauthorized`, `Forbidden`,
+  `APIError`.
+- **Compile-time checks:** each provider's `check.go` has `var _ provider.Checker = (*Provider)(nil)`.
+- **Bypass the wrappers.** `cmd/main.go:197-201` wraps every provider in
+  `WithTracing(WithMetrics(...))`, so the Checkers are collected from the **raw** providers
+  map, before that loop. As a result, checks don't inflate `provider_requests_total`.
+
+### The checks
+
+**kvm** (`internal/provider/kvm/check.go`)
+- **Concurrency:** one goroutine per `cfg.Hosts` entry (`wg.Go`), writing to `out[i]` so
+  results stay in config order.
+- **Inside `p.hosts.call`:** `ConnectGetLibVersion`, `ConnectGetHostname`,
+  `StoragePoolLookupByName(host.Pool)`, then `StoragePoolGetInfo`.
+- **Rules for `fn`:**
+  - It returns only raw RPC errors. `call` drops the shared connection on any other error
+    ([conn.go:160](internal/provider/kvm/conn.go#L160)), so "pool not running" is decided
+    after `call` returns, never returned from `fn`.
+  - It fills a captured result struct only after all four calls succeed. That struct is
+    read only when `call` returned nil, which avoids a race with an abandoned goroutine.
+- **Details:** `hostname`, `libvirtVersion` (decoded as major.minor.patch), `pool`,
+  `poolState`, `poolCapacity` and `poolAvailable` (in GiB).
+- **New RPCs:** add the three to `libvirtAPI` ([conn.go:29-47](internal/provider/kvm/conn.go#L29-L47)),
+  `tracingAPI` (`trace.go`) and `fakeAPI` (`fake_test.go`). `fakeAPI` gets a hostname, a
+  fixed version, and a per-pool state map that defaults to Running.
+- **Telling failure stages apart without matching strings:** add
+  `type dialError struct{ stage dialStage; err error }` in `conn.go`, with `Error()`
+  returning the inner message unchanged and `Unwrap()`. It wraps at four places:
+  - `lazySSH.dial`, when loading the key material fails (`ssh-material`)
+  - `dialSSH`, at the dial step (`tcp`)
+  - `dialSSH`, at the handshake (`ssh-handshake`)
+  - `connectLibvirt` (`libvirt-connect`)
+
+  Because `Unwrap` is kept, `classify` and `isClassified` still see `ErrPermanent` and
+  `fs.ErrNotExist`, so reconcile behaviour doesn't change.
+- **`checkReason(err)`, tested in order:**
+
+  | Condition | Reason |
+  |---|---|
+  | stage ssh-material | `SSHMaterialUnavailable` |
+  | stage tcp | `Unreachable` |
+  | stage ssh-handshake | `SSHHandshakeFailed` |
+  | stage libvirt-connect | `LibvirtConnectFailed` |
+  | ctx deadline or cancel | `Timeout` |
+  | `ErrNoStoragePool` | `PoolNotFound` |
+  | any other libvirt error | `LibvirtError` |
+  | anything else | `Unreachable` |
+  | success, pool not running | `PoolNotActive` |
+  | success | `Reachable` |
+- **Accepted behaviour:**
+  - A check that hits its timeout drops the shared connection. That already happens for
+    any cancelled reconcile context, and it only fires when the host isn't answering.
+  - The leader now keeps one SSH connection per host open from startup, with keepalives
+    every 15s. Before, the connection only opened with the first KVM reconcile.
+  - No forced periodic redial. `lazySSH` caches the key material forever, so a redial
+    would not pick up a rotated Secret anyway; that gap goes into the docs under
+    "Known gaps".
+
+**kubernetes** (`internal/provider/kubernetes/check.go`)
+- **Call:** `List` pods with the managed-label selector and `client.Limit(1)`,
+  cluster-wide. It uses the same uncached client, verb and RBAC as `ListByTag` and GC.
+  There is no namespace setting in the kubernetes config.
+- **Target:** one target named `apiserver`, with the address from a new `apiHost` field
+  set in `NewWithTransportWrapper`.
+- **Reasons:**
+  - `Timeout`: the context ended, or the error is `IsTimeout`/`IsServerTimeout`.
+  - `Unauthorized`, `Forbidden`.
+  - `APIError`: any other API status.
+  - `Unreachable`: anything else.
+
+**gcp** (`internal/provider/gcp/check.go`, plus a seam change in `gcp.go`)
+- **Why a new seam method:** the existing `AggregatedList` seam reads every page
+  ([gcp.go:63-78](internal/provider/gcp/gcp.go#L63-L78)), and MaxResults is only the page
+  size. So add `Probe(ctx, *computepb.AggregatedListInstancesRequest) error`, which makes
+  one `it.Next()` call and treats `iterator.Done` as nil.
+- **Request:** `Project`, `MaxResults: 1`, `ReturnPartialSuccess: true`, and the same
+  managed-label filter `ListByTag` uses.
+- **Target:** one per project, `Name: cfg.Project`, `Address: compute.googleapis.com`.
+  GCP config has no zones, and reading them from Proxy placements would tie the check to
+  the Proxy cache.
+- **Reasons:**
+  - `Timeout`: the context ended.
+  - For `*googleapi.Error`: 401 is `Unauthorized`, 403 is `Forbidden` (this also covers a
+    disabled API), anything else is `APIError`.
+  - `Unreachable`: anything else, including a failure to get a token.
+
+### The `ProviderHealth` CRD: `api/v1alpha1/providerhealth_types.go`
+
+- **Written by hand**, since kubebuilder isn't installed; `PROJECT` is edited by hand too.
+- **Scope and markers:** cluster-scoped, `shortName=ph`, status subresource, no spec.
+- **Scheme registration:** add the type to the existing `AddKnownTypes` call in
+  [proxy_types.go:362](api/v1alpha1/proxy_types.go#L362), not a second `init()`.
+- **Status:**
+  - `type`
+  - `conditions` (`Ready`; `ConditionReady` constant)
+  - `targets[]`, as a list-map keyed by `name`, each with: `name`, `address`, `ready`,
+    `reason`, `message` (at most 1024 characters), `latencyMillis`,
+    `lastTransitionTime`, `details`
+  - `targetsReady` (for example `"1/2"`, used by a printer column)
+  - `lastCheckTime`
+- **Printer columns:** Type, Ready, Targets, Last Check, Age.
+- **Viewer role:** `config/rbac/providerhealth_viewer_role.yaml`. No editor role, since the
+  operator owns the object.
+
+### Monitor: `internal/providerhealth`
+
+- **Fields:** `Monitor{Client, Reader, Targets []Target{Name, Type, Checker}, Interval, Timeout}`.
+- **Scheduling:**
+  - `NeedLeaderElection() true`.
+  - Checks once immediately, then on a ticker. Copy the ticker loop from
+    [gc.go:54-80](internal/gc/gc.go#L54-L80).
+- **Each cycle:**
+  - Runs every provider concurrently under `context.WithTimeout(Timeout)`, inside one
+    `tracing.Start` span.
+  - Stores the snapshot under a mutex; `Snapshot()` returns a deep copy.
+  - Logs changes. Going down logs `WARNING: provider target not ready` with provider,
+    target, address, reason and message. Coming back logs `Provider target ready`. A
+    target that is ready on first sight logs at V(1) only.
+- **Writes:**
+  - Get through `mgr.GetAPIReader()`, not the cache. That avoids a cluster-wide informer
+    and the watch permission it would need, whatever `--proxy-namespace` is set to.
+  - Create the object if it is missing, labelled
+    `app.kubernetes.io/managed-by=egress-proxies-operator`.
+  - Then `Status().Patch(MergeFrom(base))` when the status changed, following
+    [status.go:114-121](internal/controller/status.go#L114-L121).
+  - This means one write per provider per interval, because `lastCheckTime` moves each
+    time. That's acceptable given the 10s minimum interval.
+- **`buildStatus(prev, result, now)` is a pure function:**
+  - A target keeps its `lastTransitionTime` unless `ready` flipped.
+  - The `Ready` condition is set through `apimeta.SetStatusCondition`:
+    - `True/AllTargetsReady` when every target is ready
+    - `False/TargetsNotReady` otherwise, with the failing targets in the message
+    - `Unknown/ChecksNotImplemented` when the provider has no Checker
+    - `Unknown/NoTargets` when there are zero targets
+- **Cleanup, once at `Start`:** delete labelled objects whose name isn't in the config.
+  The config is loaded once per process, and unlabelled objects are never touched.
+- **Interval 0 (disabled):** run the cleanup with nothing kept, then return nil, so no
+  object is left showing a stale `Ready=True`.
+- **Invalid provider names:** provider names aren't validated as DNS-1123 names. For an
+  invalid name, warn once and skip the object writes; metrics are still emitted.
+- **`ValidateSchedule(interval, timeout)`** is called in `main.go` before the manager
+  starts. Interval must be 0 or at least 10s; timeout must be in (0, interval].
+- **RBAC markers on `Monitor`:**
+  - `providerhealths`: get, list, create, delete
+  - `providerhealths/status`: get, patch
+  - `make manifests` regenerates `config/rbac/role.yaml`.
+
+### Metrics and wiring
+
+- **New function:** `metrics.RegisterProviderTargets(reg, read func() []ProviderTarget) error`.
+  `Register`'s signature is left alone.
+- **Gauges:** a scrape-time collector, in the style of `constCollector`, reads the Monitor
+  snapshot and emits three gauges labelled `{provider,target}`:
+  - `proxy_operator_provider_target_up`
+  - `proxy_operator_provider_check_duration_seconds`
+  - `proxy_operator_provider_check_timestamp_seconds`
+- **Conversion closure:** the one that turns the Monitor snapshot into gauges lives in
+  `main.go`, next to `proxyPhaseCounts`. So `providerhealth` doesn't import `metrics`.
+- **No pool-bytes gauge:** that value is in status, and node-exporter is the right source
+  for disk alerts.
+- **`cmd/main.go`:**
+  - New flags `--provider-check-interval` (1m) and `--provider-check-timeout` (15s).
+  - Build the `Targets` from `cfg.Providers` in config order, with
+    `c, _ := raw[pc.Name].(provider.Checker)`.
+  - `mgr.Add(&providerhealth.Monitor{Client: mgr.GetClient(), Reader: mgr.GetAPIReader(), ...})`,
+    then register the collector.
+
+## Steps (one commit each, execution-log entry per step)
+
+**Step 0 — Save the plan.**
+- Create branch `feat/provider-health` off `main`.
+- Copy this plan to `docs/plans/2026-09-11-1526-provider-health-checks.md` and commit it
+  alone.
+- The uncommitted `.claude/settings.json` change stays out of every commit.
+
+**Step 1 — ProviderHealth API and CRD.**
+- `api/v1alpha1/providerhealth_types.go`
+- `zz_generated.deepcopy.go`, from `make generate`
+- `config/crd/bases/crawl.example.com_providerhealths.yaml`, from `make manifests`
+- the CRD and RBAC `kustomization.yaml` files, plus the viewer role
+- `PROJECT`
+
+**Step 2 — Contract plus the kubernetes and GCP checks.**
+- `internal/provider/check.go`
+- `internal/provider/kubernetes/{kubernetes.go,check.go,check_test.go}`
+- `internal/provider/gcp/{gcp.go,check.go,check_test.go,gcp_test.go}`
+
+**Step 3 — KVM check.**
+- `internal/provider/kvm/{conn.go,trace.go,check.go,check_test.go,fake_test.go,conn_test.go,trace_test.go,kvm_integration_test.go}`
+
+**Step 4 — Monitor.**
+- `internal/providerhealth/{monitor.go,status.go,monitor_test.go,status_test.go,envtest_test.go}`
+- `config/rbac/role.yaml`, from `make manifests`
+
+**Step 5 — Metrics and wiring.**
+- `internal/metrics/{metrics.go,metrics_test.go}`
+- `cmd/main.go`
+- `config/manager/manager.yaml`: a comment for the new flags
+- `test/e2e/e2e_test.go`
+
+**Step 6 — Docs.**
+- `docs/architecture.md`: a component row plus a provider-health section
+- `docs/kvm-provider.md`: the reason table, the shared-connection behaviour, and the
+  Secret-rotation gap under "Known gaps"
+- `docs/testing.md`: the new tests and the commands
+- `CLAUDE.md`: its Architecture and Gotchas sections still say "empty scaffold"; point
+  them at `docs/architecture.md`
+
+**Step 7 — Validate, record, push, open the PR.**
+- Run `go vet ./...`, `go test -short ./...`, `make test` and `make test-e2e`.
+- Hand the user the KVM commands below, and record their results in the execution log.
+- Add a CHANGELOG entry once the user confirms.
+- Push, using the token-file credential helper from memory.
+- Open the PR against `main` through the Gitea API (`tea` isn't installed).
+- Don't merge.
+
+## Tests
+
+**Provider checks** (table-driven, `Test<Func>_<scenario>`):
+- kubernetes:
+  - reachable
+  - list options: `Limit==1`, the label selector, no namespace
+  - failure reasons: forbidden, unauthorized, timeout, cancelled, internal, and a
+    network `*url.Error`
+- gcp:
+  - reachable, asserting the request fields
+  - failure reasons: 401, 403, 500, deadline, network
+  - `TestRealInstances_Probe_fetchesOnePage` against `httptest`: exactly one request with
+    `maxResults=1`, even though the page has a `nextPageToken`
+- kvm (using `hostFakes`/`newTestProvider` from `kvm_test.go`):
+  - reachable, with details filled in
+  - per-host verdicts, order preserved
+  - pool inactive, pool missing
+  - hosts checked concurrently (a sequential check would deadlock)
+  - a pool verdict keeps the connection (the dial count stays 1)
+  - a `checkReason` table
+  - `TestSSHAuth_dial_stages`, using the in-process SSH server in `conn_test.go`
+  - three new rows in the tracing table
+
+**Monitor:**
+- Setup: the fake client is built with a status subresource and a RESTMapper that
+  registers ProviderHealth as cluster-scoped.
+- Tests:
+  - `buildStatus` table
+  - objects are created with the label, and existing ones are patched
+  - the timeout bounds a checker that blocks
+  - a nil checker gives Unknown
+  - only transitions are logged (`funcr` logger)
+  - cleanup deletes only labelled objects that aren't configured
+  - the disabled path cleans up and returns
+  - an invalid name skips the writes
+  - `ValidateSchedule` table
+- An envtest round-trip, skipped under `-short`, which proves the generated schema is
+  accepted by a real apiserver.
+
+**Metrics:** `TestRegisterProviderTargets_scrape`, using a fresh registry and
+`GatherAndCompare`; an empty snapshot produces no series.
+
+**e2e (kind):** `providerhealth kubernetes` reaches `Ready=True`, which proves the
+generated RBAC works end to end.
+
+## Verification
+
+**Automated (Claude runs):** `go vet ./...`, `go test -short ./...`, `make test` (envtest
+with `-race`) and `make test-e2e`.
+
+**Against the real hypervisor (handed to the user, not run by Claude)**, with beelink as
+the host:
+
+```bash
+KVM_TEST_HOST=beelink.tail8c69c.ts.net:22 KVM_TEST_USER=novakj KVM_TEST_SSH_KEY=~/.ssh/id_ed25519 \
+KVM_TEST_KNOWN_HOSTS=~/.ssh/known_hosts KVM_TEST_BRIDGE=<bridge> KVM_TEST_IMAGE_URL=<image-url> \
+  go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/
+
+make install && go run ./cmd/main.go --providers-config <scratch copy of kvm-home config, local key paths> \
+  --metrics-bind-address :8080 --metrics-secure=false --provider-check-interval=15s
+kubectl get ph -w
+kubectl get ph kvm-home -o yaml
+curl -s localhost:8080/metrics | grep -E 'proxy_operator_provider_(target_up|check_)'
+```
+
+Each negative case is a one-line edit to the scratch config; nothing changes on the host.
+
+| Edit | Expected reason |
+|---|---|
+| `pool: nope` | `PoolNotFound` |
+| `address: …:2` | `Unreachable` |
+| `knownHostsFile: /dev/null` or `user: nobody` | `SSHHandshakeFailed` |
+| `sshKeyFile: /nonexistent` | `SSHMaterialUnavailable` |
+| `socket: /nonexistent` | `LibvirtConnectFailed` |
+
+## Out of scope
+
+- Gating Proxy creates on a host that is known to be down.
+- Tying `/readyz` to provider health.
+- Bridge checks.
+- SelfSubjectAccessReview checks of create/delete rights.
+- Hot-reloading the providers config.
+- Picking up a rotated `kvm-ssh` Secret without a restart. This is an existing gap; it
+  gets documented.

docs/plans/2026-09-16-0841-provider-blueprint.md

@@ -0,0 +1,119 @@
+# Plan: Provider blueprint (docs/providers.md)
+**Created:** 2026-09-16 08:41
+
+## Context
+
+The provider contract is spread across `docs/prompts/__initial-prompt.md` §5,
+the comments in `internal/provider/{provider,errors,check}.go`, `docs/architecture.md`
+§5 and §11, `docs/kvm-provider.md`, and plan Step 2/3/8. Many rules exist only in
+code: providerID must be self-contained, UID/CreatedAt must be read back, Create
+must survive a crash partway through, calls must stop when ctx is cancelled,
+decorators are wired in `cmd/main.go`, and so on. With "~5 providers soon", the
+next provider author needs one blueprint. It should say what a provider MUST,
+SHOULD or MAY do, show which features each existing provider (kubernetes, gcp,
+kvm) actually implements (with file:line evidence), give a checklist for adding
+one, and list the gaps.
+
+Decisions (user): store in `docs/providers.md`; scope = current state plus a
+separate "Gaps and proposed features" section (nothing implemented); wait for
+`feat/provider-health` to merge, then branch off `main`.
+
+## Step 0 — Save the plan; wait for the merge
+
+- Right after ExitPlanMode, copy this plan to `docs/plans/2026-09-16-0841-provider-blueprint.md`
+  and commit it on its own. The current branch is `feat/provider-health`, so the
+  plan reaches `main` with that merge. CLAUDE.md requires the plan commit to
+  come before any branching.
+- Stop and wait for the user to confirm that `feat/provider-health` is merged.
+  Then `git switch main && git pull`, and `git switch -c feat/provider-blueprint`.
+
+## Step 1 — Collect evidence (read-only, in parallel)
+
+Run 3 Explore agents at the same time, one each for `internal/provider/kubernetes`,
+`internal/provider/gcp` and `internal/provider/kvm`, including their tests. Each
+answers the **same questionnaire**, citing `file:line` for every answer and writing
+"not implemented" explicitly when that's the case:
+
+1. **Identity**: providerID format and parsing; what it carries (zone, host, namespace).
+2. **Create**: how it stays idempotent (name lookup, 409, DomainLookupByName);
+   whether it returns before the VM is up or waits; what happens when an earlier
+   attempt crashed partway; which Placement fields are used or ignored; how
+   cloud-init is delivered; how the port is used; how GC labels/UID are attached.
+3. **Get**: native state → InstanceState mapping; IP discovery rules; UID and
+   CreatedAt readback; how NotFound is detected.
+4. **Delete**: idempotency; what gets removed (disks, volumes, seeds); returns
+   before or after the resource is gone; unknown/malformed providerID behaviour.
+5. **ListByTag**: scope (namespaces, zones, hosts); partial-failure behaviour; how
+   managed resources are recognised.
+6. **Error classification**: table of backend error → NotFound/Quota/Transient/
+   Permanent; whether quota is detected at all.
+7. **Health check** (`Checker`): targets, provider-specific Reasons, Details keys.
+8. **Config**: YAML section, defaults, validation (`internal/provider/config*.go`),
+   credentials/Secrets, which settings are read once per process.
+9. **Connections and cancellation**: pooling, reconnects, timeouts, ctx honouring.
+10. **Observability**: log keys and verbosity levels, spans (`kvm/trace.go`, gcp
+    `wirelog.go`), anything beyond the shared metrics/tracing decorators.
+11. **Testing**: fakes, envtest/kind, `//go:build integration` tests, what is
+    not covered.
+12. **Known gaps**: TODOs, stale comments, documented limitations.
+
+Meanwhile, read the shared layer myself: `internal/provider/{provider,errors,check,
+name,config,metrics,tracing}.go`, `internal/provider/registry/registry.go`,
+`cmd/main.go` (registry wiring, decorator order, Checker collection), and the
+places that consume the contract, so every MUST rule traces to a caller that depends on it:
+`internal/controller` (error-class branching, ErrNotFound → replace/adopt),
+`internal/gc` (UID, CreatedAt age skip, ListByTag errors),
+`internal/providerhealth`. Then cross-check against the existing docs listed in Context.
+
+## Step 2 — Write `docs/providers.md`
+
+Sections:
+
+1. **What a provider is**: a few sentences plus links to architecture.md §5/§11.
+2. **The contract**: per method (Create/Get/Delete/ListByTag), MUST/SHOULD/MAY
+   rules, each followed by *why*: the reconciler or GC behaviour it protects,
+   linked to code.
+3. **Error taxonomy**: what each class makes the reconciler do; how to map backend
+   errors; "unclassified = transient" pitfall.
+4. **Identity and GC labels**: providerID guidance, `LabelManaged`/`LabelUID`,
+   CreatedAt, deterministic `NameFromUID`.
+5. **Optional interfaces**: `Checker` (shared reasons, never fails as a whole,
+   collected before decorators).
+6. **Feature matrix**: one row per questionnaire item, columns kubernetes/gcp/kvm,
+   cells `yes`/`partial`/`no` plus a short note. Put the file:line evidence in a
+   compact per-provider notes subsection under the table, not in the cells.
+7. **Adding a provider: checklist**: package layout, config section and validation,
+   registry constructor, `cmd/main.go` wiring, Checker, error mapping with tests,
+   `name`/label tests, integration test behind a build tag, provider doc page,
+   architecture.md rows, demo script (optional).
+8. **Gaps and proposed features**: current-state gaps per provider (e.g. no kvm
+   quota detection, orphan volumes invisible to GC, Secret rotation needs a
+   restart, ListByTag partial-failure inconsistency, stale `mock` references in
+   `provider.go` comments), plus proposals that are clearly marked as not
+   implemented. Only list them; don't design them.
+
+Also: link it from `docs/architecture.md` (Components table → provider contract row)
+and from the CLAUDE.md "Architecture" section. Leave per-provider docs where they are.
+
+Follow the plan-execution convention: create `docs/plans-executions/2026-09-16-0841-provider-blueprint.md`
+with the status checklist and step entries, and commit it with each step.
+
+## Step 3 — Verify, then open the MR
+
+- A verification agent (Explore) re-checks each matrix cell and each MUST rule
+  against the cited file:line and reports any mismatch. Fix what it finds.
+- Check that every relative link in the new and edited docs resolves (grep the
+  `](` targets and `ls` each one).
+- `go build ./... && go test -short ./...` as a sanity check (docs-only change,
+  expected no-op).
+- Push `feat/provider-blueprint` and open the MR against `main` via the Gitea API,
+  following the project's push/MR memory note. After the user confirms, add a CHANGELOG.md entry.
+
+## Critical files
+
+- New: `docs/providers.md`, `docs/plans/2026-09-16-0841-provider-blueprint.md`,
+  `docs/plans-executions/2026-09-16-0841-provider-blueprint.md`
+- Edited: `docs/architecture.md` (link), `CLAUDE.md` (link)
+- Read-only sources: `internal/provider/**`, `cmd/main.go`, `internal/controller`,
+  `internal/gc`, `internal/providerhealth`, `docs/prompts/__initial-prompt.md`,
+  `docs/kvm-provider.md`, `docs/gcp-vm-validation.md`

docs/plans/2026-09-16-1108-provider-docs-split.md

@@ -0,0 +1,99 @@
+# Plan: Split the provider blueprint into contract, appendix, template and index
+
+**Created:** 2026-09-16 11:08
+
+## Context
+
+`docs/providers.md` is one 731-line blueprint that mixes the generic provider contract with how kubernetes, gcp and kvm happen to meet it, plus gaps and proposals. The user wants to hand an agent a self-contained set of rules for building a new provider (Proxmox is next), so the generic part must stand alone, state explicitly where providers are allowed to differ (a pod is ready at once, a VM has a provisioning window, kvm blocks in Create, gcp does not), and leave implementation detail to a separate appendix. A third file, a fill-in spec template, lets the user write the new provider's specifics in the same vocabulary. `docs/providers.md` becomes a short index so every existing link keeps working.
+
+Decisions already made by the user: two new files plus a template; `providers.md` becomes an index; the contract is fully abstract (never names an existing provider); work continues on the checked-out branch `feat/provider-blueprint`.
+
+Conventions for all four files: code references are `path` plus function or symbol name, never line numbers; rule keywords MUST / SHOULD / MAY as today, and every MUST names the caller and what breaks; no em-dashes in prose. The contract cites only shared-layer files (`internal/provider/*.go`, `internal/provider/registry`, `internal/controller`, `internal/gc`, `internal/providerhealth`, `internal/tracing`, `cmd/main.go`, `config/manager`, `api/v1alpha1`) and never `internal/provider/{kubernetes,gcp,kvm}`.
+
+## Facts verified against the code that the new contract must carry
+
+- Callers put no deadline on the context. Only `Monitor.checkOnce` (`internal/providerhealth/monitor.go`) uses `WithTimeout`; the reconcile and GC contexts are cancel-only. `Sweeper.sweep` (`internal/gc/gc.go`) loops providers sequentially, so one hung `ListByTag` stalls orphan GC for every provider.
+- The reconciler's Stopped/Terminated branch in `reconcileManaged` (`internal/controller/proxy_controller.go`) deletes and requeues without clearing `ProviderID`, so a backend that reports a freshly created, not-yet-started instance as Stopped gets it deleted right after creation. kvm avoids this only by blocking `Create` through start.
+- `Sweeper.sweep` treats a zero `CreatedAt` as collectable now, skips `UID == ""`, and skips anything younger than `MinAge` (10m).
+- `cmd/main.go` type-asserts `provider.Checker` on the unwrapped provider before `WithTracing(WithMetrics(...))`; `registry.Build` fails the whole manager on a constructor error, so constructors must not dial.
+- Manager runtime: non-root, read-only rootfs, `/tmp` emptyDir 64Mi, 128Mi limit, distroless static image with CA roots (`config/manager/manager.yaml`, `Dockerfile`).
+- Corrections to earlier notes, for the appendix: kvm Checker `Details` also carry `user` and `pool`; the kubernetes provider has no seam interface (it injects `client.Client` via `newWithClient` and tests use the fake client); kvm `classify` has an `isClassified` pass-through for errors already wrapped with a class.
+
+## Files
+
+### 1. `docs/provider-contract.md` (new, the generic contract)
+
+Every behaviour axis gets a stable kebab-case code in backticks, used verbatim by the appendix table and the template so the three files can be cross-checked by grep.
+
+Sections:
+
+1. **Preamble**: audience, reading order (contract, template, appendix), rule keywords, "path plus symbol, never line numbers", where the rest lives.
+2. **What a provider is and who calls it**: the types in `provider.go`, `name.go`; callers and entry points (`reconcileManaged`, `replaceInstance`, `reconcileDelete`, `providerFailure`, `deletionFailure`, `placementFrom`, poll intervals, `MaxConcurrentReconciles`; `Sweeper.sweep` and `MinAge`; `Monitor.checkOnce`; decorators; `registry.Build`); what the instance is for (health probe CONNECT in `internal/health`, discovery handing out ip and port); leader-only vs every-replica; concurrency (MUST be safe, including two calls for one name); runtime envelope; time and cancellation (no caller deadline, MUST honour cancel, SHOULD bound each backend call, what breaks: a pinned worker, a stalled GC sweep, a wedged check round).
+3. **The contract, method by method**: current §2 rewritten abstractly, plus the new points. Create: idempotent by name, mark on create or within a documented window shorter than `MinAge`, self-contained ID, ctx, blocking vs submit, reject-early with `ErrPermanent` naming field and value, catalogue changes are outside the spec hash (`spechash.go`), MAY ignore Placement fields but MUST document, SHOULD deliver CloudInit, MUST NOT log it, and an eventually consistent backend MUST NOT let `Get` answer NotFound for an ID `Create` just returned. Get: NotFound only when gone, keep answering during delete, Running only with IP, Stopped/Terminated only when it will not recover, the "defined but not started" rule (block Create through start, or report Provisioning using written metadata or a task handle), behaviour during async operations and locks (last known state or Transient, never NotFound or Permanent), UID and CreatedAt read back, Permanent for an unresolvable ID. Delete: idempotent, re-entrant while deleting, removes secondary resources, MAY return before gone, SHOULD succeed where Get fails permanently. ListByTag: marked instances only with ID, UID, CreatedAt; error rather than an unvouched list; MAY omit unreadable parts; SHOULD cover the whole creatable scope; no installation identity.
+4. **Behaviour profiles**: one subsection per axis with the question, allowed profiles, what callers tolerate, obligations per profile. Axes: `readiness`, `create-return`, `delete-return`, `ip-source`, `gc-mark`, `created-at`, `list-partial`, `quota`, `not-found`, `primary-key`, `provisioning-state`, `async-ops`, `secondary-resources`, `credentials`, `client-kind`, `timeouts`, `logging`, `child-spans`, `backend-test`. Closing table: code, profiles, tolerated by which caller, template entry.
+5. **Error taxonomy**: `Wrap`, `Class`, `Error` fields, `Op` values, configured name not type; the class-to-reaction table; pitfalls; NotFound detection guidance (404, error code, list-and-miss, and why list-and-miss is forbidden with a partial list); the quota-vs-permission 403 case stated abstractly.
+6. **Identity, names, marks, CreatedAt**: `NameFromUID` shape, providerID strict parsing, marks and written metadata, provider name as DNS-1123 (`Monitor` skip, `spec.provider` MaxLength), primary key not the name.
+7. **Credentials, TLS and secret hygiene** (new): source and read time (constructor MUST NOT dial, SHOULD warn on a missing optional mount and fail at first use), never in logs, spans, error strings or `Error.Err`, TLS trust options and the naming of an insecure toggle, scratch only under `/tmp`.
+8. **Optional interfaces: Checker**: current §5 rules; `status.type` comment in `api/v1alpha1/providerhealth_types.go` must list the new type and the CRD be regenerated.
+9. **Observability**: the three layers, span and metric names, ctx and logger rules, child-span strategies by `client-kind`, no secrets or CloudInit in attributes, log tiers.
+10. **Configuration and wiring**: `ProviderConfig`, `setBlocks`, the allow-list in `Config.validate`, `UnmarshalStrict`, `config_<type>.go`, identifiers pasted into IDs, `registry.Constructor`, the map in `cmd/main.go`, decorator order, the duplicated type list.
+11. **Testing obligations**: fake behind a seam; the contract test list (current checklist item 5 plus Get during an async operation, the not-yet-started state, NotFound after Create is forbidden); classify table test; Checker tests; sentinel tests for CloudInit and credentials; span recorder test; integration test behind `//go:build integration` with `<TYPE>_TEST_*` variables, `make test-integration-<type>`, a section in `docs/testing.md`.
+12. **Checklist for adding a provider**: current §8 generalised, plus: Makefile target, testing.md section, `hack/providers-dev.yaml` decision, `ProviderHealth` type comment and `make manifests`, RBAC diff, CHANGELOG, appendix row and section, the filled template committed as `docs/<type>-provider-spec.md`, README row, architecture.md rows.
+13. **Known limits of the contract**: adoption unverified; Stopped and Terminated indistinguishable; config read once; no installation identity in GC; unconfigured provider latches; type list duplicated; Delete resubmitted every poll; no caller deadline; no conformance suite.
+14. **Related documents**.
+
+### 2. `docs/provider-spec-template.md` (new, the questionnaire)
+
+Preamble: copy to `docs/<type>-provider-spec.md`, fill every Answer line, hand it to the implementing agent with the contract and appendix; unanswered entries block implementation. Header block: type, example configured name, backend and API version, SDK or thin client, author, date.
+
+Fixed entry format: Question; Allowed answers (profile names linking to the contract §4 subsection); What each answer obliges; `Answer:` blank; `Evidence:` blank.
+
+Entries: identity and providerID shape; primary key (`primary-key`); placement mapping (Region, Zone, MachineType, Image, ignored fields); image source; cloud-init delivery; machine type mapping; network and IP discovery (`ip-source`, `readiness`); GC marks and CreatedAt (`gc-mark`, `created-at`); state mapping table incl. not yet started (`provisioning-state`); Create blocking and task tracking (`create-return`, `async-ops`); Delete and secondary resources (`delete-return`, `secondary-resources`); ListByTag scope and partial failure (`list-partial`); error classification table (`not-found`, `quota`); credentials and TLS (`credentials`); config block fields, defaults, validation; Checker targets, reasons, Details; SDK vs thin client and child spans (`client-kind`, `child-spans`); timeouts (`timeouts`); concurrency; logging tiers (`logging`); tests (`backend-test`); deployment (mounts, samples, Makefile helper, RBAC, dev config); docs; known limits accepted at design time.
+
+### 3. `docs/providers-existing.md` (new, the appendix)
+
+Sources: current `providers.md` §6 table, §7 matrix and per-provider notes, §9 per-provider gaps and proposed features, `docs/kvm-provider.md`, the gcp and kubernetes package facts. Links to architecture.md §5, §11 and Decisions, and to the README table, rather than duplicating them.
+
+Sections: preamble; **profile table** with exactly one row per contract axis code and one column per existing provider, each cell a profile name plus a symbol as evidence; **other differences** (matrix rows that are not axes: Placement used, Port used, CloudInit delivered, states produced, Error.Provider correctness, credential rotation, connection handling, Checker Details, fakes, proxy software, reachability, distinct egress, adopted-UID check); **observability status** (current §6 table); one section each for **kubernetes**, **gcp**, **kvm** with the same sub-headings: package layout and seam, config block, ID format, Create, Get and state mapping table, IP source, Delete, ListByTag, classify table, Check (targets, reasons, Details), logging, tracing layer, tests and fakes, deployment bits, known gaps; **gaps shared by the existing providers**; **proposed features** (current list, unchanged).
+
+### 4. `docs/providers.md` (rewritten in place as the index)
+
+One paragraph on the split; one line each for contract, template, appendix; the per-provider guides (`docs/kvm-provider.md`, `docs/gcp-vm-validation.md`, `docs/gcp-in-specific-project.md`); the architecture.md sections; reading order for a new provider; a note that the pre-split blueprint is in git history under this path.
+
+### 5. One-line link edits
+
+- `README.md` "### Adding a provider": point at the contract, appendix and template, with `docs/providers.md` named as the index. Keep the "Start there" sentence.
+- `CLAUDE.md` "## Architecture": same replacement for the providers.md sentence.
+- `docs/architecture.md` components row "Provider contract": rules in `provider-contract.md`, profile table in `providers-existing.md`.
+- Historical files under `docs/plans*` keep their links; they resolve to the index.
+
+## Steps (one commit per step, on `feat/provider-blueprint`)
+
+- **Step 0**: copy this plan to `docs/plans/<ts>-provider-docs-split.md` with the real timestamp and commit it alone, before anything else.
+- **Step 1**: write `docs/provider-contract.md`; freeze the axis code list. Create `docs/plans-executions/<ts>-provider-docs-split.md` with the status checklist.
+- **Step 2**: write `docs/provider-spec-template.md` from contract §4 and §12.
+- **Step 3**: write `docs/providers-existing.md`, replacing every `file:line` from the old blueprint with `file` plus symbol.
+- **Step 4**: rewrite `docs/providers.md` as the index; apply the three link edits.
+- **Step 5**: run the verification below; fix findings.
+- **Step 6**: final execution summary entry. CHANGELOG entry at the top only after the user confirms. Push and MR only when the user asks (per memory, push uses the token helper and the MR goes through the Gitea API).
+
+## Verification (read-only shell, from the repo root)
+
+1. Every MUST names a caller: `grep -n MUST docs/provider-contract.md` and confirm each bullet cites a shared-layer path or symbol and states what breaks.
+2. Every cited path exists: extract `(internal|cmd|config|docs|api|hack|test)/...` tokens from the four files and `test -e` each.
+3. Every cited symbol exists: extract backticked identifiers, drop paths, keywords and YAML keys, and `grep -rqw` each across `*.go`, `Makefile`, `*.yaml`.
+4. Contract is abstract: `grep -niE "kubernetes|gcp|kvm|libvirt|squid|google|compute engine|internal/provider/(kubernetes|gcp|kvm)" docs/provider-contract.md` is empty.
+5. Profiles line up 1:1: for each axis code from contract §4 headings, it appears in the appendix table and in the template; the appendix table has one row per code.
+6. Links resolve: extract `](target)` from the four files plus README, CLAUDE.md and architecture.md; check file targets exist and anchors match slugified headings.
+7. No em-dashes in the four files or in the added lines of the three edited files.
+8. No line references: `grep -nE '\.go:[0-9]+|:[0-9]+-[0-9]+' <four files>` is empty.
+9. Nothing lost: compare the gap and proposal bullets of `git show HEAD:docs/providers.md` against the appendix and contract §13 by keyword.
+10. `go build ./... && go test -short ./...` still pass (docs-only change).
+
+## Execution summary must record
+
+- Step 1: the frozen axis list; the two contract additions found in verification (sequential GC sweep; the Stopped branch re-deleting a not-yet-started instance); per-call timeout facts as verified for each existing provider.
+- Step 3: the fact corrections applied (kvm Details keys, kubernetes seam, kvm `isClassified`); matrix rows dropped in favour of the profile table.
+- Step 4: the exact link edits.
+- Step 5: the verification commands as run with trimmed output.
+- Worth noting: CHANGELOG headings keep the repo's own em-dash format; the stale `mock` comments in `provider.go`, `registry.go` and `proxy_types.go` stay listed as gaps and are not fixed here.

docs/provider-contract.md

@@ -0,0 +1,860 @@
+# The provider contract
+
+This document is the specification a provider must meet. It is written for
+whoever builds the next one, human or agent, and it deliberately says nothing
+about how any existing provider is built. Those details are in
+[providers-existing.md](providers-existing.md); the questionnaire a new
+provider's design must answer is [provider-spec-template.md](provider-spec-template.md).
+Read the three in that order: this contract, then the template, then the
+appendix for worked examples.
+
+Rule keywords: **MUST** means breaking the rule breaks a caller, and every MUST
+below names the caller and what breaks. **SHOULD** means every current provider
+does it and a new one needs a written reason not to. **MAY** means it is
+allowed and the choice must be recorded in the provider's spec.
+
+Code references are a file path plus a function or symbol name, never a line
+number. Where a rule is enforced by a caller, that caller is one of:
+
+- the **reconciler**, `internal/controller/proxy_controller.go`;
+- **orphan GC**, `internal/gc/gc.go`;
+- the **provider health monitor**, `internal/providerhealth/monitor.go`;
+- the **decorators**, `internal/provider/metrics.go` and
+  `internal/provider/tracing.go`;
+- the **composition root**, `cmd/main.go`.
+
+## 1. What a provider is and who calls it
+
+### 1.1 The types
+
+A provider is a Go type implementing `provider.Provider`
+(`internal/provider/provider.go`): four methods over one VM-like instance,
+identified by an opaque `providerID` string.
+
+- `Create(ctx, CreateRequest) (providerID, error)`
+- `Get(ctx, providerID) (*Instance, error)`
+- `Delete(ctx, providerID) error`
+- `ListByTag(ctx) ([]Instance, error)`
+
+`CreateRequest` carries `Name` (from `NameFromUID` in `internal/provider/name.go`,
+22 characters, `proxy-` plus base32), `UID` (the owning Proxy's UID),
+`Namespace`, `ProxyName`, `Placement` (`Region`, `Zone`, `MachineType`,
+`Image`), `CloudInit` (already resolved from inline text or a Secret) and
+`Port`.
+
+`Instance` carries `ID`, `IP`, `State` (one of `StateProvisioning`,
+`StateRunning`, `StateStopped`, `StateTerminated`), `UID` and `CreatedAt`.
+
+The GC marks are the constants `LabelManaged` (value `LabelManagedYes`) and
+`LabelUID`, also in `provider.go`.
+
+Providers are named instances of a type: two entries of one type with
+different config blocks are two providers. The configured name is what
+`Proxy.spec.provider` refers to, and it is immutable, so a Proxy never moves
+between providers.
+
+### 1.2 The callers
+
+Nothing above the provider watches backend resources. Everything the operator
+knows about an instance comes from polling `Get`.
+
+- **The reconciler** calls `Create`, `Get` and `Delete` from `reconcileManaged`,
+  `replaceInstance` and `reconcileDelete`, and branches only on the error class
+  (`providerFailure`, `deletionFailure`) and on `Instance.State`. It maps the
+  CRD's placement to `Placement` one to one in `placementFrom`. Its poll
+  intervals are set in `applyDefaults`: `ProvisioningPoll` and `DeletionPoll`
+  10s, `DriftPoll` 2m, `RequeueNow` 1s, `QuotaRetry` 5m. Up to 3 Proxies
+  reconcile at once (`MaxConcurrentReconciles` in `SetupWithManager`).
+- **Orphan GC** calls `ListByTag` and `Delete` from `Sweeper.sweep`, every
+  `Interval` (10m), and never deletes an instance younger than `MinAge` (10m).
+  It loops over providers one after another in a single goroutine.
+- **The provider health monitor** calls the optional `Checker` from
+  `Monitor.checkOnce`, every `--provider-check-interval`, all providers
+  concurrently under one timeout.
+- **The composition root** builds every provider through `registry.Build`
+  (`internal/provider/registry/registry.go`) before the manager starts, on
+  every replica, then wraps each one in `WithTracing(WithMetrics(...))`.
+
+**Where the calls come from.** The reconciler, GC, the health engine and the
+monitor run on the leader only. The constructors run on every replica at
+startup.
+
+**Concurrency.** Up to 3 reconciles, a GC sweep and a health-check round can
+call the same provider at the same time, and GC's `Delete` can overlap the
+reconciler's `Delete` for the same instance. A provider **MUST be safe for
+concurrent use, including two calls for the same instance name**. Nothing
+above the provider serialises them.
+
+### 1.3 What the instance is for
+
+The instance is a forward proxy with its own egress path. Two parties connect
+to `http://<Instance.IP>:<spec.port>`:
+
+- the health engine, from the operator pod, which sends a real `CONNECT`
+  through it (`probe` in `internal/health/probe.go`);
+- clients, which get `ip` and `port` from the discovery API (`viewOf` in
+  `internal/discovery/handlers.go`).
+
+So the IP a provider reports **MUST be reachable on that port from the
+operator's pod network and from the clients**. Whatever that takes on the
+backend (firewall rules, security groups, routing) is either done by the
+provider or documented as a prerequisite in the provider's guide.
+
+The provider does not know which proxy software runs. The proxy is installed
+and configured by the Proxy's `cloudInit`; building install logic into the
+operator is a stated non-goal (`docs/prompts/__initial-prompt.md`, non-goals).
+A backend that cannot take user-data (see `readiness` in §4) documents that
+it ignores the field and how the proxy gets configured instead.
+
+### 1.4 Where it runs
+
+The manager container (`config/manager/manager.yaml`) runs as non-root with a
+read-only root filesystem, a 64Mi `emptyDir` at `/tmp` and a 128Mi memory
+limit, from a distroless static image (`Dockerfile`) that carries the public
+CA roots and nothing else. So a provider:
+
+- **MUST stream large payloads** rather than buffering them in memory;
+- **MUST write scratch files only under `/tmp`**, and delete them;
+- gets credentials either as mounted files (the providers ConfigMap at
+  `/etc/proxy-operator`, optional Secret mounts) or from the environment or
+  the pod's own identity. See §7.
+
+### 1.5 Time and cancellation
+
+**No caller puts a deadline on the context.** The reconcile and GC contexts
+are cancelled on shutdown and on loss of leadership, and otherwise never.
+The only bounded context is the health monitor's (`Monitor.checkOnce`, one
+timeout per round).
+
+- **MUST honour `ctx`**: return promptly once it is cancelled, on every
+  method.
+- **SHOULD bound every backend call itself**: a dial timeout, a per-request
+  timeout, a cap on any wait loop. What breaks otherwise: a hung `Create`,
+  `Get` or `Delete` pins one of the 3 reconcile workers until the manager
+  stops; a hung `ListByTag` stalls orphan GC for every provider, because
+  `Sweeper.sweep` runs them sequentially; a `Check` that ignores its context
+  wedges a monitor round. Choose budgets against the polls: a `Get` that
+  routinely outlives `ProvisioningPoll` gains nothing.
+
+## 2. The contract, method by method
+
+### 2.1 Create(ctx, CreateRequest) (providerID, error)
+
+- **MUST be idempotent by `req.Name`.** A repeat call for a name that already
+  exists returns that instance's providerID with a nil error, creates no
+  duplicate and returns no error.
+  *Caller:* the reconciler records the ID in status only after `Create`
+  returns (`reconcileManaged`). A crash between the two, or a lost status
+  write, runs `Create` again for the same Proxy; `replaceInstance` recovers
+  from a crash between its two writes the same way.
+- **MUST mark the instance as managed**, with `LabelManaged=LabelManagedYes`
+  and `LabelUID=req.UID` or the backend's equivalent, in the same call that
+  creates it. When the backend's create call cannot carry the mark, the
+  window between creation and marking **MUST be far shorter than `MinAge`**
+  and documented in the provider's spec (`gc-mark` in §4).
+  *Caller:* orphan GC finds instances only through the mark and never deletes
+  one whose UID is empty (`Sweeper.sweep`). An instance that exists without
+  the mark leaks if the operator crashes before marking it.
+- **MUST return a providerID that is self-contained.** `Get` and `Delete`
+  receive only the ID, and the config or spec may have changed since
+  `Create` (a zone edit during a replacement). Put whatever `Get` needs to
+  find the instance into the ID, and validate config values that end up in
+  the ID so they cannot contain the separator. See `primary-key` in §4 for
+  backends that allocate their own identifier.
+- **MUST NOT let `Get` answer NotFound for an ID that `Create` just
+  returned.** A backend with eventual consistency between create and read
+  needs a read-after-write guarantee or a retry inside `Create`.
+  *Caller:* on NotFound the reconciler clears the ID and creates again
+  (`reconcileManaged`), so a lagging read makes a duplicate.
+- **MUST honour `ctx`** (§1.5).
+- **MAY return once the request is submitted, or block until the instance is
+  running** (`create-return` in §4). Submit-and-return is preferred: readiness
+  is discovered by polling `Get`, and a blocking `Create` holds one of the 3
+  workers for its whole duration. A provider that blocks MUST still honour
+  cancellation and SHOULD cap the wait.
+- **SHOULD reject a request it can never satisfy** with `ErrPermanent`
+  before touching the backend: a missing or unknown zone, machine type or
+  image, or any placement value not in its catalogue. The CRD cannot validate
+  `placement`, because it is provider-agnostic, so this error is the user's
+  only feedback. The message SHOULD name the offending field and value.
+- **Catalogue changes do not replace instances.** The spec hash
+  (`specHashInput` in `internal/controller/spechash.go`) covers `placement`,
+  the resolved `cloudInit` and `port`, not the provider config. When a
+  placement value names a config entry, changing what that entry means affects
+  only new instances. The provider's guide MUST say so and tell users to add a
+  new key rather than edit one in use.
+- **MAY ignore `Placement` fields that do not apply.** If a field is ignored,
+  the provider's guide MUST say so, because the field is still part of the
+  spec hash: changing it replaces the instance and has no other effect.
+- **SHOULD deliver `CloudInit` as user-data**, byte for byte, without parsing
+  or re-rendering it. A provider that cannot documents that it ignores the
+  field.
+- **MUST NOT log `CloudInit` contents**, put them in span attributes, or
+  include them in error messages. It can hold secrets. Log its byte count.
+  Test this with a sentinel string (§11).
+
+### 2.2 Get(ctx, providerID) (*Instance, error)
+
+- **MUST return `ErrNotFound` when, and only when, the instance is gone.**
+  *Caller:* `ErrNotFound` triggers three actions. In normal reconciling the
+  reconciler clears the ID and creates again (`reconcileManaged`). During
+  replacement it is the signal to create the new instance
+  (`replaceInstance`). During deletion it releases the finalizer
+  (`reconcileDelete`). A false NotFound during an outage creates a duplicate
+  or loses the instance. "Unreachable", "wrong credentials" and "cannot tell"
+  are Transient or Permanent, never NotFound.
+- **MUST keep returning the instance, not NotFound, while a delete is in
+  progress** and the name is still taken.
+  *Caller:* the replacement instance gets the same name. `replaceInstance`
+  waits for NotFound before creating it; a NotFound too early makes `Create`
+  adopt the dying instance.
+- **MUST report `StateRunning` only with a non-empty `IP`.** Running without
+  an address yet is `StateProvisioning`.
+  *Caller:* the IP from a Running answer goes straight into status and to the
+  health engine (`reconcileManaged`).
+- **MUST report `StateStopped` or `StateTerminated` only when the instance
+  will not recover by itself.** SHOULD map states it does not recognise to
+  `StateStopped`.
+  *Caller:* the reconciler reacts to both states identically: it calls
+  `Delete` and requeues after `DeletionPoll` without clearing the ID, then
+  polls `Get` and deletes again until NotFound, then creates
+  (`reconcileManaged`). Instances are disposable, so this is the safe answer
+  for anything unknown, but it makes the next rule critical.
+- **MUST NOT report a freshly created instance that has not been started yet
+  as `StateStopped`.** A backend with no provisioning state of its own, where
+  "defined but not yet started" and "stopped" look the same, MUST resolve this
+  in one of two ways (`provisioning-state` in §4): block `Create` until the
+  instance is started, or record enough at creation (written metadata, a task
+  handle) for `Get` to answer `StateProvisioning`.
+  *Caller:* the same branch of `reconcileManaged` would delete the instance
+  right after it was created, forever.
+- **During an asynchronous operation or a backend lock** (a clone or a delete
+  still running, an instance locked by the backend), `Get` MUST return either
+  the last known state or `ErrTransient`, never NotFound and never
+  Permanent (`async-ops` in §4).
+- **MUST fill `UID` and `CreatedAt` by reading them back from the backend**,
+  never from memory.
+  *Caller:* GC compares `UID` with the live Proxies and deletes anything whose
+  `CreatedAt` is older than `MinAge`. **A zero `CreatedAt` counts as very
+  old** (`Sweeper.sweep`), so a provider that forgets it makes instances still
+  being created collectable. An unparsable timestamp giving the zero time is
+  accepted: it never shields an orphan forever.
+- **SHOULD return `ErrPermanent` for an ID it can never resolve**: a malformed
+  ID, a location no longer in the config.
+  *Caller:* `reconcileDelete` then tries `Delete` anyway and releases the
+  finalizer if that works. If the provider returned Transient, deletion would
+  poll forever.
+
+### 2.3 Delete(ctx, providerID) error
+
+- **MUST be idempotent.** Deleting something already gone returns nil.
+  *Caller:* GC and the reconciler race, and a crash repeats the call.
+- **MUST accept being called again while an earlier delete is still in
+  progress.** Treat "already deleting" as success, and SHOULD keep that cheap.
+  *Caller:* while `Get` still returns the instance, `reconcileDelete` and the
+  Stopped branch of `reconcileManaged` call `Delete` again every
+  `DeletionPoll` until `Get` returns NotFound.
+- **MUST remove everything `Create` made**: disks, volumes, seeds, addresses,
+  uploaded files. Either delete them explicitly or rely on the backend
+  cascading (`secondary-resources` in §4).
+  *Caller:* GC sees only what `ListByTag` returns, so secondary resources
+  have no other cleanup path.
+- **MAY return before the instance is gone** (`delete-return` in §4), as long
+  as `Get` keeps showing it until it is.
+- **SHOULD succeed for an ID whose `Get` fails permanently** when there is
+  nothing left the provider could reach, and log that it did nothing.
+  *Caller:* on the deletion path a permanent error is not latched; it retries
+  forever (`deletionFailure`), and the Proxy is then only released through the
+  `AnnotationSkipCleanup` annotation (`api/v1alpha1/proxy_types.go`).
+
+### 2.4 ListByTag(ctx) ([]Instance, error)
+
+- **MUST return only instances that carry the managed mark**, each with an
+  `ID` that `Delete` accepts, plus `UID` and `CreatedAt`. `State` and `IP`
+  are not used by GC.
+  *Caller:* GC deletes every listed instance whose UID is not live and whose
+  age is past `MinAge` (`Sweeper.sweep`). Listing something without the mark
+  is how an operator deletes an instance it does not own.
+- **MUST return an error rather than a list it cannot vouch for.** An error
+  makes GC skip that provider for this sweep (`Sweeper.sweep`).
+- **MAY leave out instances it could not read.** Leaving something out is
+  always safe, because GC never deletes what is not listed. Pick a policy
+  (`list-partial` in §4) and document it.
+- **SHOULD cover the whole scope the provider can create in**: every
+  namespace, zone, node or host. Anything outside that scope is never
+  collected.
+- **Scope caveat:** GC protects instances by UID, not by installation. Two
+  operator installs sharing one backend delete each other's instances (§13).
+  Two provider entries of one install that share a scope are safe, because
+  they share the live-UID set.
+
+## 3. What a container-style backend and a VM-style backend have in common
+
+The contract above is the same for both, but the profiles in §4 differ.
+A backend that runs a container has an instance that is ready seconds after
+`Create`, gets its address from the platform, is deleted in one call and has
+no secondary resources; a backend that boots a VM has a provisioning window
+of minutes, may discover the address only after the guest reports it, may
+need several deletes, and leaves disks and seeds behind unless told
+otherwise. Neither is preferred. What matters is that the provider's spec
+names its profile on every axis, because the appendix table and the test list
+are keyed by those names.
+
+## 4. Behaviour profiles
+
+Each axis below has a stable code. The provider's spec (the template) answers
+every axis; the appendix maps each existing provider to its profile on every
+axis. A profile is not a permission to skip a MUST in §2; it says which of
+the allowed ways of meeting it the provider chose.
+
+### 4.1 `readiness`
+
+*When is an instance usable after `Create`?*
+
+- **immediate**: the platform starts it at once; `Get` reports Running
+  within a poll or two.
+- **provisioning-window**: minutes of booting; `Get` reports Provisioning
+  for that time.
+
+Callers tolerate both: the reconciler polls at `ProvisioningPoll` either
+way. An immediate provider still MUST report Provisioning until the IP is
+known.
+
+### 4.2 `create-return`
+
+*Does `Create` return once the request is submitted or once the instance is
+running?*
+
+- **submit**: preferred. Readiness comes from `Get`.
+- **blocking**: allowed. Costs one of 3 workers for the duration, and any
+  first-use setup (an image download) runs inside it. MUST honour `ctx`,
+  SHOULD cap the wait, and MUST keep every step idempotent so a cut-off
+  attempt resumes.
+
+### 4.3 `delete-return`
+
+*Does `Delete` return before the instance is gone?*
+
+- **returns-early**: `Get` keeps returning the instance until it is gone
+  (§2.2), and repeated `Delete` calls are cheap.
+- **waits**: `Delete` returns nil only when nothing is left. Bound the wait
+  and honour `ctx`.
+
+### 4.4 `ip-source`
+
+*Where does `Instance.IP` come from?*
+
+- **assigned-at-create**: the backend reports it with the instance.
+- **discovered**: an agent in the guest, a DHCP lease, a neighbour table.
+  Discovery failures mean "no address yet", never an error; `Get` returns
+  Provisioning. The provider's guide MUST say what the guest needs (an agent
+  package, outbound access to install it) and what happens without it.
+
+### 4.5 `gc-mark`
+
+*Where do `LabelManaged` and `LabelUID` live?*
+
+- **native-labels**: labels or tags the backend indexes; `ListByTag` can
+  filter server-side.
+- **written-metadata**: a description, an annotation or a metadata element
+  written by the provider; `ListByTag` lists everything and filters
+  client-side.
+- **mark-window**: the create call cannot carry the mark and a second call
+  writes it. Allowed only when the window is a few seconds, documented, and
+  the unmarked instance is cleaned by the same Proxy's next reconcile (§2.1).
+
+### 4.6 `created-at`
+
+*Where does `CreatedAt` come from?*
+
+- **backend-timestamp**: the backend's own creation time.
+- **written-at-create**: the provider writes the time into the mark
+  metadata in the create call. A backend with no timestamp MUST do this.
+
+### 4.7 `list-partial`
+
+*What does `ListByTag` do when part of its scope is unreachable?*
+
+- **fail-whole**: one unreachable part fails the list; GC skips the provider
+  this sweep. Nothing is collected until every part answers.
+- **omit-unreachable**: unreachable parts are left out and the rest is
+  returned. Orphans in the reachable parts are collected; the provider MUST
+  log what it left out.
+
+Both are safe (§2.4). A single unreadable instance MAY always be skipped.
+
+### 4.8 `quota`
+
+*Can the provider tell "no capacity" from other failures?*
+
+- **detected**: a recognisable quota, rate-limit or out-of-resources signal
+  maps to `ErrQuotaExceeded`.
+- **absent**: capacity errors map to `ErrTransient`. The guide MUST say so,
+  and the `Checker` SHOULD expose a capacity figure in `Details` instead.
+
+### 4.9 `not-found`
+
+*How does the provider know an instance is gone?*
+
+- **status-404**: a distinct not-found status or error code from the
+  backend.
+- **error-code**: a code or message the backend uses for "does not exist",
+  matched exactly and covered by the classify table test with real captured
+  responses.
+- **list-and-miss**: absence from a listing. Allowed only when the listing
+  can never be partial; otherwise a missing part reads as "gone" and §2.2 is
+  broken.
+
+### 4.10 `primary-key`
+
+*Is `req.Name` the backend's key?*
+
+- **name-is-key**: the backend addresses instances by name. `Create` looks
+  the name up, the ID is `<location>/<name>`.
+- **allocated-id**: the backend allocates an identifier and the name is a
+  label. `Create` MUST first find an existing instance by name (idempotency),
+  then allocate. Allocation MUST be safe against a concurrent `Create` for
+  another Proxy: a deterministic identifier derived from `req.UID`, or an
+  allocate-then-verify loop. The ID SHOULD embed the allocated key and the
+  name; `Get` and `Delete` use the allocated key. If instances can move
+  between locations (migration), the ID MUST NOT depend on the location.
+
+### 4.11 `provisioning-state`
+
+*Does the backend have a state for "being created"?*
+
+- **native**: the backend reports a provisioning or staging state, mapped to
+  `StateProvisioning`.
+- **create-blocks**: no such state; `Create` blocks until the instance is
+  started, so `Get` never sees "defined but not started".
+- **marker**: no such state and `Create` returns early; the provider records
+  at creation (metadata or a task handle) what lets `Get` answer
+  `StateProvisioning` for an instance that was never started, and
+  `StateStopped` for one that stopped afterwards.
+
+### 4.12 `async-ops`
+
+*Does the backend run create and delete as asynchronous tasks?*
+
+- **none**: calls complete synchronously.
+- **task-handle**: the backend returns a task the provider can poll. The
+  provider SHOULD keep the handle where `Get` can find it (metadata or the
+  ID) so a failed create can be reported instead of read as NotFound.
+- **backend-lock**: the backend locks the instance during a task. `Get`
+  returns the last known state or Transient while locked (§2.2); `Delete`
+  on a locked instance returns Transient, not Permanent.
+
+### 4.13 `secondary-resources`
+
+*What else does `Create` make, and who removes it?*
+
+- **none**.
+- **cascade**: the backend deletes them with the instance (auto-delete
+  flags). The provider MUST set those flags at create.
+- **explicit**: the provider deletes them by name in `Delete`, tolerating
+  their absence. Resources that can exist without an instance (a crash
+  between creating them and creating the instance) are a documented GC blind
+  spot unless `ListByTag` covers them.
+
+### 4.14 `credentials`
+
+*Where do credentials and trust material come from?*
+
+- **mounted-files**: paths in the config block, pointing into an optional
+  Secret mount (§7).
+- **environment**: standard SDK environment discovery.
+- **workload-identity**: the pod's own identity.
+
+TLS trust for an HTTPS backend with a private CA is one of **system-roots**,
+**ca-file** (a path in the config block) or **insecure** (a boolean whose
+name starts with `insecure`, warned about at startup).
+
+### 4.15 `client-kind`
+
+*What talks to the backend?*
+
+- **vendor-sdk**: put it behind a small interface (the seam) so a fake can
+  stand in.
+- **thin-client**: a `net/http` client written in the provider package.
+  Preferred when no maintained SDK exists (`CLAUDE.md`: stdlib over
+  third-party). It MUST set transport timeouts, MUST carry the backend's
+  status and body in one error type that `classify` reads, and SHOULD wrap
+  its transport for child spans (§9).
+- **rpc-over-ssh** or another non-HTTP transport: one pooled connection per
+  target, a keepalive, and a documented answer to "what does a cancelled call
+  do to the shared connection".
+
+### 4.16 `timeouts`
+
+*How is each backend call bounded?* (§1.5)
+
+- **per-call**: every request has its own deadline, independent of `ctx`.
+- **ctx-only**: the provider relies on `ctx`. Allowed only for backends
+  whose client already enforces a timeout; the spec MUST cite it.
+
+### 4.17 `logging`
+
+*Which log tiers does the provider produce?* The logger MUST come from
+`logf.FromContext(ctx)` at call time (§9).
+
+- **tiered**: V(1) one line per operation outcome and per failed backend
+  call; V(2) request and response detail; V(5) raw wire if there is one.
+- **outcome-only**: V(1) only.
+- **none**: allowed only for a backend whose client logs on its own.
+
+### 4.18 `child-spans`
+
+*How do backend calls appear under the operation span?* (§9)
+
+- **sdk-instrumentation**: the SDK emits spans itself.
+- **transport-wrapper**: a parent-gated `otelhttp` transport.
+- **span-decorator**: a decorator over the seam that starts one span per
+  call.
+- **none**: a documented gap, with log correlation through `traceID` only.
+
+### 4.19 `backend-test`
+
+*How is the provider proven against the real backend?*
+
+- **cluster-e2e**: the kind end-to-end suite.
+- **integration-tag**: a test behind `//go:build integration`, skipped
+  without its environment variables, with a `make test-integration-<type>`
+  target and a section in `docs/testing.md`.
+- **manual-runbook**: a documented by-hand run. Allowed only when the
+  backend cannot be reached from CI or a developer machine.
+
+### 4.20 Summary
+
+| Axis | Profiles | Who tolerates the difference | Template entry |
+|---|---|---|---|
+| `readiness` | immediate, provisioning-window | reconciler polls either way | 7 |
+| `create-return` | submit, blocking | reconciler; worker budget | 10 |
+| `delete-return` | returns-early, waits | reconciler re-calls Delete | 11 |
+| `ip-source` | assigned-at-create, discovered | reconciler gates Running on IP | 7 |
+| `gc-mark` | native-labels, written-metadata, mark-window | GC | 8 |
+| `created-at` | backend-timestamp, written-at-create | GC `MinAge` | 8 |
+| `list-partial` | fail-whole, omit-unreachable | GC | 12 |
+| `quota` | detected, absent | reconciler `QuotaRetry` | 13 |
+| `not-found` | status-404, error-code, list-and-miss | reconciler | 13 |
+| `primary-key` | name-is-key, allocated-id | reconciler idempotency | 2 |
+| `provisioning-state` | native, create-blocks, marker | reconciler Stopped branch | 9 |
+| `async-ops` | none, task-handle, backend-lock | reconciler | 10 |
+| `secondary-resources` | none, cascade, explicit | GC blind spots | 11 |
+| `credentials` | mounted-files, environment, workload-identity | deployment | 14 |
+| `client-kind` | vendor-sdk, thin-client, rpc-over-ssh | tests, tracing | 17 |
+| `timeouts` | per-call, ctx-only | worker and GC budget | 18 |
+| `logging` | tiered, outcome-only, none | operators | 20 |
+| `child-spans` | sdk-instrumentation, transport-wrapper, span-decorator, none | tracing | 17 |
+| `backend-test` | cluster-e2e, integration-tag, manual-runbook | CI | 21 |
+
+## 5. Error taxonomy
+
+Every error a provider method returns **MUST be built with
+`provider.Wrap(class, op, providerName, id, err)`** (`internal/provider/errors.go`).
+`op` is one of `create`, `get`, `delete` or `list`, and `providerName` is the
+**configured name** (`ProviderConfig.Name`), not the type.
+*Caller:* the decorators label metrics and spans with the class
+(`resultLabel`), and the reconciler branches on `provider.Class`.
+
+`Wrap` keeps both `errors.Is(err, class)` and `errors.As` to the backend's
+error type working (`Error.Unwrap` returns both).
+
+| Class | Use it for | During normal reconciling | While deleting |
+|---|---|---|---|
+| `ErrNotFound` | the instance does not exist | Get: recreate or advance (§2.2); other ops: returned as the error, so exponential backoff | Get: release the finalizer |
+| `ErrQuotaExceeded` | quota, rate limit, out of capacity | `Provisioned=False` with reason `QuotaExceeded`, requeue after `QuotaRetry`, not counted as an error (`providerFailure`) | requeue after `QuotaRetry` (`deletionFailure`) |
+| `ErrTransient` | network, 5xx, timeouts, cancellation, anything unknown | error returned, so workqueue exponential backoff | same |
+| `ErrPermanent` | bad config, bad argument, auth or permission denied, malformed ID | **latched**: `Provisioned=False` with reason `PermanentError`, no more provider calls until `metadata.generation` changes (`reconcileManaged`) | error returned and retried forever; a Permanent `Get` also triggers `Delete` (`reconcileDelete`) |
+
+Pitfalls:
+
+- **Unclassified means Transient.** `provider.Class` returns `ErrTransient`
+  for any error not built with `Wrap`. That is the safe default, but a
+  permanent failure that was never classified retries forever, and a quota
+  error backs off far too fast.
+- **Permanent is sticky.** Use it only for errors that editing the Proxy spec
+  or fixing the config can resolve. A permission-denied status that the
+  backend also uses for quota is the hard case: read the backend's reason if
+  it gives one, and if it does not, pick Permanent and document it.
+- **Do not return NotFound from `Create`.** The reconciler treats it as
+  Transient there, but it misleads the metrics and anyone reading logs.
+- **NotFound detection** follows the `not-found` profile (§4.9). A backend
+  that reports "does not exist" as a generic server error needs the exact
+  message or code matched, with real captured responses in the test table.
+- **Classify in one `classify` function** with a table-driven test covering
+  every class, including "not a backend error at all, so Transient".
+
+## 6. Identity, names, marks, CreatedAt
+
+- **Names:** `NameFromUID(uid)` gives `proxy-` plus 16 lowercase base32
+  characters. It is a valid DNS label. A backend with stricter rules SHOULD
+  map it deterministically rather than invent a new scheme, because
+  idempotency depends on the name being the same every time. A backend that
+  keys on an allocated identifier follows `primary-key` (§4.10).
+- **providerID:** opaque to everyone except the provider. Keep it
+  `<location>/<name>`-shaped or `<location>/<allocated-key>`-shaped, parse
+  it strictly, and return `ErrPermanent` for a malformed one (§2.2).
+- **Marks:** `LabelManaged` with value `LabelManagedYes`, and `LabelUID`.
+  Use native labels or tags where the backend has them; otherwise write them
+  into metadata in the create call (§4.5).
+- **`CreatedAt`:** the backend's creation timestamp, or written into the mark
+  metadata at creation (§4.6).
+- **Provider names:** a configured `name` MUST be a lowercase DNS-1123 label
+  of at most 63 characters (`spec.provider` in
+  `api/v1alpha1/proxy_types.go`). The `ProviderHealth` object is named after
+  the provider; `Monitor.Start` warns and `Monitor.publish` skips the object
+  for a name that is not a valid object name. Config loading does not check
+  this yet (§13).
+
+## 7. Credentials, TLS and secret hygiene
+
+- **The constructor MUST NOT dial the backend.** `registry.Build` aborts the
+  whole provider set on a constructor error, and the manager exits: one
+  unreachable backend must not take the other providers down. Fail in the
+  constructor only on config errors.
+- **Credential files SHOULD be read lazily.** A mount marked `optional: true`
+  is empty when the Secret is missing; the constructor logs a warning and the
+  first call fails Transient until the Secret lands. A file that exists but
+  does not parse is Permanent.
+- **Credentials MUST NOT appear in logs, span attributes, error strings or
+  `Error.Err`.** Backend errors that echo a request URL or header with a
+  token in it MUST be sanitised before wrapping. The same holds for
+  `CloudInit` (§2.1). Test both with a sentinel (§11).
+- **TLS trust** follows `credentials` (§4.14). An insecure toggle MUST be
+  named so that it reads as insecure, default off, and MUST be logged as a
+  warning at startup when on.
+- **Rotation.** State in the spec whether rotating the Secret takes effect
+  without a restart. Reading files on every dial makes rotation free; caching
+  them after first load needs a restart and must say so.
+- **Scratch files** only under `/tmp` (§1.4), removed when the call ends.
+
+## 8. Optional interfaces: `Checker`
+
+`provider.Checker` is `Check(ctx) []TargetStatus` (`internal/provider/check.go`).
+`cmd/main.go` type-asserts it on the **unwrapped** provider before adding the
+decorators, and the monitor treats a nil Checker as unsupported, not failed
+(`Monitor.run`).
+
+- **MUST never fail as a whole.** Every failure is a `TargetStatus` with
+  `Ready=false`.
+  *Caller:* `Monitor.checkOnce` has no error path per provider.
+- **MUST return promptly once `ctx` ends.** The monitor gives the whole round
+  one timeout.
+- **MUST return one entry per target, in a stable order.** Targets are what
+  can fail independently: an apiserver, a project, each host or node.
+  *Caller:* the monitor logs transitions per target (`logTransitions`) and
+  the gauges are labelled `{provider,target}`.
+- **MUST NOT disrupt reconciles.** A check that shares a connection with
+  provisioning must not tear that connection down when it times out.
+- **SHOULD use the shared reasons** (`ReasonReachable`, `ReasonTimeout`,
+  `ReasonUnreachable`, `ReasonUnauthorized`, `ReasonForbidden`,
+  `ReasonAPIError`). Add provider-specific reasons only for failures an admin
+  would fix differently, and document each one in the guide.
+- **SHOULD check with the same credentials, verb and scope that `ListByTag`
+  and `Create` use**, and be cheap: one page, one object.
+- **MAY fill `Details`** with facts useful when triaging: versions, capacity,
+  the account in use.
+- The `status.type` field of `ProviderHealth`
+  (`api/v1alpha1/providerhealth_types.go`) lists the known types in its doc
+  comment; add the new type there and run `make manifests`.
+
+## 9. Observability
+
+Tracing is OpenTelemetry, set up in `internal/tracing` from the standard
+`OTEL_*` environment variables. With no exporter configured, spans are no-ops
+and logs are unchanged ([architecture.md §10](architecture.md#10-tracing-internaltracing)).
+
+**Layer 1, operation spans and metrics: nothing to write.** `cmd/main.go`
+wraps every provider in `WithTracing(WithMetrics(...))`:
+
+- `WithTracing` opens one client span per call, named `provider.create`,
+  `provider.get`, `provider.delete` or `provider.list`, with
+  `provider.name`, `provider.id` and `provider.result` (the error class) as
+  attributes. `ErrNotFound` keeps the span status Ok, because polling to
+  NotFound is expected.
+- `WithMetrics` counts each call as
+  `proxy_operator_provider_requests_total{provider,op,result}`
+  (`internal/metrics/metrics.go`). A `Checker` feeds
+  `proxy_operator_provider_target_up`,
+  `proxy_operator_provider_check_duration_seconds` and
+  `proxy_operator_provider_check_timestamp_seconds`, labelled
+  `{provider,target}`.
+
+Rules:
+
+- **MUST NOT open its own per-operation span or count its own requests.**
+  They would duplicate the decorators.
+- **MUST pass the caller's `ctx` to every backend call and take the logger
+  from `logf.FromContext(ctx)` at call time.** `tracing.StartSpan`
+  (`internal/tracing/logger.go`) puts a logger carrying `traceID` and
+  `spanID` into `ctx`. A logger cached at construction, or a backend call
+  made with `context.Background()`, loses the trace.
+- **Health checks bypass the decorators.** A `Checker` runs under the
+  monitor's `providerhealth.check` span, and the same `ctx` rules apply.
+
+**Layer 2, child spans per backend call: SHOULD.** One operation span says
+that `Create` took 40 seconds; child spans say which backend call took them.
+Follow `child-spans` (§4.18):
+
+- HTTP backends: wrap the transport with `otelhttp`, gated on an existing
+  parent span as `tracing.RestConfigWrapper` does
+  (`internal/tracing/transport.go`), so background traffic (token refreshes,
+  keepalives) does not become root spans. Check first whether the SDK already
+  does this.
+- Non-HTTP backends: start child spans yourself with
+  `tracing.Start(ctx, "<type>.<call>")` around each call and around long
+  steps such as a dial or a download, in a decorator over the seam, not at
+  call sites.
+- **MUST NOT put secrets or CloudInit in span attributes.** Put identifiers,
+  sizes and error codes there.
+
+**Layer 3, logs.** Follow `logging` (§4.17). Log lines join a trace through
+the `traceID` key from layer 1.
+
+## 10. Configuration and wiring
+
+The providers file is parsed by `LoadConfig` (`internal/provider/config.go`)
+with `UnmarshalStrict`, so unknown keys already fail.
+
+- Add a `<Type>Config` struct and a `ProviderConfig` field for it.
+- Add the block to `setBlocks`, so a block that does not match `Type` is
+  rejected.
+- Add the type to the allow-list in `Config.validate`. The same list exists a
+  second time as the constructor map in `cmd/main.go`; both must change
+  (§13).
+- Add required-field validation in its own `config_<type>.go` when it is
+  more than a few lines, with rows in `config_test.go`.
+- **Validate every config value that is pasted into an identifier**: a
+  location name that becomes part of the providerID must not contain the
+  separator; a catalogue key that becomes a file or volume name must be a
+  safe filename.
+- The constructor has the signature `registry.Constructor`:
+  `func(ctx, provider.ProviderConfig) (provider.Provider, error)`. Register
+  it in the map in `cmd/main.go`. Flags a provider needs are defined there
+  too and passed through a `NewWith...` variant, never read from the
+  environment inside the package.
+- Decorators are applied by the composition root, in the order
+  `WithTracing(WithMetrics(p))`, after the Checker has been collected from
+  the unwrapped provider.
+
+## 11. Testing obligations
+
+`CLAUDE.md` applies: stdlib `testing`, table-driven, tests next to source,
+`t.Parallel()` where safe.
+
+- **A fake behind the seam.** Every unit test runs against an in-memory
+  fake of the backend, never the network.
+- **Contract tests**, each against the fake:
+  - repeat `Create` returns the same ID and creates nothing;
+  - `Get` after `Delete` returns `ErrNotFound`;
+  - `Delete` twice succeeds;
+  - malformed ID is `ErrPermanent` from `Get` and `Delete`;
+  - an unknown backend state maps to `StateStopped`;
+  - `StateRunning` requires an IP, and a running instance without one is
+    `StateProvisioning`;
+  - `UID` and `CreatedAt` are read back from the fake, not remembered;
+  - `ListByTag` excludes unmarked instances and instances with an empty UID
+    are still listed with `UID == ""`;
+  - `Get` during an in-progress delete still returns the instance;
+  - `Get` during an asynchronous operation or lock returns the last state or
+    Transient (`async-ops`);
+  - a created but not yet started instance is never `StateStopped`
+    (`provisioning-state`);
+  - a cancelled `ctx` makes every method return promptly.
+- **Classify**: one table test covering every class, every backend code the
+  provider matches, and "not a backend error at all, so Transient". Use
+  captured real responses for message matching.
+- **Checker**: a table test for the reasons, a test that `Check` returns
+  promptly with an expired `ctx`, and a test that it returns one entry per
+  target in config order.
+- **Sentinel tests**: a `CloudInit` containing a sentinel string and a
+  credential containing another never appear in any log line, span attribute
+  or error string.
+- **Tracing**: with an in-memory span recorder
+  (`sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(tracetest.NewSpanRecorder()))`,
+  as in `internal/provider/tracing_test.go`), the child spans are children
+  of the operation span.
+- **Real-backend test** per `backend-test` (§4.19). It runs the whole
+  lifecycle: create, create again, poll until Running, list, delete, delete
+  again, confirm it is gone and that no secondary resource remains. Keep it
+  compiling with `go vet -tags=integration ./...`.
+
+## 12. Checklist for adding a provider
+
+1. **Spec first.** Copy `docs/provider-spec-template.md` to
+   `docs/<type>-provider-spec.md` and answer every entry. Unanswered entries
+   block implementation.
+2. **Package** `internal/provider/<type>/` with the provider type, a
+   compile-time assertion for `provider.Provider` and one for
+   `provider.Checker`, the seam interface, and a logger built from
+   `logf.FromContext(ctx).WithName("<type>").WithValues("provider", name)`.
+3. **Config** per §10, including `config_test.go` rows.
+4. **Constructor** `New(ctx, provider.ProviderConfig)` registered in
+   `cmd/main.go`; no dialing.
+5. **Contract** per §2, tested per §11.
+6. **Errors** per §5, `classify` plus its table test, `ProviderConfig.Name`
+   in every `Wrap`.
+7. **Checker** per §8, with the `status.type` comment updated and
+   `make manifests` run.
+8. **Tracing and logging** per §9 and §4.17, §4.18.
+9. **Real-backend test** per §4.19, with a `make test-integration-<type>`
+   target and a section in `docs/testing.md`.
+10. **RBAC**: if the provider talks to the cluster's own API, add
+    `+kubebuilder:rbac` markers in their own comment block, never inside a
+    doc comment, then `make manifests` and check
+    `git diff config/rbac/role.yaml`.
+11. **Deployment**: an `optional: true` credentials mount in
+    `config/manager/manager.yaml`; a commented entry in
+    `config/manager/providers_config.yaml` and
+    `config/samples/providers-config.yaml`; a `config/samples/proxy_<type>.yaml`
+    whose cloud-init installs a proxy on the sample image; a Makefile helper
+    for the Secret if there is one; `hack/providers-dev.yaml` only if the
+    type needs no credentials.
+12. **Docs**: `docs/<type>-provider.md` (prerequisites, config reference,
+    placement mapping, what lands on the backend, the health check, logging,
+    known gaps); a row and a section in `providers-existing.md`; a row in the
+    `architecture.md` components table and a block in its §5 and §11
+    (§10 too if the provider adds spans); a row in the README "Implemented
+    providers" table; a `CHANGELOG.md` entry once the user has confirmed it
+    works.
+13. **Optional:** a demo script under `docs/demo/`.
+
+## 13. Known limits of the contract
+
+These are facts a new provider has to live with today. None is designed or
+scheduled.
+
+- **Adoption is not verified.** `Create` adopts any existing instance with
+  the right name without checking its `LabelUID`.
+- **`StateStopped` and `StateTerminated` are indistinguishable** to the
+  reconciler; both mean delete and recreate.
+- **Config is read once per process.** Adding a provider or a location, or
+  rotating credentials a provider caches, needs a manager restart.
+- **GC has no installation identity.** Two installs sharing one backend
+  delete each other's instances, because GC only knows its own live UIDs.
+- **An unconfigured provider latches the Proxy.** A Proxy naming a provider
+  missing from the config is latched `PermanentError`; adding the provider
+  and restarting does not change `metadata.generation`, and `spec.provider`
+  is immutable, so the Proxy stays latched until it is recreated.
+- **Provider names are not validated** as DNS-1123 at config load.
+- **The type list exists twice**: `Config.validate` and the constructor map
+  in `cmd/main.go`.
+- **`Delete` is resubmitted on every deletion poll** until `Get` says
+  NotFound.
+- **Callers set no deadline** (§1.5); each provider bounds itself.
+- **There is no conformance suite.** The §11 contract tests are written per
+  provider; `internal/provider/providertest` is a proposal in the appendix.
+- **Stale comments** in `internal/provider/provider.go`,
+  `internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
+  still describe a `mock` provider that no longer exists.
+
+## 14. Related documents
+
+- [providers.md](providers.md): the index.
+- [provider-spec-template.md](provider-spec-template.md): the questionnaire.
+- [providers-existing.md](providers-existing.md): how the existing providers
+  meet this contract, and their gaps.
+- [architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world),
+  [§10](architecture.md#10-tracing-internaltracing) and
+  [§11](architecture.md#11-provider-health-checks-internalproviderhealth).
+- [testing.md](testing.md).

docs/provider-spec-template.md

@@ -0,0 +1,455 @@
+# Provider spec template
+
+Copy this file to `docs/<type>-provider-spec.md`, fill in every **Answer**
+line, and hand the result to whoever implements the provider together with
+[provider-contract.md](provider-contract.md) and
+[providers-existing.md](providers-existing.md). An unanswered entry blocks
+implementation: it is a decision the contract leaves to the provider, and the
+implementer must not guess it.
+
+Where an entry names an axis in backticks, the allowed answers are the
+profiles of that axis in contract §4, and the entry number matches the last
+column of the contract's §4.20 table. **Evidence** is where the answer was
+confirmed: a link into the backend's API documentation, or the experiment
+that was run. "Verify against the target version" is a valid interim
+answer only for the Evidence line, never for the Answer.
+
+## Header
+
+- **Type** (the `type:` value in the providers config, lowercase):
+- **Example configured name**:
+- **Backend and API version this spec was written against**:
+- **Client** (`vendor-sdk` with the module path, or `thin-client`):
+- **Author and date**:
+
+## 1. Identity and providerID
+
+**Question.** What is the providerID's shape, which location parts does it
+embed, and what validates the config values pasted into it?
+
+**Allowed answers.** `<location>/<name>`, `<location>/<allocated-key>`, or a
+location-free `<allocated-key>` (contract §6; required when instances can
+migrate, see entry 2).
+
+**Obligations.** Strict parsing; `ErrPermanent` for a malformed ID from `Get`
+and `Delete`; a regexp on every config value that becomes an ID segment, in
+`config_<type>.go`.
+
+**Answer:**
+
+**Evidence:**
+
+## 2. Primary key (`primary-key`)
+
+**Question.** Is `req.Name` the backend's key, or does the backend allocate an
+identifier?
+
+**Allowed answers.** `name-is-key`; `allocated-id`.
+
+**Obligations.** `allocated-id`: `Create` finds an existing instance by name
+before allocating; allocation is safe against a concurrent `Create` for
+another Proxy (deterministic from `req.UID`, or allocate-then-verify); `Get`
+and `Delete` use the allocated key; state how a collision with an instance
+the operator does not own is detected and reported.
+
+**Answer:**
+
+**Evidence:**
+
+## 3. Placement mapping
+
+**Question.** What do `Placement.Region`, `Zone`, `MachineType` and `Image`
+mean for this backend, which are required, which are ignored, and what
+happens when a value is unknown?
+
+**Allowed answers.** Per field: a config catalogue key, a backend identifier
+passed through, or ignored (contract §2.1, MAY ignore with documentation).
+
+**Obligations.** Unknown values are rejected with `ErrPermanent` naming the
+field and value before any backend call; ignored fields are listed in the
+provider's guide because they are still in the spec hash; an empty value with
+a single-choice catalogue may default, and the guide says so.
+
+**Answer:**
+
+**Evidence:**
+
+## 4. Image source
+
+**Question.** Where does the boot image come from, and what does a catalogue
+edit do to running instances?
+
+**Allowed answers.** A catalogue in the config block (key to URL or backend
+image ID); a backend template or image referenced directly by
+`Placement.Image`; a URL fetched by the provider into the backend on first
+use.
+
+**Obligations.** If the provider fetches or converts images: streaming, no
+buffering (contract §1.4), a completeness marker so a cut-off fetch is redone,
+concurrency control so two Creates fetch once, and a written warning that
+replacing an image in use may corrupt instances that depend on it. If the
+backend owns the image: the prerequisite is documented in the guide.
+
+**Answer:**
+
+**Evidence:**
+
+## 5. Cloud-init delivery
+
+**Question.** How does `CreateRequest.CloudInit` reach the guest, byte for
+byte?
+
+**Allowed answers.** User-data metadata on the instance; a NoCloud seed
+built in-process and attached as media; a file placed on the backend and
+referenced from the instance config; not delivered (documented, with how the
+proxy gets configured instead).
+
+**Obligations.** Size limits of the channel stated; whether vendor-data or a
+second document can ride along; the sentinel test (contract §11); the channel
+must be fully reachable through the API the provider uses (no step that needs
+shell access on the backend unless the spec says the provider has it).
+
+**Answer:**
+
+**Evidence:**
+
+## 6. Machine type mapping
+
+**Question.** How does `Placement.MachineType` become CPU, memory and disk?
+
+**Allowed answers.** A catalogue of named profiles in the config block; a
+backend size name passed through.
+
+**Obligations.** A catalogue needs validation of each profile; disk size
+default and its config field named.
+
+**Answer:**
+
+**Evidence:**
+
+## 7. Network and IP discovery (`readiness`, `ip-source`)
+
+**Question.** Which network does the instance attach to, where does its
+address come from, how long until it is usable, and what makes it reachable
+from the operator pod and from clients on `spec.port`?
+
+**Allowed answers.** `readiness`: `immediate`; `provisioning-window`.
+`ip-source`: `assigned-at-create`; `discovered` (name the mechanism: guest
+agent, DHCP lease, neighbour table, and the order if several).
+
+**Obligations.** `discovered`: what the guest must run, how it gets
+installed, what happens without it; discovery failure is Provisioning, not an
+error; only routable IPv4 is reported (state the filter). Reachability:
+firewall or routing steps the provider performs, or the prerequisite the
+guide states. Whether `Port` is used by the provider at all.
+
+**Answer:**
+
+**Evidence:**
+
+## 8. GC marks and CreatedAt (`gc-mark`, `created-at`)
+
+**Question.** Where do `LabelManaged` and `LabelUID` live, where does
+`CreatedAt` come from, and which secondary resources are marked?
+
+**Allowed answers.** `gc-mark`: `native-labels`; `written-metadata`;
+`mark-window`. `created-at`: `backend-timestamp`; `written-at-create`.
+
+**Obligations.** `mark-window`: the window's length in seconds, why it is
+far below `MinAge`, and how the same Proxy's next reconcile cleans an
+unmarked instance. Any value-format constraint of the backend's labels
+(allowed characters, length) and how a Proxy UID fits it.
+
+**Answer:**
+
+**Evidence:**
+
+## 9. State mapping (`provisioning-state`)
+
+**Question.** How does every backend state map to `StateProvisioning`,
+`StateRunning`, `StateStopped`, `StateTerminated`, and how does `Get` tell
+"defined but never started" from "stopped"?
+
+**Allowed answers.** A table of backend state (plus lock or task flags) to
+provider state, with the unknown-state row mapping to `StateStopped`.
+`provisioning-state`: `native`; `create-blocks`; `marker`.
+
+**Obligations.** Running requires an IP; `marker` names what is recorded at
+creation and how `Get` reads it; `create-blocks` names the cap on the wait.
+
+**Answer:**
+
+| Backend state | Lock or task | Provider state |
+|---|---|---|
+| | | |
+
+**Evidence:**
+
+## 10. Create: blocking and task tracking (`create-return`, `async-ops`)
+
+**Question.** What does `Create` wait for before returning, and how are
+asynchronous backend tasks tracked?
+
+**Allowed answers.** `create-return`: `submit`; `blocking`. `async-ops`:
+`none`; `task-handle`; `backend-lock`.
+
+**Obligations.** `task-handle`: where the handle is kept and how a failed
+task surfaces (in `Get`, as which class, with which message). `backend-lock`:
+what `Get` and `Delete` return while locked. Every step of `Create` is
+idempotent so a cut-off attempt resumes; list the steps and what each finds
+on a retry.
+
+**Answer:**
+
+**Evidence:**
+
+## 11. Delete and secondary resources (`delete-return`, `secondary-resources`)
+
+**Question.** In what order does `Delete` tear things down, what does it do
+when called again mid-delete, and what does `Get` return meanwhile?
+
+**Allowed answers.** `delete-return`: `returns-early`; `waits`.
+`secondary-resources`: `none`; `cascade` (name the flags set at create);
+`explicit` (name each resource and how it is found by name).
+
+**Obligations.** Idempotent at every step; "already deleting" is success and
+cheap; a location no longer in the config is a logged no-op success; any
+resource that can outlive the instance without being listable is a documented
+GC blind spot.
+
+**Answer:**
+
+**Evidence:**
+
+## 12. ListByTag scope and partial failure (`list-partial`)
+
+**Question.** What is the full scope `ListByTag` enumerates, and what happens
+when part of it is unreachable?
+
+**Allowed answers.** `fail-whole`; `omit-unreachable`.
+
+**Obligations.** The scope covers everything the provider can create in;
+server-side filtering on the mark where the backend allows it; a single
+unreadable instance may be skipped with a log line; the cost of one list at
+fleet size.
+
+**Answer:**
+
+**Evidence:**
+
+## 13. Error classification (`not-found`, `quota`)
+
+**Question.** Which backend responses map to which of the four classes, and
+which signal means "does not exist"?
+
+**Allowed answers.** A table of backend status or code (with message pattern
+where needed) to class. `not-found`: `status-404`; `error-code`;
+`list-and-miss`. `quota`: `detected`; `absent`.
+
+**Obligations.** One `classify`, a table test with captured real responses
+for every matched message; the permission-denied-versus-quota case decided;
+authentication failures, host-key or certificate failures, and cancellation
+each assigned a class with a reason.
+
+**Answer:**
+
+| Backend response | Class | Why |
+|---|---|---|
+| | | |
+
+**Evidence:**
+
+## 14. Credentials and TLS (`credentials`)
+
+**Question.** What credential does the operator use, where does it come from,
+when is it read, and how is the backend's TLS certificate trusted?
+
+**Allowed answers.** `mounted-files` (Secret name, mount path, file names,
+`optional: true`); `environment`; `workload-identity`. TLS: `system-roots`;
+`ca-file`; `insecure` (the flag's exact name).
+
+**Obligations.** The minimum privilege set the credential needs, listed;
+rotation behaviour (free, or restart needed); the sentinel test for the
+credential (contract §7); the constructor never dials.
+
+**Answer:**
+
+**Evidence:**
+
+## 15. Config block
+
+**Question.** What are the fields of `<Type>Config`, their defaults, and their
+validation?
+
+**Allowed answers.** A table: field, required, default, validation, and which
+identifier or path the value ends up in.
+
+**Obligations.** Contract §10: `setBlocks`, the type allow-list in
+`Config.validate` and the constructor map in `cmd/main.go`, `config_<type>.go`
+with its tests, regexps for values pasted into IDs or file names.
+
+**Answer:**
+
+| Field | Required | Default | Validation | Ends up in |
+|---|---|---|---|---|
+| | | | | |
+
+**Evidence:**
+
+## 16. Checker
+
+**Question.** What are the check targets, what does one check do, which
+reasons can it report, and what goes into `Details`?
+
+**Allowed answers.** One target per thing that fails independently; the
+shared reasons from `internal/provider/check.go` first, provider-specific
+reasons only with a stated "what the admin does differently"; `Details` keys
+listed.
+
+**Obligations.** Same credential, verb and scope as `ListByTag` and `Create`;
+cheap (one page, one object); never disrupts a reconcile that shares a
+connection; returns promptly on `ctx`; the `status.type` comment in
+`api/v1alpha1/providerhealth_types.go` updated.
+
+**Answer:**
+
+**Evidence:**
+
+## 17. Client and child spans (`client-kind`, `child-spans`)
+
+**Question.** What talks to the backend, and how do its calls appear under the
+operation span?
+
+**Allowed answers.** `client-kind`: `vendor-sdk` (module path, and whether it
+is maintained); `thin-client`; `rpc-over-ssh`. `child-spans`:
+`sdk-instrumentation`; `transport-wrapper`; `span-decorator`; `none`.
+
+**Obligations.** The seam interface's name and the fake that implements it;
+`thin-client`: one error type carrying status and body, transport timeouts,
+parent-gated `otelhttp`; `none` is a documented gap; the span recorder test
+(contract §11).
+
+**Answer:**
+
+**Evidence:**
+
+## 18. Timeouts (`timeouts`)
+
+**Question.** What bounds each backend call, the dial, and any long step such
+as a download or a wait loop, and what does a cancelled `ctx` do to a shared
+connection?
+
+**Allowed answers.** `per-call` (the budgets, per operation); `ctx-only`
+(cite the client's own timeout).
+
+**Obligations.** Contract §1.5: no caller sets a deadline; a hung call pins a
+worker or stalls GC. Cancellation behaviour on a shared connection stated
+(abandon the call, or close the connection).
+
+**Answer:**
+
+**Evidence:**
+
+## 19. Concurrency
+
+**Question.** What is shared between concurrent calls (connections, caches,
+locks), and what happens when two calls for the same instance name overlap?
+
+**Allowed answers.** A description of the shared state and its locking.
+
+**Obligations.** Safe for 3 reconciles, a GC sweep and a check round at once;
+two `Delete` calls for one instance overlap safely; one fetch per image per
+location when several Creates race.
+
+**Answer:**
+
+**Evidence:**
+
+## 20. Logging (`logging`)
+
+**Question.** Which tiers does the provider log, with which keys on every
+line?
+
+**Allowed answers.** `tiered`; `outcome-only`; `none` (cite the client's own
+logging).
+
+**Obligations.** Logger from `logf.FromContext(ctx)` on every call; never
+`CloudInit` or credentials; `provider` and the instance name on every line
+that concerns an instance.
+
+**Answer:**
+
+**Evidence:**
+
+## 21. Tests (`backend-test`)
+
+**Question.** What is the fake, which contract tests exist, and how is the
+provider proven against the real backend?
+
+**Allowed answers.** `backend-test`: `cluster-e2e`; `integration-tag` (the
+`<TYPE>_TEST_*` variables, the `make test-integration-<type>` target); `manual-runbook`.
+
+**Obligations.** Every item of the contract §11 list, ticked here:
+
+- [ ] repeat `Create`
+- [ ] `Get` after `Delete`
+- [ ] `Delete` twice
+- [ ] malformed ID
+- [ ] unknown state maps to `StateStopped`
+- [ ] `StateRunning` requires an IP
+- [ ] `UID` and `CreatedAt` read back
+- [ ] `ListByTag` excludes unmarked instances
+- [ ] `Get` during an in-progress delete
+- [ ] `Get` during an asynchronous operation or lock
+- [ ] never started is never `StateStopped`
+- [ ] cancelled `ctx` returns promptly
+- [ ] `classify` table with captured responses
+- [ ] Checker reasons, expired `ctx`, stable order
+- [ ] sentinel tests for `CloudInit` and the credential
+- [ ] child spans under the operation span
+- [ ] real-backend lifecycle test
+
+**Answer:**
+
+**Evidence:**
+
+## 22. Deployment
+
+**Question.** What changes in `config/` and the Makefile?
+
+**Allowed answers.** The list: the `optional: true` mount in
+`config/manager/manager.yaml`; the commented entries in
+`config/manager/providers_config.yaml` and
+`config/samples/providers-config.yaml`; `config/samples/proxy_<type>.yaml`
+with a cloud-init that installs a proxy on the sample image; a Makefile
+helper for the Secret; RBAC markers if the provider talks to the cluster's
+own API; whether `hack/providers-dev.yaml` gets an entry.
+
+**Answer:**
+
+**Evidence:**
+
+## 23. Docs
+
+**Question.** Which documents are written or updated?
+
+**Allowed answers.** `docs/<type>-provider.md` with its section list; the row
+and section in `providers-existing.md`; the `architecture.md` rows (components
+table, §5, §11, and §10 if spans are added); the README row; the
+`docs/testing.md` section; the `CHANGELOG.md` entry after confirmation.
+
+**Answer:**
+
+**Evidence:**
+
+## 24. Known limits accepted at design time
+
+**Question.** Which of the contract's §13 limits bite this provider in
+practice, and which new gaps does this design accept?
+
+**Allowed answers.** A list, each with the consequence for an operator and
+the workaround.
+
+**Answer:**
+
+**Evidence:**

docs/providers-existing.md

@@ -0,0 +1,392 @@
+# The existing providers
+
+This is the appendix to [provider-contract.md](provider-contract.md): how the
+three providers that exist today (kubernetes, gcp, kvm) meet the contract,
+which profile each one chose on every axis, and what is still missing. It is
+reference material for whoever writes the next provider, not a user guide;
+the user guides are [kvm-provider.md](kvm-provider.md),
+[gcp-vm-validation.md](gcp-vm-validation.md) and
+[gcp-in-specific-project.md](gcp-in-specific-project.md). The README's
+"Implemented providers" table stays the short public summary.
+
+The per-provider call flows are drawn in
+[architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world),
+the health checks are tabulated in
+[§11](architecture.md#11-provider-health-checks-internalproviderhealth), and the
+judgment calls behind several rows below are recorded under its "Decisions"
+list. They are linked, not repeated.
+
+Code references are a file path plus a function or symbol name.
+
+## 1. Profile table
+
+One row per axis of contract §4, in the same order, with the profile name
+and the symbol that shows it.
+
+| Axis | kubernetes | gcp | kvm |
+|---|---|---|---|
+| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) |
+| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) |
+| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
+| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) |
+| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) |
+| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) |
+| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) |
+| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) |
+| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) |
+| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` |
+| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning |
+| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none |
+| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
+| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) |
+| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) |
+| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection |
+| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier |
+| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans |
+| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` |
+
+## 2. Other differences
+
+Rows that are not profile axes but that a new provider author compares
+against.
+
+| Feature | kubernetes | gcp | kvm |
+|---|---|---|---|
+| Create verifies the adopted instance's UID | no | no | no |
+| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image |
+| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
+| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no |
+| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped |
+| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a |
+| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes |
+| Checker target | `apiserver` | the project | each host, concurrently |
+| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` |
+| Credential rotation without restart | n/a | via ADC | no: cached after first successful load |
+| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) |
+| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` |
+| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
+| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN |
+| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself |
+
+## 3. Observability status
+
+| Layer | kubernetes | gcp | kvm |
+|---|---|---|---|
+| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators |
+| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** |
+| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) |
+| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no |
+
+## 4. kubernetes (`internal/provider/kubernetes`)
+
+- **Package and seam.** `kubernetes.go`, `pod.go`, `check.go`. There is no
+  seam interface: `newWithClient` injects a controller-runtime
+  `client.Client`, and tests use the fake client. `New` calls
+  `NewWithTransportWrapper`, which builds its own uncached client from
+  `ctrl.GetConfig()` with an optional `transport.WrapperFunc` for tracing.
+- **Config.** `KubernetesConfig.Image` only, defaulting to the Squid image
+  constant in `kubernetes.go`.
+- **ID.** `providerID(namespace, name)` gives `namespace/name`;
+  `parseProviderID` uses `cache.SplitMetaNamespaceKey` and rejects an empty
+  namespace with `ErrPermanent`. An empty name part is not rejected.
+- **Create.** `buildPod` (pure) then `client.Create`; `IsAlreadyExists` is
+  success. Only `Name`, `Namespace`, `UID` and `Port` are used. The Squid
+  config is generated by `squidConf` from the port and passed through the
+  `SQUID_CONF` environment variable, so there is exactly one object per
+  proxy.
+- **Get and state.** `stateFromPod`:
+
+  | Pod phase | PodIP | Provider state |
+  |---|---|---|
+  | Pending | any | Provisioning |
+  | Running | empty | Provisioning |
+  | Running | set | Running |
+  | Succeeded, Failed, Unknown | any | Terminated |
+
+  `instanceFromPod` sets `IP` only when the state is Running. A Pod with a
+  deletion timestamp still reports its phase. `UID` from the `LabelUID`
+  label, `CreatedAt` from `CreationTimestamp`.
+- **Delete.** `client.Delete`, `IsNotFound` ignored.
+- **ListByTag.** One `client.List` across all namespaces with
+  `MatchingLabels{LabelManaged: LabelManagedYes}`; this is what the
+  cluster-wide Pod RBAC is for.
+- **Errors.** `classify`: `IsNotFound` NotFound; `IsTooManyRequests`,
+  `IsServerTimeout`, `IsTimeout` Transient; `IsForbidden`, `IsInvalid`,
+  `IsBadRequest`, `IsUnauthorized` Permanent; default Transient. Nothing is
+  Quota, because a quota 403 and an RBAC 403 cannot be told apart.
+- **Check.** Lists at most one managed Pod cluster-wide (`client.Limit(1)`),
+  proving read and list access, not create. Target `apiserver`, address the
+  rest config host. `checkReason`: context errors and timeouts
+  `ReasonTimeout`; `IsUnauthorized`; `IsForbidden`; any `APIStatus`
+  `ReasonAPIError`; else `ReasonUnreachable`.
+- **Logging and tracing.** No log lines in the package. Child spans come
+  from the parent-gated transport wrapper.
+- **Tests.** `kubernetes_test.go`, `pod_test.go`, `check_test.go` over the
+  fake client; the kind e2e (`test/e2e`) is the real-backend run.
+- **Deployment.** RBAC markers for Pods live next to the reconciler's in
+  `internal/controller/proxy_controller.go`. `hack/providers-dev.yaml` holds
+  only this provider, for `make run-dev`.
+- **Known gaps.**
+  - `Error.Provider` is hardcoded to `"kubernetes"`, so errors and logs show
+    the wrong name when the configured name differs.
+  - `CloudInit` and all of `Placement` are ignored.
+  - No quota class: a quota 403 is Permanent.
+  - A Pod being deleted still reports its phase, so it can read as Running.
+  - No securityContext, no resources, and `http_access allow all` in
+    `pod.go`.
+  - No logging in the package, and no Checker `Details`.
+  - No `OwnerReference` on Pods: a bypassed finalizer leaves the Pod behind.
+  - The Pod RBAC is always installed, even when no kubernetes provider is
+    configured.
+
+## 5. gcp (`internal/provider/gcp`)
+
+- **Package and seam.** `gcp.go`, `insert.go`, `errors.go`, `check.go`,
+  `wirelog.go`. The seam is `instancesAPI` (`Insert`, `Get`, `Delete`,
+  `AggregatedList`, `Probe`); `realInstances` adapts the SDK client and
+  `newWithAPI` is the test constructor. `New` calls `NewWithWireOptions`,
+  which takes the `--gcp-wire-log-full-payloads` flag from `cmd/main.go`.
+- **Config.** `GCPConfig`: `project` required; `network`, `networkTag`,
+  `diskSizeGb` defaulted in `withDefaults`.
+- **ID.** `formatProviderID(zone, name)` gives
+  `zones/<zone>/instances/<name>`; `parseProviderID` rejects anything else
+  with `ErrPermanent`. The project comes from the config.
+- **Create.** Validates `Zone`, `MachineType` and `Image` up front and
+  returns `ErrPermanent` naming the missing field. `buildInsertRequest`
+  (pure) builds one NIC with an ephemeral external IP (`ONE_TO_ONE_NAT`) and
+  a boot disk with `AutoDelete`, labelled with both marks. `Insert` is
+  submitted and never awaited; 409 (`isAlreadyExists`) is success.
+- **Get and state.** `mapState(status, ip)`:
+
+  | Instance status | NAT IP | Provider state |
+  |---|---|---|
+  | PROVISIONING, STAGING, REPAIRING | any | Provisioning |
+  | RUNNING | empty | Provisioning |
+  | RUNNING | set | Running |
+  | STOPPING, STOPPED, SUSPENDING, SUSPENDED | any | Stopped |
+  | TERMINATED | any | Terminated |
+  | anything else | any | Stopped |
+
+  `toInstance` reads the NAT IP of the first access config of the first
+  NIC, `UID` from the label, and `CreatedAt` from `CreationTimestamp` with
+  the parse error dropped (zero time on failure).
+- **Delete.** Submits a delete operation and returns; 404 (`isNotFound`) is
+  success. Every deletion poll submits a new operation.
+- **ListByTag.** `AggregatedList` with `managedFilter` on the managed label
+  only and `ReturnPartialSuccess`, so an unreachable zone is left out
+  silently; the zone is recovered from the URL by `lastPathSegment`.
+- **Errors.** `classify`: non-`googleapi` errors Transient; 404 NotFound;
+  429 Quota; 403 with reason `quotaExceeded` or `rateLimitExceeded`
+  (`hasReason`) Quota; 400, 401 and other 403s Permanent; other codes
+  Transient. `logAPIError` records the HTTP status and reasons at V(1)
+  because `classify` collapses them.
+- **Check.** `Probe` makes one `AggregatedList` page with `maxResults=1`
+  and the managed filter: credentials, the Compute API and
+  `compute.instances.list` in one call. Target is the project, address
+  `compute.googleapis.com`. `checkReason`: context errors `ReasonTimeout`;
+  non-`googleapi` errors `ReasonUnreachable`; 401; 403; else
+  `ReasonAPIError`.
+- **Logging.** `logger(ctx)` from the context, named `gcp` with
+  `provider`. V(2) for the curated insert request (never the user-data),
+  V(1) per operation outcome, V(2) per listed instance. The wire log in
+  `wirelog.go` runs at V(5) with payloads elided unless the flag is set.
+- **Tracing.** The SDK adds `otelhttp` itself unless telemetry is disabled;
+  not parent-gated.
+- **Tests.** `insert_test.go` (pure builder), `errors_test.go` (code
+  table), `gcp_test.go` (IDs, 409, state mapping, list filter and partial
+  success) over `fakeAPI`; `check_test.go` also drives the real SDK client
+  against `httptest` to prove exactly one request with `maxResults=1`.
+  Real backend: the manual runbook in [gcp-vm-validation.md](gcp-vm-validation.md).
+- **Deployment.** Credentials by ADC; the runbook mounts a service-account
+  key and points `GOOGLE_APPLICATION_CREDENTIALS` at it. A commented entry
+  in both providers configs, `config/samples/proxy_gcp.yaml`, and demo
+  scripts under `docs/demo/`.
+- **Known gaps.**
+  - Asynchronous insert failures are never read: an insert the API accepts
+    but that fails later (quota, bad image) is seen only as NotFound on the
+    next `Get`, and the reconciler creates again, so the reason never
+    reaches status. [gcp-vm-validation.md](gcp-vm-validation.md#gotchas)
+    still says such errors do reach status.
+  - External IP only: a VM without an external NAT IP stays Provisioning
+    forever.
+  - No firewall management: the rule for the proxy port is a manual
+    prerequisite, and `req.Port` is unused.
+  - The SDK client is never closed, and `Placement.Region` is unused.
+  - Wrong label value in the docs: `gcp-vm-validation.md` filters on
+    `labels.proxy-operator-managed=yes`; the value is `true`.
+  - The sample cannot work: `config/samples/proxy_gcp.yaml` uses a
+    `debian-cloud` image, which has no cloud-init, so its Squid install never
+    runs ([gcp-in-specific-project.md](gcp-in-specific-project.md) uses
+    Ubuntu for that reason).
+
+## 6. kvm (`internal/provider/kvm`)
+
+The full user guide is [kvm-provider.md](kvm-provider.md); this section is
+the author's view.
+
+- **Package and seam.** `kvm.go` (the four methods), `conn.go` (SSH dial,
+  `hostPool`, `libvirtAPI`), `image.go` (`imageCache`), `seed.go` (NoCloud
+  ISO and vendor-data), `domainxml.go`, `volume.go`, `metadata.go`, `id.go`,
+  `errors.go`, `check.go`, `trace.go`. The seam is `libvirtAPI`, the twenty
+  go-libvirt calls the provider uses; `fake_test.go` implements it in memory
+  and `newWithDialer` injects the dialer and the HTTP client.
+- **Config.** `KVMConfig` in `internal/provider/config_kvm.go`:
+  `sshKeyFile`, `knownHostsFile` or `insecureIgnoreHostKey`, `diskSizeGb`,
+  `vendorData`, `installGuestAgent`, `images` (`url`, `sha256`, `format`,
+  `firmware`), `machineTypes` (`vcpus`, `memoryMiB`), `hosts` (`name`,
+  `address`, `user`, `socket`, `pool`, `bridge`). `validateKVM` restricts
+  image keys and host names with `kvmIdentifier` because they become a volume
+  name and an ID segment.
+- **ID.** `formatProviderID(host, name)` gives `hosts/<host>/domains/<name>`.
+- **Create.** `host` resolves `Placement.Zone` (optional with one host),
+  the machine type must be a catalogue key, `resolveImage` accepts a
+  catalogue key or an http(s) URL. Then, inside one `hostPool.call`:
+  `DomainLookupByName`, and if found `ensureRunning`; otherwise
+  `imageCache.ensure` (download on first use, streamed through the operator
+  into the pool with a sha256 marker volume), `ensureOverlay` (reused if it
+  exists), `writeSeed` (always rebuilt: a cut-off upload must not be
+  trusted), `defineDomain` with the metadata element, `ensureRunning`.
+  `Create` returns only when the domain is running.
+- **Get and state.** `mapState(state, ip)`:
+
+  | libvirt state | IP | Provider state |
+  |---|---|---|
+  | nostate | any | Provisioning |
+  | running, blocked | empty | Provisioning |
+  | running, blocked | set | Running |
+  | paused, shutdown, shutoff, crashed, pmsuspended | any | Stopped |
+
+  `address` asks the ARP table first, then the guest agent, and
+  `firstIPv4` drops loopback, link-local and unspecified addresses. Metadata
+  is read back by `metadata` and parsed by `parseInstanceMetadata`; a domain
+  without the element reports an empty UID.
+- **Delete.** `removeDomain` (destroy, tolerating not running or gone; then
+  `DomainUndefineFlags` with NVRAM, managed save and snapshot metadata) and
+  `removeVolumes` (overlay and seed by name, a missing pool is a no-op). A
+  host no longer in the config is a logged no-op success.
+- **ListByTag.** `ConnectListAllDomains` on every host; `managedInstance`
+  keeps domains whose metadata says managed and skips a domain whose
+  metadata or state cannot be read. One unreachable host fails the whole
+  list.
+- **Errors.** `classify`: `isClassified` errors keep their class;
+  `ErrNoDomain`, `ErrNoStorageVol` NotFound; XML, unsupported,
+  invalid-argument, auth and access errors, and a missing pool, Permanent;
+  everything else Transient, including SSH handshake and auth failures.
+  Nothing is Quota. `logAPIError` adds `libvirtErrorFields`.
+- **Check.** `checkHost` per host concurrently through `hostPool.probe`,
+  not `call`, so a check that runs out of time abandons its RPCs instead of
+  closing the connection a reconcile may be using. Two cheap RPCs
+  (`ConnectGetLibVersion`, `ConnectGetHostname`) then the pool must exist
+  and be running. Reasons: the shared ones plus `SSHMaterialUnavailable`,
+  `SSHHandshakeFailed`, `LibvirtConnectFailed`, `LibvirtError`,
+  `PoolNotFound`, `PoolNotActive`, chosen by `checkReason` from the
+  `dialError` stage. Details: `user`, `pool`, `hostname`, `libvirtVersion`,
+  `poolState`, `poolCapacity`, `poolAvailable`.
+- **Connections.** `hostPool` keeps one SSH plus libvirt connection per
+  host, dialled lazily (`lazySSH.dial`, `sshAuth.dial`, `dialSSH`,
+  `connectLibvirt`), with a 15s `keepalive`. `hostEntry.acquire` is a
+  one-slot semaphore so waiting can end with the caller's context. A
+  cancelled `call` closes the shared connection to unblock the RPC;
+  `probe` abandons it and leaves the connection open. The SSH material is
+  read once by `lazySSH.load` and cached.
+- **Logging.** Logger from the context, named `kvm`, with `provider`,
+  `host` and `name`. V(1): one line per operation outcome and one
+  `libvirt call` line per RPC from the `tracingAPI` decorator in `trace.go`
+  (rpc name, identifiers, duration, error fields). V(2): domain and volume
+  XML, metadata, addresses. Upload and download bodies are never logged.
+- **Tracing.** No child spans; `trace.go` is a log decorator.
+- **Tests.** Pure builders (`domainxml_test.go`, `volume_test.go`,
+  `seed_test.go`, `id_test.go`, `metadata_test.go`); `errors_test.go`,
+  `conn_test.go` (pool behaviour and a real in-process SSH server);
+  `image_test.go`, `kvm_test.go` (Create resumes after every partial step,
+  every Delete already-gone case, ListByTag policies, logging tiers, a
+  cloud-init sentinel never logged); `trace_test.go`; `check_test.go`. Real
+  backend: `kvm_integration_test.go` behind `//go:build integration`, driven
+  by `KVM_TEST_*` variables and `make test-integration-kvm`, described in
+  [testing.md](testing.md#the-kvm-integration-test-real-hypervisor).
+- **Deployment.** The optional `kvm-ssh` Secret mounted at
+  `/etc/proxy-operator/kvm` (`config/manager/manager.yaml`),
+  `make kvm-ssh-secret`, a commented entry in both providers configs,
+  `config/samples/proxy_kvm.yaml`.
+- **Known gaps** (see also [kvm-provider.md § Known gaps](kvm-provider.md#known-gaps)).
+  - No quota or capacity detection.
+  - `Create` blocks, including the first image download on each host.
+  - GC blind spots: volumes without a domain are invisible to GC, and so are
+    VMs on a host removed from the config, which `Delete` treats as done.
+  - SSH material is cached after the first successful load, so rotating
+    keys needs a restart.
+  - Cancelling one operation cuts off the others on that host: a cancelled
+    `call` closes the shared per-host connection.
+  - SSH auth failures retry forever: SSH authentication and host-key
+    failures are Transient, while libvirt auth errors are Permanent.
+  - `Create` can return `ErrNotFound` when a volume or domain disappears
+    during the call.
+  - No reused-overlay check: an existing overlay is reused without checking
+    its backing image or size.
+  - Sample mismatch: `config/samples/proxy_kvm.yaml` places on host `nas1`,
+    while both providers configs define only `beelink`.
+  - Doc drift in kvm-provider.md: it says ARP is the fallback, but the code
+    tries ARP first; it says `host` and `name` are on every log line, but
+    the check and list paths lack them.
+
+## 7. Gaps shared by the existing providers
+
+- **Adoption is not verified.** None of the three checks `LabelUID` when
+  `Create` finds an existing instance with the right name.
+- **The Create comment does not match kvm.** The interface comment in
+  `internal/provider/provider.go` says `Create` returns as soon as the
+  request is submitted; kvm blocks until the domain has started.
+- **`Stopped` and `Terminated` are produced inconsistently.** kubernetes
+  never produces Stopped, kvm never produces Terminated, and the reconciler
+  treats them the same.
+- **Different `ListByTag` partial-failure policies.** gcp leaves failed
+  parts out, kvm fails the whole list. Both are safe.
+- **Stale `mock` comments** in `internal/provider/provider.go`,
+  `internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
+  (and so the generated CRD's `spec.provider` description).
+- **GC does not know about installations.** Two clusters sharing one project
+  or hypervisor delete each other's instances. This was a merge-blocker in
+  the 2026-08-10 PR review
+  ([docs/reviews](reviews/2026-08-10-1134-proxy-operator-pr-review.md)) and
+  is still open.
+- **Tracing.** kvm has no child spans, so a kvm trace shows `provider.create`
+  and nothing under it; correlation works only through `traceID` on its V(1)
+  log lines. Only kubernetes is tested against a real collector, and that
+  test checks only the operation spans; nothing checks that gcp's SDK spans
+  or the kubernetes API spans nest under `provider.*`.
+
+## 8. Proposed features
+
+Listed only. Nothing below is designed or implemented.
+
+- **Child spans in kvm:** extend the `tracingAPI` decorator in
+  `kvm/trace.go`, which already wraps every libvirt call, to also start a
+  span per RPC, and add spans for the SSH dial and the image download. The
+  seam's methods take no `ctx` today (`hostPool.call` hands `fn` only the
+  API), so the span context has to be plumbed in.
+- **Tempo e2e checks for child spans**, and for gcp and kvm through their
+  real backend runbooks.
+- **A shared conformance suite:** for example `internal/provider/providertest`,
+  which runs the contract §11 tests against any provider plus a fake
+  backend, so a new provider gets the contract tests for free.
+- **Verify the UID when `Create` adopts an instance.** Return `ErrPermanent`
+  on a mismatch instead of adopting an instance that belongs to someone else.
+- **Report asynchronous create failures:** gcp keeps the operation name, and
+  `Get` reports the failure instead of NotFound.
+- **Capacity or quota signals** for kvm, such as pool free space and host
+  memory, mapped to `ErrQuotaExceeded`.
+- **GC for secondary resources:** mark volumes and similar resources, and
+  let `ListByTag` or a second listing method cover them.
+- **Reload credentials and config without a restart.**
+- **Checker `Details`** for kubernetes (server version) and gcp (quota
+  headroom).
+- **Internal-IP mode** for gcp, and a firewall rule managed per provider.
+- **An installation-identity mark**, set on create and filtered on in
+  `ListByTag`, so installations sharing a backend do not collect each
+  other's instances.
+- **A permanent-failure latch that sees provider availability**, so adding
+  a missing provider releases the Proxies waiting on it.
+- **Validate provider names at load**, and derive the config type
+  allow-list from the constructor map.
+- **Per-provider RBAC** as an optional kustomize component, and an
+  `OwnerReference` on kubernetes Pods.

docs/providers.md

@@ -0,0 +1,39 @@
+# Providers
+
+The provider documentation is split into three files so that a new provider
+can be specified and built from the generic rules alone, without reading how
+the existing ones happen to work.
+
+- **[provider-contract.md](provider-contract.md)**: what every provider MUST,
+  SHOULD and MAY do, which caller depends on each rule, the behaviour profiles
+  where providers are allowed to differ, the error taxonomy, credential
+  hygiene, testing obligations, the checklist for adding a provider, and the
+  contract's known limits. It never names an existing provider.
+- **[provider-spec-template.md](provider-spec-template.md)**: the questionnaire
+  a new provider's design answers, one entry per decision the contract leaves
+  open. Copy it to `docs/<type>-provider-spec.md`, fill it in, and hand it to
+  the implementer with the other two files.
+- **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp and
+  kvm meet the contract, a table mapping each of them to every profile, their
+  per-provider reference notes, shared gaps and proposed features.
+
+Per-provider user guides: [kvm-provider.md](kvm-provider.md),
+[gcp-vm-validation.md](gcp-vm-validation.md) and
+[gcp-in-specific-project.md](gcp-in-specific-project.md).
+
+Related sections of [architecture.md](architecture.md): the components table,
+[§5 what provider calls do](architecture.md#5-what-provider-calls-do-in-the-outside-world),
+[§10 tracing](architecture.md#10-tracing-internaltracing),
+[§11 provider health checks](architecture.md#11-provider-health-checks-internalproviderhealth)
+and the "Decisions" list. The test layout is in [testing.md](testing.md).
+
+## Reading order for a new provider
+
+1. The contract, end to end.
+2. The template, filled in for the new backend; every unanswered entry is a
+   design decision still to make.
+3. The appendix, for a worked example of each profile the spec chose.
+4. The user guide of the existing provider closest to the new one.
+
+The single-file blueprint that preceded this split is in the git history of
+this path.

docs/testing.md

@@ -53,18 +53,23 @@ manual runs too.
 | | `config_test.go` | providers-config parsing + fail-fast validation | pure |
 | | `metrics_test.go` | `WithMetrics` decorator: classified result labels, error passthrough | fake recorder + static provider |
 | `internal/provider/kubernetes` | `kubernetes_test.go`, `pod_test.go` | Create/Get/Delete/ListByTag over real Pod objects; pure `buildPod`/`squidConf` incl. the `max_filedescriptors` OOM-regression assertion | `client/fake` (real `corev1.Pod`s, no kubelet) |
+| | `check_test.go` | health check: a single cluster-wide list with `Limit=1` and the managed-label selector; the reason for every API error class | `client/fake` + interceptor |
 | `internal/provider/gcp` | `insert_test.go`, `errors_test.go`, `gcp_test.go` | field-by-field `buildInsertRequest`; HTTP-code classification table; zone-qualified IDs, 409-is-success, 9-row state mapping, ListByTag filter + `ReturnPartialSuccess` | pure builder needs **no fake**; the rest uses the flattened `instancesAPI` seam |
+| | `check_test.go` | health check: request shape and the reason per HTTP code or network error; the **real SDK client** makes exactly one request with `maxResults=1`, even when a next page exists | fake seam + `httptest` |
 | `internal/provider/kvm` | `domainxml_test.go`, `volume_test.go`, `seed_test.go`, `id_test.go`, `metadata_test.go` | pure builders: domain/volume XML unmarshalled back and checked field by field, seed ISO reopened with the same library, providerID round-trip, metadata in both namespace spellings | pure |
 | | `errors_test.go`, `conn_test.go` | libvirt-code classification table; connection pool (reuse, redial, drop on transport error, cancel releases a blocked RPC, one dial per host under concurrency); SSH dial, host-key policy, keepalive | fake dialer; a **real in-process `x/crypto/ssh` server** forwarding a unix socket |
 | | `image_test.go`, `kvm_test.go` | image cache (fresh fetch, ready, marker missing, checksum change/mismatch, 4xx/5xx, concurrent callers); Create resume after every partial step, leftover truncated seed, state/IP mapping, every Delete already-gone case, ListByTag across hosts / one down / unreadable domain; logging tiers (V0 silent, V1 one line per RPC, V2 XML) with cloud-init never logged | `httptest` image server + in-memory `fakeAPI` behind the `libvirtAPI` seam |
 | | `trace_test.go`, `conn_test.go`, `errors_test.go` | every `libvirtAPI` method traced exactly once (guarded by reflection on the interface), results and errors pass through, libvirt error fields, detail keys only at V2, upload content never logged; pool connect/drop lines, no wrapping below V1 | `fakeAPI` + `funcr` capture |
-| | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards | a **real hypervisor** — see below |
+| | `check_test.go`, `conn_test.go` | health check: details, per-host verdicts in config order, hosts checked concurrently, a timeout that keeps the connection and refuses a second check while the first still waits, no waiting behind a reconcile's dial, the reason table; `probe` abandons without closing, but still drops on a transport error; waiting for a host's slot ends with the caller's deadline; an inactive or missing pool and daemon errors keep the connection; every dial stage tagged through the real SSH server, with classification unchanged | `fakeAPI`, fake dialers, in-process SSH server |
+| | `kvm_integration_test.go` (`-tags=integration`) | one real VM: image fetch, Create, idempotent re-Create, Running with IP, ListByTag, Delete, pool state afterwards; `TestKVMCheck_realHost` runs the health check against the host and creates nothing | a **real hypervisor** — see below |
 | `internal/health` | `probe_test.go` | probes through a **real CONNECT-capable proxy stub** to a real TLS server: tunnel success, refused CONNECT, wrong status, dead proxy, plain-http forward | `httptest` + hijacked bidirectional tunnel |
 | | `engine_test.go` | thresholds, first-verdict, latency suppression matrix, dropped-event retry, stale-probe UID guard, tick scheduling/seeding/pruning, `Start` end-to-end | fake `client.Reader` + injected `probeFn` |
 | `internal/lease` | `store_test.go` | capacity, `MaxLeases=0`, all three selection tie-breaks, target-scoped vs global cooldowns, expiry via fake clock, report-on-retained-lease, idempotent release, **40 concurrent acquires never exceeding capacity** | injectable clock; the concurrency case is why `-race` matters |
 | `internal/discovery` | `server_test.go` | auth matrix (incl. `/healthz` exemption), list filtering, grant shape, full 409 arithmetic, invalid TTL/body/result, idempotent release, report→cooldown→409 round trip, `Start`/shutdown | `httptest` over the real handler chain, fake cache reader, **real** `lease.Store` |
 | `internal/gc` | `gc_test.go` | the true-orphan matrix (live/deleting/young/unlabelled all kept), broken-provider isolation, list-failure skips sweep, namespace guard, sweep loop | fake reader + canned-list provider |
-| `internal/metrics` | `metrics_test.go` | scrape-time collectors (`GatherAndCompare`), per-proxy series cleanup via `ForgetProxy`, label counts, fresh-registry-per-test property | `prometheus/client_golang/testutil` |
+| `internal/providerhealth` | `status_test.go`, `monitor_test.go` | `buildStatus` (the Ready condition, per-target transition times, truncation); objects created with the label and patched, the round timeout, no-checker Unknown, invalid names, transition-only logging, cleanup of labelled unconfigured objects, the checks-off path, `ValidateSchedule` | fake client with a cluster-scoped RESTMapper + `funcr` capture |
+| | `envtest_test.go` | the generated `ProviderHealth` schema against a real API server: cluster scope, status subresource, required fields, list-map keys, the 1024-character message limit | envtest (skipped under `-short`) |
+| `internal/metrics` | `metrics_test.go` | scrape-time collectors (`GatherAndCompare`) incl. the provider-target gauges, per-proxy series cleanup via `ForgetProxy`, label counts, fresh-registry-per-test property | `prometheus/client_golang/testutil` |
 
 Conventions (from `CLAUDE.md`): stdlib `testing`, table-driven by default,
 `t.Parallel()` where safe, tests next to source, ginkgo only in the
@@ -89,7 +94,7 @@ envtest suite the scaffold generated.
 - **Live GCP**: no test talks to Google. The seam boundary
   (`buildInsertRequest` + classification) is tested exhaustively instead.
 - **Live libvirt, by default**: the KVM unit tests run over an in-memory
-  fake of the 17 libvirt calls the provider uses. Whether libvirt accepts
+  fake of the 20 libvirt calls the provider uses. Whether libvirt accepts
   the domain XML, whether a raw qcow2 upload plus pool refresh yields a
   usable backing image, and whether the guest actually boots and reports
   an address are provable only on a real host — that is the opt-in
@@ -109,13 +114,19 @@ had.
 export KVM_TEST_HOST=nas1.home.lan:22          # SSH endpoint
 export KVM_TEST_USER=virt                      # must be in the libvirt group
 export KVM_TEST_SSH_KEY=$HOME/.ssh/id_ed25519  # private key for that user
+export KVM_TEST_KNOWN_HOSTS=$HOME/.ssh/kvm_known_hosts  # the hypervisors' host keys
 export KVM_TEST_BRIDGE=br0                     # bridge to a network with DHCP
 export KVM_TEST_IMAGE_URL=https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
 make test-integration-kvm
 ```
 
-Optional: `KVM_TEST_KNOWN_HOSTS` (without it host keys are **not**
-verified — the log says so), `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL`
+`KVM_TEST_KNOWN_HOSTS` is technically optional, but leaving it out turns
+host-key verification **off** for the run (the log says so); keep a
+`kvm_known_hosts` holding just the hypervisors rather than pointing the
+test at your everyday `known_hosts`, so a re-keyed hypervisor is a
+one-line fix in a file nothing else uses.
+
+Also optional: `KVM_TEST_IMAGE_SHA256`, `KVM_TEST_POOL`
 (default `default`), `KVM_TEST_SOCKET`, `KVM_TEST_TIMEOUT` (default
 `20m`, must cover the first-time image download), `KVM_TEST_KEEP`
 to leave the VM behind for `virsh console` when something goes wrong,
@@ -139,13 +150,24 @@ but no per-VM overlay or seed. It also tries a TCP connect to port 22 on
 the guest, but only logs the outcome — the machine running the test may
 not route to the bridge network.
 
+`TestKVMCheck_realHost` runs the provider health check
+([kvm-provider.md](kvm-provider.md#health-check)) against the same host.
+It must find the host reachable and the pool running, and it creates
+nothing. It needs the same variables; to run only this test:
+
+```sh
+go test -tags=integration -count=1 -v -run TestKVMCheck ./internal/provider/kvm/
+```
+
 The test is not part of `make test` and not run in CI. Keep it compiling
 with `go vet -tags=integration ./...` when the provider's internals move.
 
 ## The scaffolded `make test-e2e` suite
 
 `test/e2e/` is the kubebuilder-generated smoke suite (build image → kind
-cluster → deploy → assert the manager pod runs and serves metrics). It
+cluster → deploy → assert the manager pod runs, publishes a Ready
+`ProviderHealth` for the `kubernetes` provider, and serves metrics that
+include that provider's `proxy_operator_provider_target_up` series). It
 compiles only under `-tags=e2e`, manages its own kind cluster
 (`make test-e2e` / `make cleanup-test-e2e`), and has been kept compiling
 (`go vet -tags=e2e ./...` is part of the routine) but is **not part of

internal/metrics/metrics.go

@@ -117,3 +117,51 @@ func (c *constCollector) Collect(ch chan<- prometheus.Metric) {
 		ch <- prometheus.MustNewConstMetric(c.desc, prometheus.GaugeValue, float64(value), label)
 	}
 }
+
+// ProviderTarget is one provider health-check target as the gauges see it.
+type ProviderTarget struct {
+	Provider, Target string
+	Up               bool
+	Duration         time.Duration
+	CheckedAt        time.Time
+}
+
+// RegisterProviderTargets registers the provider-check gauges. Like the
+// other scrape-time collectors they are read from the source at every
+// scrape, so a target that goes away stops being reported rather than
+// leaving a stale series behind.
+func RegisterProviderTargets(reg prometheus.Registerer, read func() []ProviderTarget) error {
+	labels := []string{"provider", "target"}
+	return reg.Register(&providerTargetCollector{
+		up: prometheus.NewDesc("proxy_operator_provider_target_up",
+			"Whether the target passed its latest provider health check (1) or not (0).", labels, nil),
+		duration: prometheus.NewDesc("proxy_operator_provider_check_duration_seconds",
+			"Duration of the target's latest provider health check.", labels, nil),
+		checked: prometheus.NewDesc("proxy_operator_provider_check_timestamp_seconds",
+			"Unix time of the target's latest provider health check.", labels, nil),
+		read: read,
+	})
+}
+
+type providerTargetCollector struct {
+	up, duration, checked *prometheus.Desc
+	read                  func() []ProviderTarget
+}
+
+func (c *providerTargetCollector) Describe(ch chan<- *prometheus.Desc) {
+	ch <- c.up
+	ch <- c.duration
+	ch <- c.checked
+}
+
+func (c *providerTargetCollector) Collect(ch chan<- prometheus.Metric) {
+	for _, t := range c.read() {
+		up := 0.0
+		if t.Up {
+			up = 1
+		}
+		ch <- prometheus.MustNewConstMetric(c.up, prometheus.GaugeValue, up, t.Provider, t.Target)
+		ch <- prometheus.MustNewConstMetric(c.duration, prometheus.GaugeValue, t.Duration.Seconds(), t.Provider, t.Target)
+		ch <- prometheus.MustNewConstMetric(c.checked, prometheus.GaugeValue, float64(t.CheckedAt.UnixNano())/1e9, t.Provider, t.Target)
+	}
+}

internal/metrics/metrics_test.go

@@ -111,3 +111,45 @@ func TestRegister_freshRegistryPerTest(t *testing.T) {
 	register(t, m1, nil, 0)
 	register(t, m2, nil, 0)
 }
+
+func TestRegisterProviderTargets_scrape(t *testing.T) {
+	t.Parallel()
+	checked := time.Unix(1757602800, 0)
+	targets := []ProviderTarget{
+		{Provider: "kvm-home", Target: "nas1", Up: true, Duration: 84 * time.Millisecond, CheckedAt: checked},
+		{Provider: "kvm-home", Target: "nas2", Up: false, Duration: 15 * time.Second, CheckedAt: checked},
+	}
+	reg := prometheus.NewRegistry()
+	if err := RegisterProviderTargets(reg, func() []ProviderTarget { return targets }); err != nil {
+		t.Fatalf("RegisterProviderTargets: %v", err)
+	}
+
+	expected := `
+# HELP proxy_operator_provider_check_duration_seconds Duration of the target's latest provider health check.
+# TYPE proxy_operator_provider_check_duration_seconds gauge
+proxy_operator_provider_check_duration_seconds{provider="kvm-home",target="nas1"} 0.084
+proxy_operator_provider_check_duration_seconds{provider="kvm-home",target="nas2"} 15
+# HELP proxy_operator_provider_check_timestamp_seconds Unix time of the target's latest provider health check.
+# TYPE proxy_operator_provider_check_timestamp_seconds gauge
+proxy_operator_provider_check_timestamp_seconds{provider="kvm-home",target="nas1"} 1.7576028e+09
+proxy_operator_provider_check_timestamp_seconds{provider="kvm-home",target="nas2"} 1.7576028e+09
+# HELP proxy_operator_provider_target_up Whether the target passed its latest provider health check (1) or not (0).
+# TYPE proxy_operator_provider_target_up gauge
+proxy_operator_provider_target_up{provider="kvm-home",target="nas1"} 1
+proxy_operator_provider_target_up{provider="kvm-home",target="nas2"} 0
+`
+	if err := testutil.GatherAndCompare(reg, strings.NewReader(expected)); err != nil {
+		t.Error(err)
+	}
+}
+
+func TestRegisterProviderTargets_emptySnapshotHasNoSeries(t *testing.T) {
+	t.Parallel()
+	reg := prometheus.NewRegistry()
+	if err := RegisterProviderTargets(reg, func() []ProviderTarget { return nil }); err != nil {
+		t.Fatalf("RegisterProviderTargets: %v", err)
+	}
+	if n, err := testutil.GatherAndCount(reg); err != nil || n != 0 {
+		t.Errorf("series before the first check = %d (%v), want 0", n, err)
+	}
+}

internal/provider/check.go

@@ -0,0 +1,46 @@
+package provider
+
+import (
+	"context"
+	"time"
+)
+
+// Reasons shared by every Checker, so a listing reads the same across
+// providers. Providers add their own for failures only they can tell
+// apart (KVM's SSHHandshakeFailed or PoolNotActive, say).
+const (
+	ReasonReachable    = "Reachable"
+	ReasonTimeout      = "Timeout"
+	ReasonUnreachable  = "Unreachable"
+	ReasonUnauthorized = "Unauthorized"
+	ReasonForbidden    = "Forbidden"
+	ReasonAPIError     = "APIError"
+)
+
+// TargetStatus is one target's verdict from a Checker.
+type TargetStatus struct {
+	// Name identifies the target within its provider: a KVM host name,
+	// "apiserver", a GCP project.
+	Name string
+	// Address is what the check connected to, for humans.
+	Address string
+	Ready   bool
+	// Reason is ReasonReachable when Ready, otherwise why not.
+	Reason string
+	// Message is human-readable detail, usually the error.
+	Message string
+	Latency time.Duration
+	// Details are provider-specific facts gathered by the check.
+	Details map[string]string
+}
+
+// Checker is the optional interface of a provider that can verify its
+// backend is reachable and usable without touching any instance. The
+// provider health monitor type-asserts for it on the unwrapped providers,
+// so it stays out of the Provider contract.
+type Checker interface {
+	// Check returns one verdict per target, in a stable order. It never
+	// fails as a whole — a failure is a verdict — and must return promptly
+	// once ctx ends.
+	Check(ctx context.Context) []TargetStatus
+}

internal/provider/gcp/check.go

@@ -0,0 +1,78 @@
+package gcp
+
+import (
+	"context"
+	"errors"
+	"net/http"
+	"time"
+
+	"cloud.google.com/go/compute/apiv1/computepb"
+	"google.golang.org/api/googleapi"
+	"google.golang.org/api/iterator"
+	"google.golang.org/protobuf/proto"
+
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
+)
+
+var _ provider.Checker = (*Provider)(nil)
+
+// computeAddress is the endpoint the SDK talks to, shown only to humans.
+const computeAddress = "compute.googleapis.com"
+
+// managedFilter selects the instances this operator created.
+const managedFilter = "labels." + provider.LabelManaged + " = " + provider.LabelManagedYes
+
+// Check asks the project for one page holding at most one managed
+// instance. That exercises the credentials, the Compute API and the
+// compute.instances.list permission orphan GC depends on, in a single
+// request whatever the project's size. The target is the project: the
+// provider config names no zones.
+func (p *Provider) Check(ctx context.Context) []provider.TargetStatus {
+	start := time.Now()
+	err := p.api.Probe(ctx, &computepb.AggregatedListInstancesRequest{
+		Project:              p.cfg.Project,
+		Filter:               proto.String(managedFilter),
+		MaxResults:           proto.Uint32(1),
+		ReturnPartialSuccess: proto.Bool(true),
+	})
+	ts := provider.TargetStatus{Name: p.cfg.Project, Address: computeAddress, Latency: time.Since(start)}
+	if err != nil {
+		logAPIError(p.logger(ctx), "check", err)
+		ts.Reason = checkReason(ctx, err)
+		ts.Message = err.Error()
+		return []provider.TargetStatus{ts}
+	}
+	ts.Ready = true
+	ts.Reason = provider.ReasonReachable
+	return []provider.TargetStatus{ts}
+}
+
+// Probe fetches the first page of an aggregated list and discards it. The
+// SDK treats MaxResults as the page size, so AggregatedList, which drains
+// the iterator, would still walk every page of the project.
+func (r *realInstances) Probe(ctx context.Context, req *computepb.AggregatedListInstancesRequest) error {
+	_, err := r.client.AggregatedList(ctx, req).Next()
+	if err == iterator.Done {
+		return nil
+	}
+	return err
+}
+
+// checkReason maps a failed probe to a reason. A token that cannot be
+// obtained never reaches the API, so it has no status code and lands on
+// Unreachable, with the detail in the message.
+func checkReason(ctx context.Context, err error) string {
+	var gerr *googleapi.Error
+	switch {
+	case ctx.Err() != nil, errors.Is(err, context.DeadlineExceeded):
+		return provider.ReasonTimeout
+	case !errors.As(err, &gerr):
+		return provider.ReasonUnreachable
+	case gerr.Code == http.StatusUnauthorized:
+		return provider.ReasonUnauthorized
+	case gerr.Code == http.StatusForbidden:
+		return provider.ReasonForbidden
+	default:
+		return provider.ReasonAPIError
+	}
+}

internal/provider/gcp/check_test.go

@@ -0,0 +1,165 @@
+package gcp
+
+import (
+	"context"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"sync"
+	"syscall"
+	"testing"
+
+	compute "cloud.google.com/go/compute/apiv1"
+	"cloud.google.com/go/compute/apiv1/computepb"
+	"google.golang.org/api/googleapi"
+	"google.golang.org/api/option"
+	"google.golang.org/protobuf/proto"
+
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
+)
+
+func TestCheck_reachable(t *testing.T) {
+	t.Parallel()
+	api := &fakeAPI{}
+	p := newTestProvider(api)
+
+	got := p.Check(context.Background())
+
+	if len(got) != 1 {
+		t.Fatalf("Check() returned %d targets, want 1", len(got))
+	}
+	ts := got[0]
+	if !ts.Ready || ts.Reason != provider.ReasonReachable || ts.Message != "" {
+		t.Errorf("Check() = %+v, want ready and Reachable", ts)
+	}
+	if ts.Name != "my-project" || ts.Address != "compute.googleapis.com" {
+		t.Errorf("target = %q at %q, want my-project at compute.googleapis.com", ts.Name, ts.Address)
+	}
+	want := &computepb.AggregatedListInstancesRequest{
+		Project:              "my-project",
+		Filter:               proto.String("labels.proxy-operator-managed = true"),
+		MaxResults:           proto.Uint32(1),
+		ReturnPartialSuccess: proto.Bool(true),
+	}
+	if !proto.Equal(api.probeReq, want) {
+		t.Errorf("probe request = %v, want %v", api.probeReq, want)
+	}
+}
+
+func TestCheck_failureReasons(t *testing.T) {
+	t.Parallel()
+	ended, cancel := context.WithCancel(context.Background())
+	cancel()
+
+	tests := []struct {
+		name string
+		ctx  context.Context
+		err  error
+		want string
+	}{
+		{"unauthorized", context.Background(), gerr(401), provider.ReasonUnauthorized},
+		{"forbidden", context.Background(), gerr(403, "forbidden"), provider.ReasonForbidden},
+		{"api disabled", context.Background(), gerr(403, "accessNotConfigured"), provider.ReasonForbidden},
+		{"server error", context.Background(), gerr(500), provider.ReasonAPIError},
+		{"deadline", context.Background(), &url.Error{
+			Op: "Get", URL: "https://compute.googleapis.com/", Err: context.DeadlineExceeded,
+		}, provider.ReasonTimeout},
+		{"context ended", ended, context.Canceled, provider.ReasonTimeout},
+		{"connection refused", context.Background(), &url.Error{
+			Op: "Get", URL: "https://compute.googleapis.com/", Err: syscall.ECONNREFUSED,
+		}, provider.ReasonUnreachable},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			p := newTestProvider(&fakeAPI{probeErr: tc.err})
+
+			got := p.Check(tc.ctx)
+
+			if len(got) != 1 {
+				t.Fatalf("Check() returned %d targets, want 1", len(got))
+			}
+			if got[0].Ready || got[0].Reason != tc.want || got[0].Message != tc.err.Error() {
+				t.Errorf("Check() = %+v, want not ready, reason %s, message %q", got[0], tc.want, tc.err.Error())
+			}
+		})
+	}
+}
+
+// TestRealInstances_Probe runs the real SDK client against a local server,
+// since the point of Probe is how many requests the SDK makes.
+func TestRealInstances_Probe(t *testing.T) {
+	t.Parallel()
+	tests := []struct {
+		name     string
+		status   int
+		body     string
+		wantCode int
+	}{
+		{
+			name:   "fetches only the first page",
+			status: http.StatusOK,
+			body:   `{"items":{"zones/europe-west1-b":{"instances":[{"name":"proxy-a"}]}},"nextPageToken":"page-2"}`,
+		},
+		{
+			name:   "empty project",
+			status: http.StatusOK,
+			body:   `{}`,
+		},
+		{
+			name:     "api error keeps its status",
+			status:   http.StatusForbidden,
+			body:     `{"error":{"code":403,"message":"Compute Engine API has not been used","errors":[{"reason":"accessNotConfigured"}]}}`,
+			wantCode: http.StatusForbidden,
+		},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			var (
+				mu   sync.Mutex
+				reqs []*url.URL
+			)
+			srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+				mu.Lock()
+				reqs = append(reqs, r.URL)
+				mu.Unlock()
+				w.Header().Set("Content-Type", "application/json")
+				w.WriteHeader(tc.status)
+				_, _ = w.Write([]byte(tc.body))
+			}))
+			t.Cleanup(srv.Close)
+			ctx := context.Background()
+			c, err := compute.NewInstancesRESTClient(ctx, option.WithEndpoint(srv.URL), option.WithoutAuthentication())
+			if err != nil {
+				t.Fatalf("NewInstancesRESTClient: %v", err)
+			}
+			t.Cleanup(func() { _ = c.Close() })
+
+			err = (&realInstances{client: c}).Probe(ctx, &computepb.AggregatedListInstancesRequest{
+				Project:    "my-project",
+				MaxResults: proto.Uint32(1),
+			})
+
+			var gerr *googleapi.Error
+			switch {
+			case tc.wantCode == 0 && err != nil:
+				t.Fatalf("Probe() = %v, want nil", err)
+			case tc.wantCode != 0 && (!errors.As(err, &gerr) || gerr.Code != tc.wantCode):
+				t.Fatalf("Probe() = %v, want a googleapi error with code %d", err, tc.wantCode)
+			}
+			mu.Lock()
+			defer mu.Unlock()
+			if len(reqs) != 1 {
+				t.Fatalf("server saw %d requests, want 1", len(reqs))
+			}
+			if got := reqs[0].Query().Get("maxResults"); got != "1" {
+				t.Errorf("maxResults = %q, want 1", got)
+			}
+			if want := "/compute/v1/projects/my-project/aggregated/instances"; reqs[0].Path != want {
+				t.Errorf("path = %q, want %q", reqs[0].Path, want)
+			}
+		})
+	}
+}

internal/provider/gcp/gcp.go

@@ -33,6 +33,8 @@ type instancesAPI interface {
 	Get(ctx context.Context, req *computepb.GetInstanceRequest) (*computepb.Instance, error)
 	Delete(ctx context.Context, req *computepb.DeleteInstanceRequest) (opName string, err error)
 	AggregatedList(ctx context.Context, req *computepb.AggregatedListInstancesRequest) ([]*computepb.Instance, error)
+	// Probe fetches only the first page of an aggregated list; see Check.
+	Probe(ctx context.Context, req *computepb.AggregatedListInstancesRequest) error
 }
 
 // realInstances adapts *compute.InstancesClient to the seam.
@@ -221,17 +223,16 @@ func (p *Provider) Delete(ctx context.Context, providerID string) error {
 // entire GC sweep.
 func (p *Provider) ListByTag(ctx context.Context) ([]provider.Instance, error) {
 	log := p.logger(ctx)
-	filter := fmt.Sprintf("labels.%s = %s", provider.LabelManaged, provider.LabelManagedYes)
 	instances, err := p.api.AggregatedList(ctx, &computepb.AggregatedListInstancesRequest{
 		Project:              p.cfg.Project,
-		Filter:               proto.String(filter),
+		Filter:               proto.String(managedFilter),
 		ReturnPartialSuccess: proto.Bool(true),
 	})
 	if err != nil {
 		logAPIError(log, "list", err)
 		return nil, p.wrapErr("list", "", err)
 	}
-	log.V(1).Info("GCP instances listed", "filter", filter, "count", len(instances))
+	log.V(1).Info("GCP instances listed", "filter", managedFilter, "count", len(instances))
 	out := make([]provider.Instance, 0, len(instances))
 	for _, inst := range instances {
 		conv := toInstance(inst, lastPathSegment(inst.GetZone()))

internal/provider/gcp/gcp_test.go

@@ -31,6 +31,14 @@ type fakeAPI struct {
 	listReq   *computepb.AggregatedListInstancesRequest
 	listInsts []*computepb.Instance
 	listErr   error
+
+	probeReq *computepb.AggregatedListInstancesRequest
+	probeErr error
+}
+
+func (f *fakeAPI) Probe(_ context.Context, req *computepb.AggregatedListInstancesRequest) error {
+	f.probeReq = req
+	return f.probeErr
 }
 
 func (f *fakeAPI) Insert(_ context.Context, req *computepb.InsertInstanceRequest) (string, error) {

internal/provider/kubernetes/check.go

@@ -0,0 +1,53 @@
+package kubernetes
+
+import (
+	"context"
+	"errors"
+	"time"
+
+	corev1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
+)
+
+var _ provider.Checker = (*Provider)(nil)
+
+// Check lists at most one managed Pod across the cluster: the same client,
+// verb and scope ListByTag and orphan GC rely on, so a pass means the API
+// server answers and the operator's RBAC still covers them. It does not
+// prove create or delete rights.
+func (p *Provider) Check(ctx context.Context) []provider.TargetStatus {
+	start := time.Now()
+	var pods corev1.PodList
+	err := p.client.List(ctx, &pods, client.MatchingLabels{
+		provider.LabelManaged: provider.LabelManagedYes,
+	}, client.Limit(1))
+	ts := provider.TargetStatus{Name: "apiserver", Address: p.apiHost, Latency: time.Since(start)}
+	if err != nil {
+		ts.Reason = checkReason(ctx, err)
+		ts.Message = err.Error()
+		return []provider.TargetStatus{ts}
+	}
+	ts.Ready = true
+	ts.Reason = provider.ReasonReachable
+	return []provider.TargetStatus{ts}
+}
+
+func checkReason(ctx context.Context, err error) string {
+	var status apierrors.APIStatus
+	switch {
+	case ctx.Err() != nil, errors.Is(err, context.DeadlineExceeded),
+		apierrors.IsTimeout(err), apierrors.IsServerTimeout(err):
+		return provider.ReasonTimeout
+	case apierrors.IsUnauthorized(err):
+		return provider.ReasonUnauthorized
+	case apierrors.IsForbidden(err):
+		return provider.ReasonForbidden
+	case errors.As(err, &status):
+		return provider.ReasonAPIError
+	default:
+		return provider.ReasonUnreachable
+	}
+}

internal/provider/kubernetes/check_test.go

@@ -0,0 +1,114 @@
+package kubernetes
+
+import (
+	"context"
+	"errors"
+	"net/url"
+	"syscall"
+	"testing"
+
+	corev1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
+
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
+)
+
+type listFunc func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error
+
+func newCheckProvider(list listFunc) *Provider {
+	scheme := runtime.NewScheme()
+	if err := corev1.AddToScheme(scheme); err != nil {
+		panic(err)
+	}
+	c := fake.NewClientBuilder().WithScheme(scheme).WithInterceptorFuncs(interceptor.Funcs{List: list}).Build()
+	p := newWithClient(c, provider.ProviderConfig{Name: "kubernetes"})
+	p.apiHost = "https://10.96.0.1:443"
+	return p
+}
+
+func TestCheck_reachable(t *testing.T) {
+	t.Parallel()
+	p := newTestProvider(testPod("ns", "a", "uid-a", corev1.PodRunning, "10.0.0.1"))
+	p.apiHost = "https://10.96.0.1:443"
+
+	got := p.Check(context.Background())
+
+	if len(got) != 1 {
+		t.Fatalf("Check() returned %d targets, want 1", len(got))
+	}
+	ts := got[0]
+	if !ts.Ready || ts.Reason != provider.ReasonReachable || ts.Message != "" {
+		t.Errorf("Check() = %+v, want ready and Reachable", ts)
+	}
+	if ts.Name != "apiserver" || ts.Address != "https://10.96.0.1:443" {
+		t.Errorf("target = %q at %q, want apiserver at https://10.96.0.1:443", ts.Name, ts.Address)
+	}
+}
+
+func TestCheck_listOptions(t *testing.T) {
+	t.Parallel()
+	var got client.ListOptions
+	p := newCheckProvider(func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error {
+		got.ApplyOptions(opts)
+		return c.List(ctx, list, opts...)
+	})
+
+	p.Check(context.Background())
+
+	if got.Limit != 1 {
+		t.Errorf("Limit = %d, want 1", got.Limit)
+	}
+	if got.Namespace != "" {
+		t.Errorf("Namespace = %q, want cluster-wide", got.Namespace)
+	}
+	want := provider.LabelManaged + "=" + provider.LabelManagedYes
+	if got.LabelSelector == nil || got.LabelSelector.String() != want {
+		t.Errorf("LabelSelector = %v, want %s", got.LabelSelector, want)
+	}
+}
+
+func TestCheck_failureReasons(t *testing.T) {
+	t.Parallel()
+	pods := schema.GroupResource{Resource: "pods"}
+	ended, cancel := context.WithCancel(context.Background())
+	cancel()
+
+	tests := []struct {
+		name string
+		ctx  context.Context
+		err  error
+		want string
+	}{
+		{"forbidden", context.Background(), apierrors.NewForbidden(pods, "", errors.New("rbac denied")), provider.ReasonForbidden},
+		{"unauthorized", context.Background(), apierrors.NewUnauthorized("token expired"), provider.ReasonUnauthorized},
+		{"api timeout", context.Background(), apierrors.NewTimeoutError("slow etcd", 1), provider.ReasonTimeout},
+		{"server timeout", context.Background(), apierrors.NewServerTimeout(pods, "list", 1), provider.ReasonTimeout},
+		{"context ended", ended, context.Canceled, provider.ReasonTimeout},
+		{"internal error", context.Background(), apierrors.NewInternalError(errors.New("etcd down")), provider.ReasonAPIError},
+		{"connection refused", context.Background(), &url.Error{
+			Op: "Get", URL: "https://10.96.0.1:443/api/v1/pods", Err: syscall.ECONNREFUSED,
+		}, provider.ReasonUnreachable},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			p := newCheckProvider(func(context.Context, client.WithWatch, client.ObjectList, ...client.ListOption) error {
+				return tc.err
+			})
+
+			got := p.Check(tc.ctx)
+
+			if len(got) != 1 {
+				t.Fatalf("Check() returned %d targets, want 1", len(got))
+			}
+			if got[0].Ready || got[0].Reason != tc.want || got[0].Message != tc.err.Error() {
+				t.Errorf("Check() = %+v, want not ready, reason %s, message %q", got[0], tc.want, tc.err.Error())
+			}
+		})
+	}
+}

internal/provider/kubernetes/kubernetes.go

@@ -36,6 +36,8 @@ const defaultImage = "ubuntu/squid:6.6-24.04_edge"
 type Provider struct {
 	client client.Client
 	image  string
+	// apiHost is the API server URL, shown by Check; empty in tests.
+	apiHost string
 }
 
 // New builds a kubernetes Provider from its config block. Satisfies
@@ -64,7 +66,9 @@ func NewWithTransportWrapper(_ context.Context, cfg provider.ProviderConfig, wra
 	if err != nil {
 		return nil, fmt.Errorf("kubernetes provider %q: building client: %w", cfg.Name, err)
 	}
-	return newWithClient(c, cfg), nil
+	p := newWithClient(c, cfg)
+	p.apiHost = restCfg.Host
+	return p, nil
 }
 
 // newWithClient builds a Provider around an already-constructed client,

internal/provider/kvm/check.go

@@ -0,0 +1,166 @@
+package kvm
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"sync"
+	"time"
+
+	libvirt "github.com/digitalocean/go-libvirt"
+
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
+)
+
+var _ provider.Checker = (*Provider)(nil)
+
+// Reasons only the KVM check can tell apart; the rest are provider-wide.
+const (
+	reasonSSHMaterialUnavailable = "SSHMaterialUnavailable"
+	reasonSSHHandshakeFailed     = "SSHHandshakeFailed"
+	reasonLibvirtConnectFailed   = "LibvirtConnectFailed"
+	reasonLibvirtError           = "LibvirtError"
+	reasonPoolNotFound           = "PoolNotFound"
+	reasonPoolNotActive          = "PoolNotActive"
+)
+
+// hostFacts is what a successful check learns about a host.
+type hostFacts struct {
+	version   uint64
+	hostname  string
+	poolState libvirt.StoragePoolState
+	capacity  uint64
+	available uint64
+}
+
+// Check verifies every configured host concurrently, in kvm.hosts order.
+// It goes through the same pooled connection Create and Get use, so the
+// SSH and libvirt handshakes only happen on the first check or after the
+// link dropped; then two cheap RPCs prove the daemon answers, and the
+// storage pool VMs are created in must exist and be running.
+func (p *Provider) Check(ctx context.Context) []provider.TargetStatus {
+	out := make([]provider.TargetStatus, len(p.cfg.Hosts))
+	var wg sync.WaitGroup
+	for i, h := range p.cfg.Hosts {
+		wg.Go(func() { out[i] = p.checkHost(ctx, h) })
+	}
+	wg.Wait()
+	return out
+}
+
+// checkHost goes through probe, so a check that runs out of time abandons
+// its RPCs rather than closing the connection the reconciles share. fn
+// returns the RPC errors untouched, since a transport error still drops the
+// link; a pool that exists but is stopped says nothing about the link, so
+// that verdict is reached afterwards. facts is read only after probe
+// returned nil, since an abandoned fn may still write it.
+func (p *Provider) checkHost(ctx context.Context, host provider.KVMHost) provider.TargetStatus {
+	ts := provider.TargetStatus{
+		Name:    host.Name,
+		Address: host.Address,
+		Details: map[string]string{"user": host.User, "pool": host.Pool},
+	}
+	start := time.Now()
+	var facts hostFacts
+	err := p.hosts.probe(ctx, p.logger(ctx), host, func(api libvirtAPI) error {
+		version, err := api.ConnectGetLibVersion()
+		if err != nil {
+			return err
+		}
+		hostname, err := api.ConnectGetHostname()
+		if err != nil {
+			return err
+		}
+		pool, err := api.StoragePoolLookupByName(host.Pool)
+		if err != nil {
+			return err
+		}
+		state, capacity, _, available, err := api.StoragePoolGetInfo(pool)
+		if err != nil {
+			return err
+		}
+		facts = hostFacts{
+			version:   version,
+			hostname:  hostname,
+			poolState: libvirt.StoragePoolState(state),
+			capacity:  capacity,
+			available: available,
+		}
+		return nil
+	})
+	ts.Latency = time.Since(start)
+	if err != nil {
+		ts.Reason = checkReason(err)
+		ts.Message = err.Error()
+		return ts
+	}
+	ts.Details["hostname"] = facts.hostname
+	ts.Details["libvirtVersion"] = formatLibVersion(facts.version)
+	ts.Details["poolState"] = poolStateName(facts.poolState)
+	ts.Details["poolCapacity"] = formatGiB(facts.capacity)
+	ts.Details["poolAvailable"] = formatGiB(facts.available)
+	if facts.poolState != libvirt.StoragePoolRunning {
+		ts.Reason = reasonPoolNotActive
+		ts.Message = fmt.Sprintf("storage pool %q is %s", host.Pool, poolStateName(facts.poolState))
+		return ts
+	}
+	ts.Ready = true
+	ts.Reason = provider.ReasonReachable
+	return ts
+}
+
+// checkReason names where a failed check broke. The dial stage wins over
+// a timeout: "the SSH handshake did not finish" says more than "the check
+// ran out of time", and the message still carries the deadline.
+func checkReason(err error) string {
+	var de *dialError
+	if errors.As(err, &de) {
+		switch de.stage {
+		case stageSSHMaterial:
+			return reasonSSHMaterialUnavailable
+		case stageSSHHandshake:
+			return reasonSSHHandshakeFailed
+		case stageLibvirtConnect:
+			return reasonLibvirtConnectFailed
+		default:
+			return provider.ReasonUnreachable
+		}
+	}
+	switch {
+	case errors.Is(err, context.DeadlineExceeded), errors.Is(err, context.Canceled), errors.Is(err, errProbeBusy):
+		return provider.ReasonTimeout
+	case hasCode(err, libvirt.ErrNoStoragePool):
+		return reasonPoolNotFound
+	case isLibvirtError(err):
+		return reasonLibvirtError
+	default:
+		return provider.ReasonUnreachable
+	}
+}
+
+// formatLibVersion decodes libvirt's major*1e6 + minor*1e3 + micro.
+func formatLibVersion(v uint64) string {
+	return fmt.Sprintf("%d.%d.%d", v/1_000_000, v/1_000%1_000, v%1_000)
+}
+
+// poolStateName uses virsh's words for a pool state.
+func poolStateName(s libvirt.StoragePoolState) string {
+	switch s {
+	case libvirt.StoragePoolInactive:
+		return "inactive"
+	case libvirt.StoragePoolBuilding:
+		return "building"
+	case libvirt.StoragePoolRunning:
+		return "running"
+	case libvirt.StoragePoolDegraded:
+		return "degraded"
+	case libvirt.StoragePoolInaccessible:
+		return "inaccessible"
+	default:
+		return fmt.Sprintf("state %d", s)
+	}
+}
+
+func formatGiB(bytes uint64) string {
+	return fmt.Sprintf("%.1fGiB", float64(bytes)/(1<<30))
+}

internal/provider/kvm/check_test.go

@@ -0,0 +1,319 @@
+package kvm
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"maps"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	libvirt "github.com/digitalocean/go-libvirt"
+
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
+)
+
+// withoutLatency zeroes the one field a test cannot predict.
+func withoutLatency(in []provider.TargetStatus) []provider.TargetStatus {
+	out := make([]provider.TargetStatus, len(in))
+	for i, ts := range in {
+		ts.Latency = 0
+		out[i] = ts
+	}
+	return out
+}
+
+func TestCheck_reachable(t *testing.T) {
+	t.Parallel()
+	api := newFakeAPI("default")
+	api.poolCapacity = 10 << 30
+	api.pools["default"]["disk.qcow2"] = &fakeVol{capacity: 4 << 30}
+	p := newTestProvider(hostFakes{"nas1": api}, "nas1")
+
+	got := withoutLatency(p.Check(context.Background()))
+
+	want := provider.TargetStatus{
+		Name:    "nas1",
+		Address: "nas1:22",
+		Ready:   true,
+		Reason:  provider.ReasonReachable,
+		Details: map[string]string{
+			"user":           "virt",
+			"pool":           "default",
+			"hostname":       "fake-host",
+			"libvirtVersion": "10.2.0",
+			"poolState":      "running",
+			"poolCapacity":   "10.0GiB",
+			"poolAvailable":  "6.0GiB",
+		},
+	}
+	if len(got) != 1 {
+		t.Fatalf("Check() returned %d targets, want 1", len(got))
+	}
+	if got[0].Name != want.Name || got[0].Address != want.Address || got[0].Ready != want.Ready ||
+		got[0].Reason != want.Reason || got[0].Message != "" || !maps.Equal(got[0].Details, want.Details) {
+		t.Errorf("Check() = %+v, want %+v", got[0], want)
+	}
+}
+
+func TestCheck_perHostVerdicts(t *testing.T) {
+	t.Parallel()
+	p := newTestProvider(hostFakes{"nas1": newFakeAPI("default"), "nas3": newFakeAPI("default")}, "nas1", "nas2", "nas3")
+
+	got := p.Check(context.Background())
+
+	if len(got) != 3 {
+		t.Fatalf("Check() returned %d targets, want 3", len(got))
+	}
+	for i, want := range []struct {
+		name  string
+		ready bool
+	}{{"nas1", true}, {"nas2", false}, {"nas3", true}} {
+		if got[i].Name != want.name || got[i].Ready != want.ready {
+			t.Errorf("target %d = %s ready=%v, want %s ready=%v", i, got[i].Name, got[i].Ready, want.name, want.ready)
+		}
+	}
+	if got[1].Reason != provider.ReasonUnreachable || !strings.Contains(got[1].Message, errHostOff.Error()) {
+		t.Errorf("unreachable host = %s %q, want Unreachable carrying %q", got[1].Reason, got[1].Message, errHostOff)
+	}
+	if got[1].Details["pool"] != "default" || got[1].Details["hostname"] != "" {
+		t.Errorf("unreachable host details = %v, want the configured pool and nothing learned", got[1].Details)
+	}
+}
+
+// TestCheck_libvirtVerdicts also checks twice per case: none of these
+// verdicts is evidence against the link, so the connection must survive.
+func TestCheck_libvirtVerdicts(t *testing.T) {
+	t.Parallel()
+	tests := []struct {
+		name        string
+		setup       func(api *fakeAPI)
+		pools       []string
+		wantReason  string
+		wantMessage string
+	}{
+		{
+			name:        "pool inactive",
+			pools:       []string{"default"},
+			setup:       func(api *fakeAPI) { api.poolStates["default"] = libvirt.StoragePoolInactive },
+			wantReason:  reasonPoolNotActive,
+			wantMessage: `storage pool "default" is inactive`,
+		},
+		{
+			name:        "pool missing",
+			pools:       []string{"other"},
+			wantReason:  reasonPoolNotFound,
+			wantMessage: "no storage pool with matching name 'default'",
+		},
+		{
+			name:        "daemon error",
+			pools:       []string{"default"},
+			setup:       func(api *fakeAPI) { api.fail["ConnectGetHostname"] = lvError(libvirt.ErrInternalError, "boom") },
+			wantReason:  reasonLibvirtError,
+			wantMessage: "boom",
+		},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			api := newFakeAPI(tc.pools...)
+			if tc.setup != nil {
+				tc.setup(api)
+			}
+			var dials atomic.Int32
+			dial := func(ctx context.Context, h provider.KVMHost) (*hostClient, error) {
+				dials.Add(1)
+				return hostFakes{"nas1": api}.dial(ctx, h)
+			}
+			pc := testConfig("nas1")
+			p := newWithDialer(pc.Name, *pc.KVM, dial, nil)
+
+			for range 2 {
+				got := p.Check(context.Background())
+				if got[0].Ready || got[0].Reason != tc.wantReason || !strings.Contains(got[0].Message, tc.wantMessage) {
+					t.Errorf("Check() = %+v, want not ready, %s, message containing %q", got[0], tc.wantReason, tc.wantMessage)
+				}
+			}
+			if n := dials.Load(); n != 1 {
+				t.Errorf("dialed %d times over two checks, want 1", n)
+			}
+		})
+	}
+}
+
+func TestCheck_hostsCheckedConcurrently(t *testing.T) {
+	t.Parallel()
+	nas2Dialing := make(chan struct{})
+	dial := func(ctx context.Context, h provider.KVMHost) (*hostClient, error) {
+		switch h.Name {
+		case "nas1":
+			select {
+			case <-nas2Dialing:
+			case <-ctx.Done():
+				return nil, ctx.Err()
+			}
+		case "nas2":
+			close(nas2Dialing)
+		}
+		return &hostClient{api: newFakeAPI("default"), close: func() {}}, nil
+	}
+	pc := testConfig("nas1", "nas2")
+	p := newWithDialer(pc.Name, *pc.KVM, dial, nil)
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+
+	got := p.Check(ctx)
+
+	for _, ts := range got {
+		if !ts.Ready {
+			t.Errorf("%s = %s %q, want ready: a sequential check would time out nas1", ts.Name, ts.Reason, ts.Message)
+		}
+	}
+}
+
+// A daemon that stopped answering times the check out, but the connection
+// the reconciles share stays up, and the next check does not stack a
+// second hung RPC on top of the first.
+func TestCheck_timeoutKeepsTheConnection(t *testing.T) {
+	t.Parallel()
+	api := newFakeAPI("default")
+	block := make(chan struct{})
+	t.Cleanup(func() { close(block) })
+	blocked := &blockingAPI{fakeAPI: api, block: block}
+	var closed atomic.Bool
+	dial := func(context.Context, provider.KVMHost) (*hostClient, error) {
+		return &hostClient{api: blocked, close: func() { closed.Store(true) }}, nil
+	}
+	pc := testConfig("nas1")
+	p := newWithDialer(pc.Name, *pc.KVM, dial, nil)
+	ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
+	defer cancel()
+
+	got := p.Check(ctx)
+
+	if got[0].Ready || got[0].Reason != provider.ReasonTimeout {
+		t.Errorf("Check() = %+v, want Timeout", got[0])
+	}
+	if closed.Load() {
+		t.Error("the timed-out check closed the connection the reconciles share")
+	}
+	again := p.Check(context.Background())
+	if again[0].Reason != provider.ReasonTimeout || !strings.Contains(again[0].Message, "still waiting") {
+		t.Errorf("second Check() = %+v, want an immediate Timeout while the first still waits", again[0])
+	}
+}
+
+// A reconcile redialing a dead host holds the host's slot for minutes; the
+// check must give up at its own deadline rather than wait that out.
+func TestCheck_doesNotWaitBehindAReconcileDial(t *testing.T) {
+	t.Parallel()
+	release := make(chan struct{})
+	dialing := make(chan struct{})
+	var once sync.Once
+	dial := func(context.Context, provider.KVMHost) (*hostClient, error) {
+		once.Do(func() { close(dialing) })
+		<-release
+		return nil, errHostOff
+	}
+	pc := testConfig("nas1")
+	p := newWithDialer(pc.Name, *pc.KVM, dial, nil)
+	reconcile := make(chan error, 1)
+	go func() {
+		_, err := p.Get(context.Background(), formatProviderID("nas1", testName))
+		reconcile <- err
+	}()
+	<-dialing
+	ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
+	defer cancel()
+	start := time.Now()
+
+	got := p.Check(ctx)
+
+	if waited := time.Since(start); waited > time.Second {
+		t.Errorf("Check() took %s behind the reconcile's dial, want about 50ms", waited)
+	}
+	if got[0].Reason != provider.ReasonTimeout {
+		t.Errorf("Check() = %+v, want Timeout", got[0])
+	}
+	close(release)
+	<-reconcile
+}
+
+// blockingAPI hangs in ConnectGetLibVersion until block closes, like a
+// daemon that stopped answering.
+type blockingAPI struct {
+	*fakeAPI
+	block chan struct{}
+}
+
+func (b *blockingAPI) ConnectGetLibVersion() (uint64, error) {
+	<-b.block
+	return 0, io.EOF
+}
+
+func TestCheckReason(t *testing.T) {
+	t.Parallel()
+	tests := []struct {
+		name string
+		err  error
+		want string
+	}{
+		{"ssh material", &dialError{stage: stageSSHMaterial, err: fmt.Errorf("%w: parsing key", provider.ErrPermanent)}, reasonSSHMaterialUnavailable},
+		{"tcp", &dialError{stage: stageTCP, err: errors.New("ssh dial: connection refused")}, provider.ReasonUnreachable},
+		{"ssh handshake", &dialError{stage: stageSSHHandshake, err: errors.New("ssh handshake: unable to authenticate")}, reasonSSHHandshakeFailed},
+		{"libvirt connect", &dialError{stage: stageLibvirtConnect, err: errors.New("libvirt connect via /sock: EOF")}, reasonLibvirtConnectFailed},
+		{"stage survives the pool's wrapping", fmt.Errorf("connecting to host nas1 (nas1:22): %w",
+			&dialError{stage: stageSSHHandshake, err: context.DeadlineExceeded}), reasonSSHHandshakeFailed},
+		{"deadline", context.DeadlineExceeded, provider.ReasonTimeout},
+		{"cancelled", context.Canceled, provider.ReasonTimeout},
+		{"previous check still waiting", errProbeBusy, provider.ReasonTimeout},
+		{"no pool", lvError(libvirt.ErrNoStoragePool, "no pool"), reasonPoolNotFound},
+		{"other libvirt error", lvError(libvirt.ErrInternalError, "boom"), reasonLibvirtError},
+		{"link died mid-call", io.EOF, provider.ReasonUnreachable},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			if got := checkReason(tc.err); got != tc.want {
+				t.Errorf("checkReason(%v) = %s, want %s", tc.err, got, tc.want)
+			}
+		})
+	}
+}
+
+func TestDialError_keepsClassification(t *testing.T) {
+	t.Parallel()
+	err := fmt.Errorf("connecting to host nas1: %w",
+		&dialError{stage: stageSSHMaterial, err: fmt.Errorf("%w: parsing kvm.sshKeyFile: bad", provider.ErrPermanent)})
+
+	if got := classify(err); !errors.Is(got, provider.ErrPermanent) {
+		t.Errorf("classify() = %v, want ErrPermanent through the dialError", got)
+	}
+	if !isClassified(err) {
+		t.Error("isClassified() = false, want true")
+	}
+	if want := "connecting to host nas1: permanent: parsing kvm.sshKeyFile: bad"; !strings.HasSuffix(err.Error(), "parsing kvm.sshKeyFile: bad") {
+		t.Errorf("message = %q, want it unchanged, like %q", err.Error(), want)
+	}
+}
+
+func TestFormatLibVersion(t *testing.T) {
+	t.Parallel()
+	tests := []struct {
+		in   uint64
+		want string
+	}{
+		{10002000, "10.2.0"},
+		{11006001, "11.6.1"},
+		{9000000, "9.0.0"},
+	}
+	for _, tc := range tests {
+		if got := formatLibVersion(tc.in); got != tc.want {
+			t.Errorf("formatLibVersion(%d) = %s, want %s", tc.in, got, tc.want)
+		}
+	}
+}

internal/provider/kvm/conn.go

@@ -9,6 +9,7 @@ import (
 	"net"
 	"os"
 	"sync"
+	"sync/atomic"
 	"time"
 
 	libvirt "github.com/digitalocean/go-libvirt"
@@ -44,6 +45,9 @@ type libvirtAPI interface {
 	StorageVolUpload(vol libvirt.StorageVol, r io.Reader, offset, length uint64, flags libvirt.StorageVolUploadFlags) error
 	StorageVolDownload(vol libvirt.StorageVol, w io.Writer, offset, length uint64, flags libvirt.StorageVolDownloadFlags) error
 	StorageVolDelete(vol libvirt.StorageVol, flags libvirt.StorageVolDeleteFlags) error
+	ConnectGetLibVersion() (uint64, error)
+	ConnectGetHostname() (string, error)
+	StoragePoolGetInfo(pool libvirt.StoragePool) (state uint8, capacity, allocation, available uint64, err error)
 }
 
 // hostClient is one live connection to a hypervisor.
@@ -68,6 +72,27 @@ func (c *hostClient) alive() bool {
 // real one; tests inject their own.
 type hostDialer func(ctx context.Context, host provider.KVMHost) (*hostClient, error)
 
+// dialStage names the step of a connection attempt that failed, so the
+// health check can say why a host is unreachable without parsing messages.
+type dialStage string
+
+const (
+	stageSSHMaterial    dialStage = "ssh-material"
+	stageTCP            dialStage = "tcp"
+	stageSSHHandshake   dialStage = "ssh-handshake"
+	stageLibvirtConnect dialStage = "libvirt-connect"
+)
+
+// dialError tags a dial failure with its stage. It keeps the message and
+// unwraps to the cause, so classify sees exactly what it saw before.
+type dialError struct {
+	stage dialStage
+	err   error
+}
+
+func (e *dialError) Error() string { return e.err.Error() }
+func (e *dialError) Unwrap() error { return e.err }
+
 // hostPool caches one connection per host, dialing lazily and redialing
 // after the connection drops. Dials for different hosts don't block each
 // other; concurrent callers for the same host share a single dial.
@@ -78,9 +103,16 @@ type hostPool struct {
 	entries map[string]*hostEntry
 }
 
+// hostEntry is one host's slot in the pool. lock is a one-slot semaphore
+// rather than a mutex so that waiting for it can end with the caller's
+// context: whoever holds it may be dialing a dead host for as long as the
+// dial takes.
 type hostEntry struct {
-	mu     sync.Mutex
+	lock   chan struct{}
 	client *hostClient
+	// probing is set while a probe's fn runs, including one whose caller
+	// has already given up on it.
+	probing atomic.Bool
 }
 
 func newHostPool(dial hostDialer) *hostPool {
@@ -92,16 +124,37 @@ func (p *hostPool) entry(name string) *hostEntry {
 	defer p.mu.Unlock()
 	e, ok := p.entries[name]
 	if !ok {
-		e = &hostEntry{}
+		e = &hostEntry{lock: make(chan struct{}, 1)}
 		p.entries[name] = e
 	}
 	return e
 }
 
+// acquire takes e.lock. Only waiting is bounded by ctx: a free slot is
+// taken even when ctx has already ended, so an uncontended caller behaves
+// as it did with a mutex, and a contended one gives up with ctx's error.
+func (e *hostEntry) acquire(ctx context.Context) error {
+	select {
+	case e.lock <- struct{}{}:
+		return nil
+	default:
+	}
+	select {
+	case e.lock <- struct{}{}:
+		return nil
+	case <-ctx.Done():
+		return ctx.Err()
+	}
+}
+
+func (e *hostEntry) release() { <-e.lock }
+
 func (p *hostPool) get(ctx context.Context, log logr.Logger, host provider.KVMHost) (*hostClient, error) {
 	e := p.entry(host.Name)
-	e.mu.Lock()
-	defer e.mu.Unlock()
+	if err := e.acquire(ctx); err != nil {
+		return nil, fmt.Errorf("waiting for the connection to host %s (%s): %w", host.Name, host.Address, err)
+	}
+	defer e.release()
 	if e.client != nil {
 		if e.client.alive() {
 			return e.client, nil
@@ -125,8 +178,8 @@ func (p *hostPool) get(ctx context.Context, log logr.Logger, host provider.KVMHo
 // connection that was already replaced by a redial is left alone.
 func (p *hostPool) drop(host string, c *hostClient) {
 	e := p.entry(host)
-	e.mu.Lock()
-	defer e.mu.Unlock()
+	e.lock <- struct{}{}
+	defer e.release()
 	if e.client == c {
 		e.client = nil
 	}
@@ -149,19 +202,61 @@ func (p *hostPool) call(ctx context.Context, log logr.Logger, host provider.KVMH
 	if err != nil {
 		return err
 	}
-	api := c.api
-	if log.V(1).Enabled() {
-		api = &tracingAPI{inner: c.api, log: log}
-	}
-	err = awaitOrCancel(ctx, func() error { return fn(api) }, func() {
+	err = awaitOrCancel(ctx, func() error { return fn(traced(c, log)) }, func() {
 		log.V(1).Info("libvirt connection dropped", "reason", "context cancelled", "error", ctx.Err().Error())
 		p.drop(host.Name, c)
 	})
+	p.dropOnTransportError(ctx, log, host, c, err)
+	return err
+}
+
+// errProbeBusy refuses a probe while the previous one on the same host has
+// not come back yet.
+var errProbeBusy = errors.New("the previous check of this host is still waiting for libvirt to answer")
+
+// probe is call for the health check, which must never cost the reconciles
+// their connection: when ctx ends first, fn is abandoned and the connection
+// left alone. A link that is really dead is closed by the keepalive, which
+// ends fn too. At most one probe per host runs at a time, an abandoned one
+// included, so a daemon that stopped answering cannot pile up goroutines;
+// until it answers, further probes get errProbeBusy.
+func (p *hostPool) probe(ctx context.Context, log logr.Logger, host provider.KVMHost, fn func(api libvirtAPI) error) error {
+	log = log.WithName("libvirt")
+	e := p.entry(host.Name)
+	if !e.probing.CompareAndSwap(false, true) {
+		return errProbeBusy
+	}
+	c, err := p.get(ctx, log, host)
+	if err != nil {
+		e.probing.Store(false)
+		return err
+	}
+	err = awaitOrCancel(ctx, func() error {
+		defer e.probing.Store(false)
+		return fn(traced(c, log))
+	}, func() {
+		log.V(1).Info("libvirt probe abandoned, connection kept", "error", ctx.Err().Error())
+	})
+	p.dropOnTransportError(ctx, log, host, c, err)
+	return err
+}
+
+// traced returns c's API, logging every RPC when log has V(1) enabled.
+func traced(c *hostClient, log logr.Logger) libvirtAPI {
+	if log.V(1).Enabled() {
+		return &tracingAPI{inner: c.api, log: log}
+	}
+	return c.api
+}
+
+// dropOnTransportError drops c when err is evidence against the link:
+// neither a libvirt error, nor a verdict already classified, nor the
+// caller's own context ending.
+func (p *hostPool) dropOnTransportError(ctx context.Context, log logr.Logger, host provider.KVMHost, c *hostClient, err error) {
 	if err != nil && !isLibvirtError(err) && !isClassified(err) && !errors.Is(err, ctx.Err()) {
 		log.V(1).Info("libvirt connection dropped", "reason", "transport error", "error", err.Error())
 		p.drop(host.Name, c)
 	}
-	return err
 }
 
 // awaitOrCancel runs fn in the background and returns its result, unless ctx
@@ -254,7 +349,7 @@ func (l *lazySSH) load() (*sshAuth, error) {
 func (l *lazySSH) dial(ctx context.Context, host provider.KVMHost) (*hostClient, error) {
 	a, err := l.load()
 	if err != nil {
-		return nil, err
+		return nil, &dialError{stage: stageSSHMaterial, err: err}
 	}
 	return a.dial(ctx, host)
 }
@@ -285,7 +380,7 @@ func (a *sshAuth) dialSSH(ctx context.Context, host provider.KVMHost) (*ssh.Clie
 	var d net.Dialer
 	nc, err := d.DialContext(ctx, "tcp", host.Address)
 	if err != nil {
-		return nil, fmt.Errorf("ssh dial: %w", err)
+		return nil, &dialError{stage: stageTCP, err: fmt.Errorf("ssh dial: %w", err)}
 	}
 	_ = nc.SetDeadline(deadline)
 	conn, chans, reqs, err := ssh.NewClientConn(nc, host.Address, &ssh.ClientConfig{
@@ -295,7 +390,7 @@ func (a *sshAuth) dialSSH(ctx context.Context, host provider.KVMHost) (*ssh.Clie
 	})
 	if err != nil {
 		nc.Close()
-		return nil, fmt.Errorf("ssh handshake: %w", err)
+		return nil, &dialError{stage: stageSSHHandshake, err: fmt.Errorf("ssh handshake: %w", err)}
 	}
 	_ = nc.SetDeadline(time.Time{})
 	return ssh.NewClient(conn, chans, reqs), nil
@@ -319,7 +414,7 @@ func (u unixForward) Dial() (net.Conn, error) {
 func connectLibvirt(ctx context.Context, client *ssh.Client, socket string) (*hostClient, error) {
 	lv := libvirt.NewWithDialer(unixForward{client: client, path: socket})
 	if err := awaitOrCancel(ctx, lv.Connect, func() { client.Close() }); err != nil {
-		return nil, fmt.Errorf("libvirt connect via %s: %w", socket, err)
+		return nil, &dialError{stage: stageLibvirtConnect, err: fmt.Errorf("libvirt connect via %s: %w", socket, err)}
 	}
 	stop := make(chan struct{})
 	var once sync.Once

internal/provider/kvm/conn_test.go

@@ -232,6 +232,97 @@ func TestHostPool_call_hostsAreIndependent(t *testing.T) {
 	}
 }
 
+// A dial to a dead host holds the host's slot for as long as it takes; a
+// caller with a deadline must not wait past it.
+func TestHostPool_get_waitingRespectsContext(t *testing.T) {
+	t.Parallel()
+	release := make(chan struct{})
+	dialing := make(chan struct{})
+	var once sync.Once
+	dial := func(context.Context, provider.KVMHost) (*hostClient, error) {
+		once.Do(func() { close(dialing) })
+		<-release
+		return nil, errors.New("ssh dial: connection timed out")
+	}
+	p := newHostPool(dial)
+	slow := make(chan error, 1)
+	go func() { slow <- p.call(context.Background(), discard, testHost, noop) }()
+	<-dialing
+	ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
+	defer cancel()
+	start := time.Now()
+
+	err := p.call(ctx, discard, testHost, noop)
+
+	if !errors.Is(err, context.DeadlineExceeded) {
+		t.Errorf("call() error = %v, want the caller's deadline", err)
+	}
+	if waited := time.Since(start); waited > time.Second {
+		t.Errorf("call() waited %s behind another caller's dial, want about 50ms", waited)
+	}
+	close(release)
+	if err := <-slow; err == nil {
+		t.Error("the blocked dial succeeded, want its error")
+	}
+}
+
+func TestHostPool_probe_abandonsWithoutClosing(t *testing.T) {
+	t.Parallel()
+	d := &fakeDialer{}
+	p := newHostPool(d.dial)
+	unblock := make(chan struct{})
+	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
+	defer cancel()
+
+	err := p.probe(ctx, discard, testHost, func(libvirtAPI) error {
+		<-unblock // a daemon that stopped answering
+		return nil
+	})
+
+	if !errors.Is(err, context.DeadlineExceeded) {
+		t.Fatalf("probe() error = %v, want DeadlineExceeded", err)
+	}
+	if d.clients[0].closed.Load() {
+		t.Error("probe closed the shared connection when it gave up")
+	}
+	if err := p.probe(context.Background(), discard, testHost, noop); !errors.Is(err, errProbeBusy) {
+		t.Errorf("probe() while the abandoned one still waits = %v, want errProbeBusy", err)
+	}
+	if err := p.call(context.Background(), discard, testHost, noop); err != nil {
+		t.Errorf("call() on the kept connection = %v", err)
+	}
+	close(unblock)
+	deadline := time.Now().Add(time.Second)
+	for {
+		err := p.probe(context.Background(), discard, testHost, noop)
+		if err == nil {
+			break
+		}
+		if !errors.Is(err, errProbeBusy) || time.Now().After(deadline) {
+			t.Fatalf("probe() after the abandoned one returned = %v, want nil", err)
+		}
+		time.Sleep(time.Millisecond)
+	}
+	if d.dialCount() != 1 {
+		t.Errorf("dials = %d, want 1: nothing should have redialed", d.dialCount())
+	}
+}
+
+func TestHostPool_probe_dropsConnectionOnTransportError(t *testing.T) {
+	t.Parallel()
+	d := &fakeDialer{}
+	p := newHostPool(d.dial)
+
+	err := p.probe(context.Background(), discard, testHost, func(libvirtAPI) error { return io.ErrUnexpectedEOF })
+
+	if !errors.Is(err, io.ErrUnexpectedEOF) {
+		t.Fatalf("probe() error = %v, want io.ErrUnexpectedEOF passed through", err)
+	}
+	if !d.clients[0].closed.Load() {
+		t.Error("a transport error during a probe left the connection cached")
+	}
+}
+
 // --- SSH dial against an in-process server ---
 
 func newKeyPair(t *testing.T) (ssh.Signer, []byte) {
@@ -614,6 +705,86 @@ func TestNew_missingSSHMaterialStillBuilds(t *testing.T) {
 	}
 }
 
+// The health check reports where a connection broke, so every stage of a
+// real dial must come back tagged.
+func TestLazySSH_dial_tagsTheFailingStage(t *testing.T) {
+	t.Parallel()
+	tests := []struct {
+		name       string
+		mutate     func(t *testing.T, f *sshFixture)
+		want       dialStage
+		wantReason string
+	}{
+		{
+			name:       "key material missing",
+			mutate:     func(t *testing.T, f *sshFixture) { f.cfg.SSHKeyFile = filepath.Join(t.TempDir(), "absent") },
+			want:       stageSSHMaterial,
+			wantReason: reasonSSHMaterialUnavailable,
+		},
+		{
+			name: "nothing listening",
+			mutate: func(t *testing.T, f *sshFixture) {
+				ln, err := net.Listen("tcp", "127.0.0.1:0")
+				if err != nil {
+					t.Fatal(err)
+				}
+				f.host.Address = ln.Addr().String()
+				ln.Close()
+			},
+			want:       stageTCP,
+			wantReason: provider.ReasonUnreachable,
+		},
+		{
+			name: "host key not in known_hosts",
+			mutate: func(t *testing.T, f *sshFixture) {
+				impostor, _ := newKeyPair(t)
+				line := knownhosts.Line([]string{f.server.addr}, impostor.PublicKey())
+				if err := os.WriteFile(f.cfg.KnownHostsFile, []byte(line+"\n"), 0o600); err != nil {
+					t.Fatal(err)
+				}
+			},
+			want:       stageSSHHandshake,
+			wantReason: reasonSSHHandshakeFailed,
+		},
+		{
+			name: "client key rejected",
+			mutate: func(t *testing.T, f *sshFixture) {
+				_, otherPEM := newKeyPair(t)
+				if err := os.WriteFile(f.keyFile, otherPEM, 0o600); err != nil {
+					t.Fatal(err)
+				}
+			},
+			want:       stageSSHHandshake,
+			wantReason: reasonSSHHandshakeFailed,
+		},
+		{
+			name:       "libvirt socket missing",
+			mutate:     func(_ *testing.T, f *sshFixture) { f.host.Socket = "/nonexistent/libvirt-sock" },
+			want:       stageLibvirtConnect,
+			wantReason: reasonLibvirtConnectFailed,
+		},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			f := newSSHFixture(t, true)
+			tc.mutate(t, f)
+			ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+			defer cancel()
+
+			_, err := newLazySSH(f.cfg).dial(ctx, f.host)
+
+			var de *dialError
+			if !errors.As(err, &de) || de.stage != tc.want {
+				t.Fatalf("dial() error = %v, want a %s dialError", err, tc.want)
+			}
+			if got := checkReason(err); got != tc.wantReason {
+				t.Errorf("checkReason() = %s, want %s", got, tc.wantReason)
+			}
+		})
+	}
+}
+
 func TestKeepalive_closesClientWhenServerStalls(t *testing.T) {
 	t.Parallel()
 	f := newSSHFixture(t, false)

internal/provider/kvm/fake_test.go

@@ -28,6 +28,12 @@ type fakeAPI struct {
 	// uploadHook, when set, sees every StorageVolUpload before the fake
 	// reads the stream; it can return an error to simulate a broken upload.
 	uploadHook func(vol string, r io.Reader) error
+	// hostname, libVersion, poolCapacity and poolStates answer the health
+	// check's RPCs. A pool missing from poolStates is running.
+	hostname     string
+	libVersion   uint64
+	poolCapacity uint64
+	poolStates   map[string]libvirt.StoragePoolState
 }
 
 type fakeVol struct {
@@ -53,6 +59,11 @@ func newFakeAPI(pools ...string) *fakeAPI {
 		domains:  map[string]*fakeDomain{},
 		fail:     map[string]error{},
 		failOnce: map[string]error{},
+
+		hostname:     "fake-host",
+		libVersion:   10002000,
+		poolCapacity: 100 << 30,
+		poolStates:   map[string]libvirt.StoragePoolState{},
 	}
 	for _, p := range pools {
 		f.pools[p] = map[string]*fakeVol{}
@@ -409,3 +420,46 @@ func (f *fakeAPI) StorageVolDelete(vol libvirt.StorageVol, _ libvirt.StorageVolD
 	delete(f.pools[vol.Pool], vol.Name)
 	return nil
 }
+
+// --- host ---
+
+func (f *fakeAPI) ConnectGetLibVersion() (uint64, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	if err := f.enter("ConnectGetLibVersion"); err != nil {
+		return 0, err
+	}
+	return f.libVersion, nil
+}
+
+func (f *fakeAPI) ConnectGetHostname() (string, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	if err := f.enter("ConnectGetHostname"); err != nil {
+		return "", err
+	}
+	return f.hostname, nil
+}
+
+// StoragePoolGetInfo reports the pool's state and poolCapacity minus the
+// volumes' capacities as available space.
+func (f *fakeAPI) StoragePoolGetInfo(pool libvirt.StoragePool) (uint8, uint64, uint64, uint64, error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	if err := f.enter("StoragePoolGetInfo"); err != nil {
+		return 0, 0, 0, 0, err
+	}
+	vols, ok := f.pools[pool.Name]
+	if !ok {
+		return 0, 0, 0, 0, lvError(libvirt.ErrNoStoragePool, "no pool %s", pool.Name)
+	}
+	var used uint64
+	for _, v := range vols {
+		used += uint64(v.capacity)
+	}
+	state := libvirt.StoragePoolRunning
+	if s, ok := f.poolStates[pool.Name]; ok {
+		state = s
+	}
+	return uint8(state), f.poolCapacity, used, f.poolCapacity - used, nil
+}

internal/provider/kvm/kvm_integration_test.go

@@ -347,6 +347,36 @@ func TestKVM_lifecycle(t *testing.T) {
 	env.keep = false
 }
 
+// TestKVMCheck_realHost runs the provider health check against the host.
+// It creates nothing: it must reach the host and find the pool running.
+func TestKVMCheck_realHost(t *testing.T) {
+	env := loadIntegrationEnv(t)
+	log := integrationLogger(t, env.logLevel)
+	ctx, cancel := context.WithTimeout(logr.NewContext(context.Background(), log), time.Minute)
+	defer cancel()
+
+	prov, err := New(ctx, env.cfg)
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	got := prov.(*Provider).Check(ctx)
+
+	if len(got) != 1 {
+		t.Fatalf("Check returned %d targets, want 1", len(got))
+	}
+	ts := got[0]
+	t.Logf("%s (%s): ready=%v reason=%s latency=%s details=%v",
+		ts.Name, ts.Address, ts.Ready, ts.Reason, ts.Latency.Round(time.Millisecond), ts.Details)
+	if !ts.Ready {
+		t.Fatalf("check failed: %s: %s", ts.Reason, ts.Message)
+	}
+	for _, key := range []string{"hostname", "libvirtVersion", "poolState", "poolAvailable"} {
+		if ts.Details[key] == "" {
+			t.Errorf("details[%s] is empty, want what the host reported", key)
+		}
+	}
+}
+
 // waitForRunning polls Get until the instance is Running with an IP or ctx
 // ends. Transient errors are retried; anything else fails the test.
 func waitForRunning(ctx context.Context, t *testing.T, p *Provider, id string) *provider.Instance {

internal/provider/kvm/trace.go

@@ -209,3 +209,24 @@ func (t *tracingAPI) StorageVolDelete(vol libvirt.StorageVol, flags libvirt.Stor
 	c.done(err)
 	return err
 }
+
+func (t *tracingAPI) ConnectGetLibVersion() (uint64, error) {
+	c := t.begin("ConnectGetLibVersion")
+	v, err := t.inner.ConnectGetLibVersion()
+	c.done(err, "version", formatLibVersion(v))
+	return v, err
+}
+
+func (t *tracingAPI) ConnectGetHostname() (string, error) {
+	c := t.begin("ConnectGetHostname")
+	h, err := t.inner.ConnectGetHostname()
+	c.done(err, "hostname", h)
+	return h, err
+}
+
+func (t *tracingAPI) StoragePoolGetInfo(pool libvirt.StoragePool) (uint8, uint64, uint64, uint64, error) {
+	c := t.begin("StoragePoolGetInfo", "pool", pool.Name)
+	state, capacity, allocation, available, err := t.inner.StoragePoolGetInfo(pool)
+	c.done(err, "state", poolStateName(libvirt.StoragePoolState(state)), "capacity", capacity, "available", available)
+	return state, capacity, allocation, available, err
+}

internal/provider/kvm/trace_test.go

@@ -103,6 +103,9 @@ func TestTracingAPI_logsEveryMethod(t *testing.T) {
 			return api.StorageVolDownload(overlay, io.Discard, 0, 0, 0)
 		}},
 		{"StorageVolDelete", func(api libvirtAPI) error { return api.StorageVolDelete(overlay, 0) }},
+		{"ConnectGetLibVersion", func(api libvirtAPI) error { _, err := api.ConnectGetLibVersion(); return err }},
+		{"ConnectGetHostname", func(api libvirtAPI) error { _, err := api.ConnectGetHostname(); return err }},
+		{"StoragePoolGetInfo", func(api libvirtAPI) error { _, _, _, _, err := api.StoragePoolGetInfo(pool); return err }},
 	}
 	if n := reflect.TypeOf((*libvirtAPI)(nil)).Elem().NumMethod(); n != len(cases) {
 		t.Fatalf("libvirtAPI has %d methods, this test covers %d; trace the new one", n, len(cases))

internal/providerhealth/envtest_test.go

@@ -0,0 +1,100 @@
+package providerhealth
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+	"unicode/utf8"
+
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/envtest"
+
+	crawlv1alpha1 "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/api/v1alpha1"
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
+)
+
+// TestMonitor_envtestRoundTrip writes through a real API server, which is
+// the only thing that validates the generated schema: cluster scope, the
+// status subresource, required fields, list-map keys and the message limit.
+func TestMonitor_envtestRoundTrip(t *testing.T) {
+	if testing.Short() {
+		t.Skip("envtest skipped under -short")
+	}
+	env := &envtest.Environment{
+		CRDDirectoryPaths:     []string{filepath.Join("..", "..", "config", "crd", "bases")},
+		ErrorIfCRDPathMissing: true,
+		BinaryAssetsDirectory: envtestBinaryDir(),
+	}
+	cfg, err := env.Start()
+	if err != nil {
+		t.Fatalf("starting envtest (run `make setup-envtest` first): %v", err)
+	}
+	t.Cleanup(func() { _ = env.Stop() })
+	scheme := runtime.NewScheme()
+	if err := crawlv1alpha1.AddToScheme(scheme); err != nil {
+		t.Fatal(err)
+	}
+	c, err := client.New(cfg, client.Options{Scheme: scheme})
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx := context.Background()
+	m := &Monitor{Client: c, Reader: c}
+
+	first := Result{Provider: "kvm-home", Type: "kvm", Supported: true, CheckedAt: t0, Targets: []provider.TargetStatus{
+		{Name: "nas1", Address: "nas1:22", Ready: true, Reason: provider.ReasonReachable,
+			Latency: 42 * time.Millisecond, Details: map[string]string{"libvirtVersion": "10.2.0"}},
+		{Name: "nas2", Address: "nas2:22", Reason: provider.ReasonUnreachable, Message: strings.Repeat("x", 2000)},
+	}}
+	if err := m.publish(ctx, first); err != nil {
+		t.Fatalf("first publish: %v", err)
+	}
+	got := getHealth(t, c, "kvm-home")
+	if got.Status.TargetsReady != "1/2" || got.Status.Targets[0].Details["libvirtVersion"] != "10.2.0" ||
+		got.Status.Targets[0].LatencyMillis != 42 || utf8.RuneCountInString(got.Status.Targets[1].Message) != maxMessageLen {
+		t.Errorf("status after the first round = %+v", got.Status)
+	}
+
+	second := Result{Provider: "kvm-home", Type: "kvm", Supported: true, CheckedAt: t1,
+		Targets: []provider.TargetStatus{up("nas1"), up("nas2")}}
+	if err := m.publish(ctx, second); err != nil {
+		t.Fatalf("second publish: %v", err)
+	}
+	got = getHealth(t, c, "kvm-home")
+	if cond := readyCondition(got); cond == nil || cond.Reason != reasonAllTargetsReady {
+		t.Errorf("Ready after recovery = %+v, want %s", cond, reasonAllTargetsReady)
+	}
+	if lt := got.Status.Targets[0].LastTransitionTime.Time; !lt.Equal(t0) {
+		t.Errorf("nas1 lastTransitionTime = %s, want %s", lt, t0)
+	}
+	if lt := got.Status.Targets[1].LastTransitionTime.Time; !lt.Equal(t1) {
+		t.Errorf("nas2 lastTransitionTime = %s, want %s", lt, t1)
+	}
+
+	m.cleanup(ctx, nil)
+	var gone crawlv1alpha1.ProviderHealth
+	if err := c.Get(ctx, client.ObjectKey{Name: "kvm-home"}, &gone); !apierrors.IsNotFound(err) {
+		t.Errorf("Get after cleanup = %v, want NotFound", err)
+	}
+}
+
+// envtestBinaryDir finds the binaries `make setup-envtest` downloads, so
+// the test also runs outside make (which sets KUBEBUILDER_ASSETS instead).
+func envtestBinaryDir() string {
+	base := filepath.Join("..", "..", "bin", "k8s")
+	entries, err := os.ReadDir(base)
+	if err != nil {
+		return ""
+	}
+	for _, e := range entries {
+		if e.IsDir() {
+			return filepath.Join(base, e.Name())
+		}
+	}
+	return ""
+}

internal/providerhealth/monitor.go

@@ -0,0 +1,296 @@
+// Package providerhealth checks every configured provider on an interval
+// and publishes the results: one ProviderHealth object per provider, a log
+// line whenever a target goes down or comes back, and a snapshot the
+// metrics collector reads at scrape time.
+package providerhealth
+
+import (
+	"context"
+	"fmt"
+	"slices"
+	"sync"
+	"time"
+
+	"github.com/go-logr/logr"
+	"go.opentelemetry.io/otel/attribute"
+	"k8s.io/apimachinery/pkg/api/equality"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/util/validation"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	logf "sigs.k8s.io/controller-runtime/pkg/log"
+
+	crawlv1alpha1 "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/api/v1alpha1"
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/tracing"
+)
+
+const (
+	// labelManagedBy marks the ProviderHealth objects this operator owns;
+	// cleanup never deletes an object without it.
+	labelManagedBy = "app.kubernetes.io/managed-by"
+	managedByValue = "egress-proxies-operator"
+
+	minInterval    = 10 * time.Second
+	defaultTimeout = 15 * time.Second
+)
+
+// Target is one configured provider. Checker is nil for a provider that
+// cannot check itself.
+type Target struct {
+	Name    string
+	Type    string
+	Checker provider.Checker
+}
+
+// Result is the outcome of one provider's latest check.
+type Result struct {
+	Provider string
+	Type     string
+	// Supported is false when the provider has no Checker.
+	Supported bool
+	Targets   []provider.TargetStatus
+	CheckedAt time.Time
+}
+
+// controller-gen reads RBAC markers only at package level, so they stay
+// out of Monitor's doc comment.
+// +kubebuilder:rbac:groups=crawl.example.com,resources=providerhealths,verbs=get;list;create;delete
+// +kubebuilder:rbac:groups=crawl.example.com,resources=providerhealths/status,verbs=get;patch
+
+// Monitor is the manager Runnable that checks the providers.
+type Monitor struct {
+	// Client writes the ProviderHealth objects.
+	Client client.Client
+	// Reader reads them; pass the manager's API reader. The objects are
+	// cluster-scoped, and reading them through the cache would start a
+	// cluster-wide informer needing watch rights, whatever --proxy-namespace
+	// says, for a handful of reads a minute.
+	Reader client.Reader
+	// Targets are the configured providers, in config order.
+	Targets []Target
+	// Interval between rounds of checks; 0 turns checking off. The first
+	// round runs at start: nothing races it, and early visibility is the
+	// point.
+	Interval time.Duration
+	// Timeout bounds each round; 0 means 15s, or the interval when that is
+	// shorter.
+	Timeout time.Duration
+
+	now func() time.Time
+
+	mu     sync.Mutex
+	latest []Result
+
+	// prev holds each target's previous verdict for transition logging.
+	// Only the loop goroutine touches it.
+	prev map[string]provider.TargetStatus
+}
+
+// NeedLeaderElection is true: only the leader dials the hypervisors and
+// writes the objects, and only its gauges describe the checks.
+func (m *Monitor) NeedLeaderElection() bool { return true }
+
+// ValidateSchedule checks --provider-check-interval and
+// --provider-check-timeout. The interval is 0 (off) or at least 10s, which
+// is what keeps one status write per provider per round cheap. The timeout
+// is 0 (derived from the interval, see Monitor.Timeout) or positive and no
+// longer than the interval.
+func ValidateSchedule(interval, timeout time.Duration) error {
+	switch {
+	case interval == 0:
+		return nil
+	case interval < minInterval:
+		return fmt.Errorf("--provider-check-interval must be 0 (off) or at least %s, got %s", minInterval, interval)
+	case timeout < 0 || timeout > interval:
+		return fmt.Errorf("--provider-check-timeout must be at most the interval (%s), got %s", interval, timeout)
+	}
+	return nil
+}
+
+func (m *Monitor) setDefaults() {
+	if m.Timeout == 0 {
+		m.Timeout = min(defaultTimeout, m.Interval)
+	}
+	if m.now == nil {
+		m.now = time.Now
+	}
+}
+
+// Start deletes the objects of providers that are no longer configured,
+// then checks every Interval until ctx ends. The providers config is read
+// once per process, so cleaning up once is enough. With checking off,
+// every object goes, so none is left claiming a stale Ready.
+func (m *Monitor) Start(ctx context.Context) error {
+	m.setDefaults()
+	log := logf.FromContext(ctx).WithName("provider-health")
+	ctx = logf.IntoContext(ctx, log)
+
+	keep := map[string]bool{}
+	if m.Interval > 0 {
+		for _, t := range m.Targets {
+			keep[t.Name] = true
+		}
+	}
+	m.cleanup(ctx, keep)
+	if m.Interval == 0 {
+		log.Info("Provider checks are off (--provider-check-interval=0)")
+		return nil
+	}
+	for _, t := range m.Targets {
+		if errs := validation.IsDNS1123Subdomain(t.Name); len(errs) > 0 {
+			log.Info("WARNING: provider name is not a valid object name; its ProviderHealth object is not written, its metrics are",
+				"provider", t.Name, "problems", errs)
+		}
+	}
+
+	m.checkOnce(ctx)
+	ticker := time.NewTicker(m.Interval)
+	defer ticker.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return nil
+		case <-ticker.C:
+			m.checkOnce(ctx)
+		}
+	}
+}
+
+// Snapshot returns the latest results in config order.
+func (m *Monitor) Snapshot() []Result {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	out := make([]Result, len(m.latest))
+	for i, r := range m.latest {
+		r.Targets = slices.Clone(r.Targets)
+		out[i] = r
+	}
+	return out
+}
+
+// checkOnce runs every provider's check concurrently under one timeout,
+// then records, logs and publishes the results provider by provider. A
+// failed write is logged and retried by the next round.
+func (m *Monitor) checkOnce(ctx context.Context) {
+	ctx, span := tracing.Start(ctx, "providerhealth.check")
+	defer span.End()
+	log := logf.FromContext(ctx)
+
+	checkCtx, cancel := context.WithTimeout(ctx, m.Timeout)
+	defer cancel()
+	results := make([]Result, len(m.Targets))
+	var wg sync.WaitGroup
+	for i, t := range m.Targets {
+		wg.Go(func() { results[i] = m.run(checkCtx, t) })
+	}
+	wg.Wait()
+
+	m.mu.Lock()
+	m.latest = results
+	m.mu.Unlock()
+
+	notReady := 0
+	for _, r := range results {
+		m.logTransitions(log, r)
+		for _, ts := range r.Targets {
+			if !ts.Ready {
+				notReady++
+			}
+		}
+		if err := m.publish(ctx, r); err != nil {
+			log.Error(err, "Writing provider health", "provider", r.Provider)
+		}
+	}
+	span.SetAttributes(
+		attribute.Int("providerhealth.providers", len(results)),
+		attribute.Int("providerhealth.targets_not_ready", notReady),
+	)
+}
+
+func (m *Monitor) run(ctx context.Context, t Target) Result {
+	r := Result{Provider: t.Name, Type: t.Type, Supported: t.Checker != nil}
+	if t.Checker != nil {
+		r.Targets = t.Checker.Check(ctx)
+	}
+	r.CheckedAt = m.now()
+	return r
+}
+
+// logTransitions logs a target going down, staying down for a different
+// reason, or coming back. Steady states stay quiet; a target that is ready
+// the first time it is seen only logs at V(1).
+func (m *Monitor) logTransitions(log logr.Logger, r Result) {
+	if m.prev == nil {
+		m.prev = map[string]provider.TargetStatus{}
+	}
+	for _, ts := range r.Targets {
+		key := r.Provider + "/" + ts.Name
+		prev, seen := m.prev[key]
+		m.prev[key] = ts
+		kv := []any{"provider", r.Provider, "target", ts.Name, "address", ts.Address, "reason", ts.Reason}
+		switch {
+		case !ts.Ready && (!seen || prev.Ready || prev.Reason != ts.Reason):
+			log.Info("WARNING: provider target not ready", append(kv, "message", ts.Message)...)
+		case ts.Ready && seen && !prev.Ready:
+			log.Info("Provider target ready", append(kv, "latency", ts.Latency.Round(time.Millisecond))...)
+		case ts.Ready && !seen:
+			log.V(1).Info("Provider target ready", append(kv, "latency", ts.Latency.Round(time.Millisecond))...)
+		}
+	}
+}
+
+// publish creates the provider's ProviderHealth object when missing and
+// patches its status. The status changes on every round, since
+// lastCheckTime moves; the comparison only skips the patch when a clock
+// did not advance.
+func (m *Monitor) publish(ctx context.Context, r Result) error {
+	if len(validation.IsDNS1123Subdomain(r.Provider)) > 0 {
+		return nil
+	}
+	var ph crawlv1alpha1.ProviderHealth
+	err := m.Reader.Get(ctx, client.ObjectKey{Name: r.Provider}, &ph)
+	switch {
+	case apierrors.IsNotFound(err):
+		ph = crawlv1alpha1.ProviderHealth{ObjectMeta: metav1.ObjectMeta{
+			Name:   r.Provider,
+			Labels: map[string]string{labelManagedBy: managedByValue},
+		}}
+		if err := m.Client.Create(ctx, &ph); err != nil {
+			return fmt.Errorf("creating ProviderHealth %s: %w", r.Provider, err)
+		}
+	case err != nil:
+		return fmt.Errorf("reading ProviderHealth %s: %w", r.Provider, err)
+	}
+	base := ph.DeepCopy()
+	ph.Status = buildStatus(ph.Status, r)
+	if equality.Semantic.DeepEqual(base.Status, ph.Status) {
+		return nil
+	}
+	if err := m.Client.Status().Patch(ctx, &ph, client.MergeFrom(base)); err != nil {
+		return fmt.Errorf("patching ProviderHealth %s status: %w", r.Provider, err)
+	}
+	return nil
+}
+
+// cleanup deletes this operator's ProviderHealth objects whose name is not
+// in keep.
+func (m *Monitor) cleanup(ctx context.Context, keep map[string]bool) {
+	log := logf.FromContext(ctx)
+	var list crawlv1alpha1.ProviderHealthList
+	if err := m.Reader.List(ctx, &list, client.MatchingLabels{labelManagedBy: managedByValue}); err != nil {
+		log.Error(err, "Listing provider health objects; stale ones stay until the next start")
+		return
+	}
+	for i := range list.Items {
+		ph := &list.Items[i]
+		if keep[ph.Name] {
+			continue
+		}
+		if err := m.Client.Delete(ctx, ph); client.IgnoreNotFound(err) != nil {
+			log.Error(err, "Deleting provider health of an unconfigured provider", "name", ph.Name)
+			continue
+		}
+		log.Info("Deleted provider health of an unconfigured provider", "name", ph.Name)
+	}
+}

internal/providerhealth/monitor_test.go

@@ -0,0 +1,377 @@
+package providerhealth
+
+import (
+	"context"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/go-logr/logr"
+	"github.com/go-logr/logr/funcr"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	apimeta "k8s.io/apimachinery/pkg/api/meta"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+
+	crawlv1alpha1 "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/api/v1alpha1"
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
+)
+
+var (
+	t0 = time.Date(2026, 9, 11, 15, 0, 0, 0, time.UTC)
+	t1 = t0.Add(time.Minute)
+)
+
+type checkFunc func(ctx context.Context) []provider.TargetStatus
+
+func (f checkFunc) Check(ctx context.Context) []provider.TargetStatus { return f(ctx) }
+
+func verdicts(ts ...provider.TargetStatus) checkFunc {
+	return func(context.Context) []provider.TargetStatus { return ts }
+}
+
+func up(name string) provider.TargetStatus {
+	return provider.TargetStatus{Name: name, Address: name + ":22", Ready: true, Reason: provider.ReasonReachable}
+}
+
+func down(name, reason string) provider.TargetStatus {
+	return provider.TargetStatus{Name: name, Address: name + ":22", Reason: reason, Message: name + " is down"}
+}
+
+func managed(name string) *crawlv1alpha1.ProviderHealth {
+	return &crawlv1alpha1.ProviderHealth{ObjectMeta: metav1.ObjectMeta{
+		Name:   name,
+		Labels: map[string]string{labelManagedBy: managedByValue},
+	}}
+}
+
+// newFakeClient registers ProviderHealth as cluster-scoped; the fake's
+// default mapper would treat it as namespaced.
+func newFakeClient(t *testing.T, objs ...client.Object) client.WithWatch {
+	t.Helper()
+	scheme := runtime.NewScheme()
+	if err := crawlv1alpha1.AddToScheme(scheme); err != nil {
+		t.Fatal(err)
+	}
+	mapper := apimeta.NewDefaultRESTMapper([]schema.GroupVersion{crawlv1alpha1.GroupVersion})
+	mapper.Add(crawlv1alpha1.GroupVersion.WithKind("ProviderHealth"), apimeta.RESTScopeRoot)
+	return fake.NewClientBuilder().
+		WithScheme(scheme).
+		WithRESTMapper(mapper).
+		WithStatusSubresource(&crawlv1alpha1.ProviderHealth{}).
+		WithObjects(objs...).
+		Build()
+}
+
+func newTestMonitor(c client.Client, targets ...Target) *Monitor {
+	return &Monitor{
+		Client:   c,
+		Reader:   c,
+		Targets:  targets,
+		Interval: time.Minute,
+		Timeout:  time.Second,
+		now:      func() time.Time { return t1 },
+	}
+}
+
+func getHealth(t *testing.T, c client.Reader, name string) *crawlv1alpha1.ProviderHealth {
+	t.Helper()
+	var ph crawlv1alpha1.ProviderHealth
+	if err := c.Get(context.Background(), client.ObjectKey{Name: name}, &ph); err != nil {
+		t.Fatalf("Get ProviderHealth %s: %v", name, err)
+	}
+	return &ph
+}
+
+func readyCondition(ph *crawlv1alpha1.ProviderHealth) *metav1.Condition {
+	return apimeta.FindStatusCondition(ph.Status.Conditions, crawlv1alpha1.ConditionReady)
+}
+
+func TestMonitor_checkOnce_createsLabelledObjects(t *testing.T) {
+	t.Parallel()
+	c := newFakeClient(t)
+	m := newTestMonitor(c,
+		Target{Name: "kvm-home", Type: "kvm", Checker: verdicts(up("nas1"), down("nas2", provider.ReasonUnreachable))},
+		Target{Name: "local", Type: "kubernetes", Checker: verdicts(up("apiserver"))},
+	)
+
+	m.checkOnce(context.Background())
+
+	kvm := getHealth(t, c, "kvm-home")
+	if kvm.Labels[labelManagedBy] != managedByValue {
+		t.Errorf("labels = %v, want %s=%s", kvm.Labels, labelManagedBy, managedByValue)
+	}
+	if cond := readyCondition(kvm); cond == nil || cond.Status != metav1.ConditionFalse || cond.Reason != reasonTargetsNotReady {
+		t.Errorf("kvm-home Ready = %+v, want False/%s", cond, reasonTargetsNotReady)
+	}
+	if kvm.Status.Type != "kvm" || kvm.Status.TargetsReady != "1/2" || kvm.Status.LastCheckTime == nil || !kvm.Status.LastCheckTime.Time.Equal(t1) {
+		t.Errorf("kvm-home status = %+v, want type kvm, 1/2 ready, checked at %s", kvm.Status, t1)
+	}
+	local := getHealth(t, c, "local")
+	if cond := readyCondition(local); cond == nil || cond.Status != metav1.ConditionTrue {
+		t.Errorf("local Ready = %+v, want True", cond)
+	}
+}
+
+func TestMonitor_checkOnce_patchesExisting(t *testing.T) {
+	t.Parallel()
+	existing := managed("local")
+	existing.Status = crawlv1alpha1.ProviderHealthStatus{
+		Type:    "kubernetes",
+		Targets: []crawlv1alpha1.ProviderTarget{readyAt("apiserver", t0)},
+	}
+	c := newFakeClient(t, existing)
+	m := newTestMonitor(c, Target{Name: "local", Type: "kubernetes", Checker: verdicts(up("apiserver"))})
+
+	m.checkOnce(context.Background())
+
+	got := getHealth(t, c, "local")
+	if got.Status.LastCheckTime == nil || !got.Status.LastCheckTime.Time.Equal(t1) {
+		t.Errorf("lastCheckTime = %v, want %s", got.Status.LastCheckTime, t1)
+	}
+	if lt := got.Status.Targets[0].LastTransitionTime.Time; !lt.Equal(t0) {
+		t.Errorf("apiserver lastTransitionTime = %s, want %s kept: it was ready before too", lt, t0)
+	}
+}
+
+func TestMonitor_checkOnce_timeoutBoundsEachRound(t *testing.T) {
+	t.Parallel()
+	c := newFakeClient(t)
+	var hadDeadline atomic.Bool
+	hanging := checkFunc(func(ctx context.Context) []provider.TargetStatus {
+		_, ok := ctx.Deadline()
+		hadDeadline.Store(ok)
+		<-ctx.Done()
+		return []provider.TargetStatus{down("nas1", provider.ReasonTimeout)}
+	})
+	m := newTestMonitor(c,
+		Target{Name: "kvm-home", Type: "kvm", Checker: hanging},
+		Target{Name: "local", Type: "kubernetes", Checker: verdicts(up("apiserver"))},
+	)
+	m.Timeout = 50 * time.Millisecond
+
+	done := make(chan struct{})
+	go func() {
+		m.checkOnce(context.Background())
+		close(done)
+	}()
+	select {
+	case <-done:
+	case <-time.After(5 * time.Second):
+		t.Fatal("checkOnce still running 5s after a 50ms timeout")
+	}
+
+	if !hadDeadline.Load() {
+		t.Error("the checker got a context without a deadline")
+	}
+	if cond := readyCondition(getHealth(t, c, "kvm-home")); cond == nil || cond.Status != metav1.ConditionFalse {
+		t.Errorf("kvm-home Ready = %+v, want False", cond)
+	}
+	if cond := readyCondition(getHealth(t, c, "local")); cond == nil || cond.Status != metav1.ConditionTrue {
+		t.Errorf("local Ready = %+v, want True despite the slow neighbour", cond)
+	}
+}
+
+func TestMonitor_checkOnce_noCheckerIsUnknown(t *testing.T) {
+	t.Parallel()
+	c := newFakeClient(t)
+	m := newTestMonitor(c, Target{Name: "gcp-eu", Type: "gcp"})
+
+	m.checkOnce(context.Background())
+
+	got := getHealth(t, c, "gcp-eu")
+	if cond := readyCondition(got); cond == nil || cond.Status != metav1.ConditionUnknown || cond.Reason != reasonChecksNotImplemented {
+		t.Errorf("Ready = %+v, want Unknown/%s", cond, reasonChecksNotImplemented)
+	}
+}
+
+func TestMonitor_checkOnce_invalidNameSkipsTheObject(t *testing.T) {
+	t.Parallel()
+	c := newFakeClient(t)
+	m := newTestMonitor(c, Target{Name: "Home_KVM", Type: "kvm", Checker: verdicts(up("nas1"))})
+
+	m.checkOnce(context.Background())
+
+	var list crawlv1alpha1.ProviderHealthList
+	if err := c.List(context.Background(), &list); err != nil {
+		t.Fatal(err)
+	}
+	if len(list.Items) != 0 {
+		t.Errorf("wrote %d objects, want none for an invalid name", len(list.Items))
+	}
+	snap := m.Snapshot()
+	if len(snap) != 1 || snap[0].Provider != "Home_KVM" || !snap[0].Targets[0].Ready {
+		t.Errorf("Snapshot() = %+v, want the result kept for metrics", snap)
+	}
+}
+
+func TestMonitor_logsOnlyTransitions(t *testing.T) {
+	t.Parallel()
+	var mu sync.Mutex
+	var lines []string
+	log := funcr.New(func(_, args string) {
+		mu.Lock()
+		defer mu.Unlock()
+		lines = append(lines, args)
+	}, funcr.Options{})
+	ctx := logr.NewContext(context.Background(), log)
+	rounds := [][]provider.TargetStatus{
+		{up("nas1")},
+		{up("nas1")},
+		{down("nas1", provider.ReasonUnreachable)},
+		{down("nas1", provider.ReasonUnreachable)},
+		{down("nas1", provider.ReasonTimeout)},
+		{up("nas1")},
+	}
+	round := 0
+	m := newTestMonitor(newFakeClient(t), Target{Name: "kvm-home", Type: "kvm", Checker: checkFunc(
+		func(context.Context) []provider.TargetStatus { return rounds[round] })})
+
+	for round = range rounds {
+		m.checkOnce(ctx)
+	}
+
+	mu.Lock()
+	defer mu.Unlock()
+	want := [][]string{
+		{`"msg"="WARNING: provider target not ready"`, `"reason"="Unreachable"`},
+		{`"msg"="WARNING: provider target not ready"`, `"reason"="Timeout"`},
+		{`"msg"="Provider target ready"`},
+	}
+	if len(lines) != len(want) {
+		t.Fatalf("logged %d lines, want %d:\n%s", len(lines), len(want), strings.Join(lines, "\n"))
+	}
+	for i, subs := range want {
+		for _, sub := range subs {
+			if !strings.Contains(lines[i], sub) {
+				t.Errorf("line %d missing %s:\n%s", i, sub, lines[i])
+			}
+		}
+	}
+}
+
+func TestMonitor_cleanup_deletesOnlyLabelledUnconfigured(t *testing.T) {
+	t.Parallel()
+	foreign := &crawlv1alpha1.ProviderHealth{ObjectMeta: metav1.ObjectMeta{Name: "foreign"}}
+	c := newFakeClient(t, managed("kvm-home"), managed("retired"), foreign)
+	m := newTestMonitor(c)
+
+	m.cleanup(context.Background(), map[string]bool{"kvm-home": true})
+
+	getHealth(t, c, "kvm-home")
+	getHealth(t, c, "foreign")
+	var gone crawlv1alpha1.ProviderHealth
+	if err := c.Get(context.Background(), client.ObjectKey{Name: "retired"}, &gone); !apierrors.IsNotFound(err) {
+		t.Errorf("Get retired = %v, want NotFound", err)
+	}
+}
+
+func TestMonitor_Start_offCleansUpAndReturns(t *testing.T) {
+	t.Parallel()
+	c := newFakeClient(t, managed("kvm-home"))
+	var checked atomic.Bool
+	m := newTestMonitor(c, Target{Name: "kvm-home", Type: "kvm", Checker: checkFunc(
+		func(context.Context) []provider.TargetStatus { checked.Store(true); return nil })})
+	m.Interval = 0
+
+	errc := make(chan error, 1)
+	go func() { errc <- m.Start(context.Background()) }()
+	select {
+	case err := <-errc:
+		if err != nil {
+			t.Fatalf("Start() = %v, want nil", err)
+		}
+	case <-time.After(5 * time.Second):
+		t.Fatal("Start did not return with checking off")
+	}
+
+	if checked.Load() {
+		t.Error("a check ran with checking off")
+	}
+	var gone crawlv1alpha1.ProviderHealth
+	if err := c.Get(context.Background(), client.ObjectKey{Name: "kvm-home"}, &gone); !apierrors.IsNotFound(err) {
+		t.Errorf("Get kvm-home = %v, want NotFound so no stale Ready is left behind", err)
+	}
+}
+
+func TestMonitor_Start_checksAtOnceAndStops(t *testing.T) {
+	t.Parallel()
+	c := newFakeClient(t)
+	m := newTestMonitor(c, Target{Name: "local", Type: "kubernetes", Checker: verdicts(up("apiserver"))})
+	m.Interval = time.Hour
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+
+	errc := make(chan error, 1)
+	go func() { errc <- m.Start(ctx) }()
+	deadline := time.Now().Add(5 * time.Second)
+	for len(m.Snapshot()) == 0 {
+		if time.Now().After(deadline) {
+			t.Fatal("no check within 5s of Start, want one right away")
+		}
+		time.Sleep(10 * time.Millisecond)
+	}
+	cancel()
+
+	select {
+	case err := <-errc:
+		if err != nil {
+			t.Errorf("Start() = %v, want nil", err)
+		}
+	case <-time.After(5 * time.Second):
+		t.Fatal("Start did not return after its context ended")
+	}
+}
+
+func TestValidateSchedule(t *testing.T) {
+	t.Parallel()
+	tests := []struct {
+		name              string
+		interval, timeout time.Duration
+		wantErr           bool
+	}{
+		{"off", 0, 0, false},
+		{"defaults", time.Minute, 15 * time.Second, false},
+		{"shortest interval", 10 * time.Second, 10 * time.Second, false},
+		{"too frequent", 5 * time.Second, time.Second, true},
+		{"negative interval", -time.Minute, time.Second, true},
+		{"timeout left to derive, short interval", 12 * time.Second, 0, false},
+		{"negative timeout", time.Minute, -time.Second, true},
+		{"timeout longer than interval", time.Minute, 2 * time.Minute, true},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			if err := ValidateSchedule(tc.interval, tc.timeout); (err != nil) != tc.wantErr {
+				t.Errorf("ValidateSchedule(%s, %s) = %v, want error %v", tc.interval, tc.timeout, err, tc.wantErr)
+			}
+		})
+	}
+}
+
+func TestMonitor_setDefaults_timeout(t *testing.T) {
+	t.Parallel()
+	tests := []struct {
+		name                    string
+		interval, timeout, want time.Duration
+	}{
+		{"default", time.Minute, 0, 15 * time.Second},
+		{"interval shorter than the default", 12 * time.Second, 0, 12 * time.Second},
+		{"explicit", time.Minute, 20 * time.Second, 20 * time.Second},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			m := &Monitor{Interval: tc.interval, Timeout: tc.timeout}
+			m.setDefaults()
+			if m.Timeout != tc.want {
+				t.Errorf("Timeout = %s, want %s", m.Timeout, tc.want)
+			}
+		})
+	}
+}

internal/providerhealth/status.go

@@ -0,0 +1,98 @@
+package providerhealth
+
+import (
+	"fmt"
+	"slices"
+	"strings"
+	"unicode/utf8"
+
+	apimeta "k8s.io/apimachinery/pkg/api/meta"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	crawlv1alpha1 "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/api/v1alpha1"
+)
+
+// Reasons on the Ready condition.
+const (
+	reasonAllTargetsReady      = "AllTargetsReady"
+	reasonTargetsNotReady      = "TargetsNotReady"
+	reasonChecksNotImplemented = "ChecksNotImplemented"
+	reasonNoTargets            = "NoTargets"
+)
+
+// maxMessageLen matches the CRD's limit on a target's message.
+const maxMessageLen = 1024
+
+// buildStatus derives a ProviderHealth status from the previous one and a
+// fresh result. Every timestamp comes from r.CheckedAt, and prev is left
+// untouched. A target keeps its lastTransitionTime while ready does not
+// flip, which is how several hosts each show when they went down.
+func buildStatus(prev crawlv1alpha1.ProviderHealthStatus, r Result) crawlv1alpha1.ProviderHealthStatus {
+	now := metav1.NewTime(r.CheckedAt)
+	out := crawlv1alpha1.ProviderHealthStatus{
+		Type:          r.Type,
+		Conditions:    slices.Clone(prev.Conditions),
+		LastCheckTime: &now,
+	}
+	before := make(map[string]crawlv1alpha1.ProviderTarget, len(prev.Targets))
+	for _, t := range prev.Targets {
+		before[t.Name] = t
+	}
+	ready := 0
+	var failing []string
+	for _, ts := range r.Targets {
+		t := crawlv1alpha1.ProviderTarget{
+			Name:               ts.Name,
+			Address:            ts.Address,
+			Ready:              ts.Ready,
+			Reason:             ts.Reason,
+			Message:            truncate(ts.Message, maxMessageLen),
+			LatencyMillis:      ts.Latency.Milliseconds(),
+			LastTransitionTime: now,
+			Details:            ts.Details,
+		}
+		if b, ok := before[ts.Name]; ok && b.Ready == ts.Ready {
+			t.LastTransitionTime = b.LastTransitionTime
+		}
+		if ts.Ready {
+			ready++
+		} else {
+			failing = append(failing, ts.Name+": "+ts.Reason)
+		}
+		out.Targets = append(out.Targets, t)
+	}
+	if r.Supported {
+		out.TargetsReady = fmt.Sprintf("%d/%d", ready, len(r.Targets))
+	}
+
+	cond := metav1.Condition{Type: crawlv1alpha1.ConditionReady, LastTransitionTime: now}
+	switch {
+	case !r.Supported:
+		cond.Status = metav1.ConditionUnknown
+		cond.Reason = reasonChecksNotImplemented
+		cond.Message = fmt.Sprintf("%s providers have no health check", r.Type)
+	case len(r.Targets) == 0:
+		cond.Status = metav1.ConditionUnknown
+		cond.Reason = reasonNoTargets
+		cond.Message = "the provider has nothing to check"
+	case len(failing) == 0:
+		cond.Status = metav1.ConditionTrue
+		cond.Reason = reasonAllTargetsReady
+		cond.Message = fmt.Sprintf("%d of %d targets ready", ready, len(r.Targets))
+	default:
+		cond.Status = metav1.ConditionFalse
+		cond.Reason = reasonTargetsNotReady
+		cond.Message = truncate(strings.Join(failing, "; "), maxMessageLen)
+	}
+	apimeta.SetStatusCondition(&out.Conditions, cond)
+	return out
+}
+
+// truncate cuts s to at most n characters, which is how the API server
+// counts maxLength.
+func truncate(s string, n int) string {
+	if utf8.RuneCountInString(s) <= n {
+		return s
+	}
+	return string([]rune(s)[:n-1]) + "…"
+}

internal/providerhealth/status_test.go

@@ -0,0 +1,135 @@
+package providerhealth
+
+import (
+	"strings"
+	"testing"
+	"time"
+	"unicode/utf8"
+
+	apimeta "k8s.io/apimachinery/pkg/api/meta"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	crawlv1alpha1 "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/api/v1alpha1"
+	"gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider"
+)
+
+func readyAt(name string, at time.Time) crawlv1alpha1.ProviderTarget {
+	return crawlv1alpha1.ProviderTarget{Name: name, Ready: true, Reason: provider.ReasonReachable, LastTransitionTime: metav1.NewTime(at)}
+}
+
+func TestBuildStatus(t *testing.T) {
+	t.Parallel()
+	tests := []struct {
+		name             string
+		prev             crawlv1alpha1.ProviderHealthStatus
+		result           Result
+		wantStatus       metav1.ConditionStatus
+		wantReason       string
+		wantMessage      string
+		wantTargetsReady string
+		// wantTransitions is each target's lastTransitionTime, in order.
+		wantTransitions []time.Time
+	}{
+		{
+			name:             "all ready",
+			result:           Result{Type: "kvm", Supported: true, Targets: []provider.TargetStatus{up("nas1"), up("nas2")}, CheckedAt: t1},
+			wantStatus:       metav1.ConditionTrue,
+			wantReason:       reasonAllTargetsReady,
+			wantMessage:      "2 of 2 targets ready",
+			wantTargetsReady: "2/2",
+			wantTransitions:  []time.Time{t1, t1},
+		},
+		{
+			name:             "one target down",
+			result:           Result{Type: "kvm", Supported: true, Targets: []provider.TargetStatus{up("nas1"), down("nas2", provider.ReasonUnreachable)}, CheckedAt: t1},
+			wantStatus:       metav1.ConditionFalse,
+			wantReason:       reasonTargetsNotReady,
+			wantMessage:      "nas2: Unreachable",
+			wantTargetsReady: "1/2",
+			wantTransitions:  []time.Time{t1, t1},
+		},
+		{
+			name: "only a flip moves a target's transition time",
+			prev: crawlv1alpha1.ProviderHealthStatus{Targets: []crawlv1alpha1.ProviderTarget{readyAt("nas1", t0), readyAt("nas2", t0)}},
+			result: Result{Type: "kvm", Supported: true, CheckedAt: t1,
+				Targets: []provider.TargetStatus{up("nas1"), down("nas2", provider.ReasonTimeout)}},
+			wantStatus:       metav1.ConditionFalse,
+			wantReason:       reasonTargetsNotReady,
+			wantMessage:      "nas2: Timeout",
+			wantTargetsReady: "1/2",
+			wantTransitions:  []time.Time{t0, t1},
+		},
+		{
+			name:       "no checker",
+			result:     Result{Type: "gcp", CheckedAt: t1},
+			wantStatus: metav1.ConditionUnknown,
+			wantReason: reasonChecksNotImplemented,
+		},
+		{
+			name:             "nothing to check",
+			result:           Result{Type: "kvm", Supported: true, CheckedAt: t1},
+			wantStatus:       metav1.ConditionUnknown,
+			wantReason:       reasonNoTargets,
+			wantTargetsReady: "0/0",
+		},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			got := buildStatus(tc.prev, tc.result)
+
+			cond := apimeta.FindStatusCondition(got.Conditions, crawlv1alpha1.ConditionReady)
+			if cond == nil || cond.Status != tc.wantStatus || cond.Reason != tc.wantReason {
+				t.Fatalf("Ready = %+v, want %s/%s", cond, tc.wantStatus, tc.wantReason)
+			}
+			if tc.wantMessage != "" && cond.Message != tc.wantMessage {
+				t.Errorf("Ready message = %q, want %q", cond.Message, tc.wantMessage)
+			}
+			if got.TargetsReady != tc.wantTargetsReady {
+				t.Errorf("targetsReady = %q, want %q", got.TargetsReady, tc.wantTargetsReady)
+			}
+			if got.Type != tc.result.Type || got.LastCheckTime == nil || !got.LastCheckTime.Time.Equal(t1) {
+				t.Errorf("type/lastCheckTime = %q/%v, want %q/%s", got.Type, got.LastCheckTime, tc.result.Type, t1)
+			}
+			if len(got.Targets) != len(tc.wantTransitions) {
+				t.Fatalf("got %d targets, want %d", len(got.Targets), len(tc.wantTransitions))
+			}
+			for i, want := range tc.wantTransitions {
+				if lt := got.Targets[i].LastTransitionTime.Time; !lt.Equal(want) {
+					t.Errorf("target %s lastTransitionTime = %s, want %s", got.Targets[i].Name, lt, want)
+				}
+			}
+		})
+	}
+}
+
+func TestBuildStatus_conditionKeepsTransitionWhileUnchanged(t *testing.T) {
+	t.Parallel()
+	first := buildStatus(crawlv1alpha1.ProviderHealthStatus{},
+		Result{Type: "kvm", Supported: true, Targets: []provider.TargetStatus{up("nas1")}, CheckedAt: t0})
+	firstCond := apimeta.FindStatusCondition(first.Conditions, crawlv1alpha1.ConditionReady)
+
+	second := buildStatus(first, Result{Type: "kvm", Supported: true, Targets: []provider.TargetStatus{up("nas1")}, CheckedAt: t1})
+
+	cond := apimeta.FindStatusCondition(second.Conditions, crawlv1alpha1.ConditionReady)
+	if !cond.LastTransitionTime.Time.Equal(t0) {
+		t.Errorf("Ready lastTransitionTime = %s, want %s from the first round", cond.LastTransitionTime, t0)
+	}
+	if !firstCond.LastTransitionTime.Time.Equal(t0) || len(first.Conditions) != 1 {
+		t.Errorf("buildStatus modified the previous status: %+v", first.Conditions)
+	}
+}
+
+func TestBuildStatus_truncatesLongMessages(t *testing.T) {
+	t.Parallel()
+	long := strings.Repeat("é", 3000)
+	got := buildStatus(crawlv1alpha1.ProviderHealthStatus{}, Result{Type: "kvm", Supported: true, CheckedAt: t1,
+		Targets: []provider.TargetStatus{{Name: "nas1", Reason: provider.ReasonUnreachable, Message: long}}})
+
+	if n := utf8.RuneCountInString(got.Targets[0].Message); n != maxMessageLen {
+		t.Errorf("message has %d characters, want %d", n, maxMessageLen)
+	}
+	if !strings.HasSuffix(got.Targets[0].Message, "…") {
+		t.Error("truncated message does not end with an ellipsis")
+	}
+}

test/e2e/e2e_test.go

@@ -173,6 +173,20 @@ var _ = Describe("Manager", Ordered, func() {
 			Eventually(verifyControllerUp).Should(Succeed())
 		})
 
+		// Runs before the metrics spec on purpose: once a ProviderHealth is
+		// Ready, the first round of checks is done and its gauges exist.
+		It("should publish provider health", func() {
+			By("waiting for the kubernetes provider's ProviderHealth to report Ready")
+			verifyProviderReady := func(g Gomega) {
+				cmd := exec.Command("kubectl", "get", "providerhealth", "kubernetes",
+					"-o", `jsonpath={.status.conditions[?(@.type=="Ready")].status}`)
+				output, err := utils.Run(cmd)
+				g.Expect(err).NotTo(HaveOccurred())
+				g.Expect(output).To(Equal("True"), "kubernetes provider not Ready")
+			}
+			Eventually(verifyProviderReady).Should(Succeed())
+		})
+
 		It("should ensure the metrics endpoint is serving metrics", func() {
 			By("creating a ClusterRoleBinding for the service account to allow access to metrics")
 			cmd := exec.Command("kubectl", "create", "clusterrolebinding", metricsRoleBindingName,
@@ -264,6 +278,8 @@ var _ = Describe("Manager", Ordered, func() {
 				g.Expect(err).NotTo(HaveOccurred(), "Failed to retrieve logs from curl pod")
 				g.Expect(metricsOutput).NotTo(BeEmpty())
 				g.Expect(metricsOutput).To(ContainSubstring("< HTTP/1.1 200 OK"))
+				g.Expect(metricsOutput).To(ContainSubstring(
+					`proxy_operator_provider_target_up{provider="kubernetes",target="apiserver"} 1`))
 			}
 			Eventually(verifyMetricsAvailable, 2*time.Minute).Should(Succeed())
 		})