Provider health checks: ProviderHealth CRD, per-provider checks and gauges #6

Merged
kacerr merged 12 commits from feat/provider-health into main 2026-09-16 10:46:47 +02:00
Showing only changes of commit 80b2e9c85e - Show all commits
@@ -0,0 +1,119 @@
# Plan: Provider blueprint (docs/providers.md)
**Created:** 2026-09-16 08:41
## Context
The provider contract is spread across `docs/prompts/__initial-prompt.md` §5,
the comments in `internal/provider/{provider,errors,check}.go`, `docs/architecture.md`
§5 and §11, `docs/kvm-provider.md`, and plan Step 2/3/8. Many rules exist only in
code: providerID must be self-contained, UID/CreatedAt must be read back, Create
must survive a crash partway through, calls must stop when ctx is cancelled,
decorators are wired in `cmd/main.go`, and so on. With "~5 providers soon", the
next provider author needs one blueprint. It should say what a provider MUST,
SHOULD or MAY do, show which features each existing provider (kubernetes, gcp,
kvm) actually implements (with file:line evidence), give a checklist for adding
one, and list the gaps.
Decisions (user): store in `docs/providers.md`; scope = current state plus a
separate "Gaps and proposed features" section (nothing implemented); wait for
`feat/provider-health` to merge, then branch off `main`.
## Step 0 — Save the plan; wait for the merge
- Right after ExitPlanMode, copy this plan to `docs/plans/2026-09-16-0841-provider-blueprint.md`
and commit it on its own. The current branch is `feat/provider-health`, so the
plan reaches `main` with that merge. CLAUDE.md requires the plan commit to
come before any branching.
- Stop and wait for the user to confirm that `feat/provider-health` is merged.
Then `git switch main && git pull`, and `git switch -c feat/provider-blueprint`.
## Step 1 — Collect evidence (read-only, in parallel)
Run 3 Explore agents at the same time, one each for `internal/provider/kubernetes`,
`internal/provider/gcp` and `internal/provider/kvm`, including their tests. Each
answers the **same questionnaire**, citing `file:line` for every answer and writing
"not implemented" explicitly when that's the case:
1. **Identity**: providerID format and parsing; what it carries (zone, host, namespace).
2. **Create**: how it stays idempotent (name lookup, 409, DomainLookupByName);
whether it returns before the VM is up or waits; what happens when an earlier
attempt crashed partway; which Placement fields are used or ignored; how
cloud-init is delivered; how the port is used; how GC labels/UID are attached.
3. **Get**: native state → InstanceState mapping; IP discovery rules; UID and
CreatedAt readback; how NotFound is detected.
4. **Delete**: idempotency; what gets removed (disks, volumes, seeds); returns
before or after the resource is gone; unknown/malformed providerID behaviour.
5. **ListByTag**: scope (namespaces, zones, hosts); partial-failure behaviour; how
managed resources are recognised.
6. **Error classification**: table of backend error → NotFound/Quota/Transient/
Permanent; whether quota is detected at all.
7. **Health check** (`Checker`): targets, provider-specific Reasons, Details keys.
8. **Config**: YAML section, defaults, validation (`internal/provider/config*.go`),
credentials/Secrets, which settings are read once per process.
9. **Connections and cancellation**: pooling, reconnects, timeouts, ctx honouring.
10. **Observability**: log keys and verbosity levels, spans (`kvm/trace.go`, gcp
`wirelog.go`), anything beyond the shared metrics/tracing decorators.
11. **Testing**: fakes, envtest/kind, `//go:build integration` tests, what is
not covered.
12. **Known gaps**: TODOs, stale comments, documented limitations.
Meanwhile, read the shared layer myself: `internal/provider/{provider,errors,check,
name,config,metrics,tracing}.go`, `internal/provider/registry/registry.go`,
`cmd/main.go` (registry wiring, decorator order, Checker collection), and the
places that consume the contract, so every MUST rule traces to a caller that depends on it:
`internal/controller` (error-class branching, ErrNotFound → replace/adopt),
`internal/gc` (UID, CreatedAt age skip, ListByTag errors),
`internal/providerhealth`. Then cross-check against the existing docs listed in Context.
## Step 2 — Write `docs/providers.md`
Sections:
1. **What a provider is**: a few sentences plus links to architecture.md §5/§11.
2. **The contract**: per method (Create/Get/Delete/ListByTag), MUST/SHOULD/MAY
rules, each followed by *why*: the reconciler or GC behaviour it protects,
linked to code.
3. **Error taxonomy**: what each class makes the reconciler do; how to map backend
errors; "unclassified = transient" pitfall.
4. **Identity and GC labels**: providerID guidance, `LabelManaged`/`LabelUID`,
CreatedAt, deterministic `NameFromUID`.
5. **Optional interfaces**: `Checker` (shared reasons, never fails as a whole,
collected before decorators).
6. **Feature matrix**: one row per questionnaire item, columns kubernetes/gcp/kvm,
cells `yes`/`partial`/`no` plus a short note. Put the file:line evidence in a
compact per-provider notes subsection under the table, not in the cells.
7. **Adding a provider: checklist**: package layout, config section and validation,
registry constructor, `cmd/main.go` wiring, Checker, error mapping with tests,
`name`/label tests, integration test behind a build tag, provider doc page,
architecture.md rows, demo script (optional).
8. **Gaps and proposed features**: current-state gaps per provider (e.g. no kvm
quota detection, orphan volumes invisible to GC, Secret rotation needs a
restart, ListByTag partial-failure inconsistency, stale `mock` references in
`provider.go` comments), plus proposals that are clearly marked as not
implemented. Only list them; don't design them.
Also: link it from `docs/architecture.md` (Components table → provider contract row)
and from the CLAUDE.md "Architecture" section. Leave per-provider docs where they are.
Follow the plan-execution convention: create `docs/plans-executions/2026-09-16-0841-provider-blueprint.md`
with the status checklist and step entries, and commit it with each step.
## Step 3 — Verify, then open the MR
- A verification agent (Explore) re-checks each matrix cell and each MUST rule
against the cited file:line and reports any mismatch. Fix what it finds.
- Check that every relative link in the new and edited docs resolves (grep the
`](` targets and `ls` each one).
- `go build ./... && go test -short ./...` as a sanity check (docs-only change,
expected no-op).
- Push `feat/provider-blueprint` and open the MR against `main` via the Gitea API,
following the project's push/MR memory note. After the user confirms, add a CHANGELOG.md entry.
## Critical files
- New: `docs/providers.md`, `docs/plans/2026-09-16-0841-provider-blueprint.md`,
`docs/plans-executions/2026-09-16-0841-provider-blueprint.md`
- Edited: `docs/architecture.md` (link), `CLAUDE.md` (link)
- Read-only sources: `internal/provider/**`, `cmd/main.go`, `internal/controller`,
`internal/gc`, `internal/providerhealth`, `docs/prompts/__initial-prompt.md`,
`docs/kvm-provider.md`, `docs/gcp-vm-validation.md`