kvm2: user guide, appendix column and section, architecture and README entries
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5.1
+90
-1
@@ -10,6 +10,7 @@
|
||||
| kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) |
|
||||
| gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
|
||||
| kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds |
|
||||
| kvm2 provider | `internal/provider/kvm2` | library | — | the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial |
|
||||
| Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events |
|
||||
| Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
|
||||
| Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
|
||||
@@ -168,6 +169,39 @@ providerID = hosts/<host>/domains/<name>; GC identity lives in the domain's
|
||||
<metadata> element (libvirt has no labels). All RPCs run under a context-
|
||||
bounded call: go-libvirt never times out on its own, so a cancelled reconcile
|
||||
closes the SSH connection to unblock it.
|
||||
|
||||
kvm2 provider (internal/provider/kvm2/) - the same transport, a second type built from the contract
|
||||
prov.Create ──► placement checked against the catalogue (Permanent before any dial)
|
||||
DomainLookupByName: exists → uid guard (foreign uid = Permanent), ─┐ every step
|
||||
start if needed, wait for running, done │ idempotent,
|
||||
else: StoragePoolGetInfo precheck: image (if uncached) + diskSizeGb │ under a
|
||||
+ 16 MiB must fit, else ErrQuotaExceeded (reconciler │ per-name lock
|
||||
waits 5m) │ shared with
|
||||
ensure base image: marker present = cached; otherwise GET │ Delete
|
||||
streamed into StorageVolUpload on a DEDICATED connection, │
|
||||
StoragePoolRefresh, marker written last │
|
||||
overlay volume (exists = reuse) │
|
||||
seed ISO (always rebuilt) │
|
||||
DomainDefineXML (<metadata> in kvm2's own namespace: │
|
||||
managed, uid, createdAt) + DomainCreate, poll until │
|
||||
running (2m cap) ─┘
|
||||
prov.Get ──► DomainLookupByName + DomainGetMetadata + DomainGetState
|
||||
+ DomainInterfaceAddresses (guest agent, then ARP; first routable IPv4)
|
||||
running without IP = Provisioning; nostate = Provisioning;
|
||||
only ErrNoDomain is NotFound
|
||||
prov.Delete ──► destroy (not running = fine) → undefine (nvram, managed save, snapshots)
|
||||
→ delete overlay + seed by name; unconfigured host = logged no-op
|
||||
prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose kvm2
|
||||
<metadata> says managed=true; one host down fails the sweep,
|
||||
one unreadable domain is skipped
|
||||
Check ──► per host, on a FRESH connection: version, hostname, pool lookup,
|
||||
pool info, close; never the pooled one
|
||||
providerID = hosts/<host>/domains/<name>. Every RPC has its own budget (dial 10s,
|
||||
RPC 30s, DomainCreate 90s, seed upload 60s, image transfer 20m): a call that
|
||||
outlives its ctx or budget kills the pooled connection (the only way to unblock
|
||||
go-libvirt), so other calls in flight on that host fail once with Transient and
|
||||
retry; the next call redials. Each RPC is a child span kvm2.<Method> under
|
||||
provider.*, plus kvm2.dial and kvm2.download.
|
||||
```
|
||||
|
||||
The reconciler never watches provider-side resources (Pods or GCP VMs).
|
||||
@@ -343,6 +377,15 @@ incoming trace context):
|
||||
- `provider.create|get|delete|list` (client spans, `provider.WithTracing`
|
||||
decorator wired outermost around `WithMetrics`); the GCP SDK's own
|
||||
otelhttp transport contributes HTTP child spans automatically.
|
||||
- kvm2 child spans: under `provider.*`, one `kvm2.<Method>` client span per
|
||||
libvirt RPC (for example `kvm2.DomainDefineXML`, attributes `kvm2.host`,
|
||||
`kvm2.domain`, `kvm2.pool`, `kvm2.volume`, `kvm2.libvirt_code` on
|
||||
failure), `kvm2.dial` around the SSH handshake and libvirt connect, and
|
||||
`kvm2.download` around an image fetch; under `providerhealth.check`,
|
||||
`kvm2.dial` and the four `kvm2.<Method>` spans of a check. They come
|
||||
from a span decorator over the seam, applied once in the dialer, so no
|
||||
RPC is unspanned. The `libvirt call` log line of an RPC is written
|
||||
inside its span. The `kvm` provider has no child spans.
|
||||
- Kubernetes API calls become client child spans via a wrapped
|
||||
`rest.Config` transport — gated on an existing parent span, so informer
|
||||
list/watch long-polls and leader-election renewals never create root
|
||||
@@ -368,11 +411,12 @@ the slog handler's ctx.
|
||||
Answers "can this cluster reach each provider's backend right now?"
|
||||
without waiting for a Proxy to fail. A provider opts in by implementing
|
||||
`provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per
|
||||
target and never fails as a whole. All three providers do:
|
||||
target and never fails as a whole. All four providers do:
|
||||
|
||||
| Provider | Target | What one check does |
|
||||
|---|---|---|
|
||||
| kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) |
|
||||
| kvm2 | each `kvm2.hosts` entry | on a fresh SSH+libvirt connection dialled for the check and closed after it: libvirt version and hostname, then the configured pool must exist and be running; `details` carry `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB` and `poolAvailableGiB` ([details](kvm2-provider.md#health-check)) |
|
||||
| kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on |
|
||||
| gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` |
|
||||
|
||||
@@ -613,3 +657,48 @@ they read as choices, not accidents. Chronological by build step.
|
||||
- **A provider name that is not a valid object name** gets metrics but no
|
||||
`ProviderHealth` object, with a warning at start. Tightening config
|
||||
validation instead would crash-loop existing deployments on upgrade.
|
||||
- **kvm2 is a second provider type built clean-room from the contract,**
|
||||
not a rewrite of `kvm` in place. Its spec
|
||||
(`docs/kvm2-provider-spec.md`) and code were written from
|
||||
[provider-contract.md](provider-contract.md), the kvm user guide and
|
||||
go-libvirt alone, never from `internal/provider/kvm/`, as a test of
|
||||
whether the contract is enough to build a provider. Both types ship,
|
||||
share the `KVMConfig` block shape and the `kvm-ssh` Secret, and may
|
||||
share a host.
|
||||
- **kvm2 image downloads run on a dedicated connection.** A cancelled or
|
||||
overlong call kills the connection it ran on (the only way to unblock
|
||||
go-libvirt), so a 435 MB upload on the pooled connection could die
|
||||
because a poll for another Proxy was cancelled. The transfer dials its
|
||||
own connection and closes it afterwards; the cheap checks and the marker
|
||||
stay on the pooled one.
|
||||
- **kvm2 health checks dial fresh connections**, the opposite of the kvm
|
||||
decision above. With credentials re-read on every dial a fresh
|
||||
connection does pick up a rotated Secret, and a check that runs out of
|
||||
time then closes only its own connection, so it cannot disrupt a
|
||||
reconcile. The cost is one sshd session per host per round.
|
||||
- **A cancelled kvm2 call kills the pooled connection.** Every RPC has a
|
||||
budget; when a call outlives its ctx or budget the adapter closes the
|
||||
socket and the SSH client, returns Transient, and marks the connection
|
||||
dead so the next call redials. Other calls in flight on that host fail
|
||||
once with Transient and their callers retry. Documented in the guide
|
||||
rather than avoided: the alternative, leaving a hung RPC blocked
|
||||
forever, pins a worker.
|
||||
- **kvm2 marks domains in its own metadata namespace**, so it never lists
|
||||
or adopts domains the `kvm` provider made. Switching an entry from `kvm`
|
||||
to `kvm2` means deleting its Proxies first; the two types can share a
|
||||
host without one collecting the other's VMs.
|
||||
- **kvm2 turns pool free space into the quota signal.** libvirt reports a
|
||||
full pool as an ordinary error, which would retry on the fast backoff.
|
||||
Before creating any volume, `Create` compares `StoragePoolGetInfo`'s
|
||||
available bytes with the image (if uncached) plus `diskSizeGb` plus 16
|
||||
MiB for the seed, and returns `ErrQuotaExceeded` below that, so the
|
||||
reconciler waits 5 minutes instead.
|
||||
- **kvm2 classifies SSH authentication and host-key failures as
|
||||
Permanent.** A key missing from `authorized_keys` or a host key missing
|
||||
from or different in known_hosts needs an admin; kvm retried these
|
||||
forever. A missing key or known_hosts file stays Transient, because the
|
||||
optional Secret may land later.
|
||||
- **kvm2 re-reads the SSH key and known_hosts on every dial**, so rotating
|
||||
the Secret needs no restart; nothing is cached. The constructor only
|
||||
checks that the files exist, to warn early without failing the provider
|
||||
set.
|
||||
|
||||
Reference in New Issue
Block a user