kvm2: user guide, appendix column and section, architecture and README entries
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5.1
+90
-1
@@ -10,6 +10,7 @@
|
||||
| kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) |
|
||||
| gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops |
|
||||
| kvm provider | `internal/provider/kvm` | library | — | VMs on libvirt hosts over SSH-forwarded sockets; image cache, overlay disks, NoCloud seeds |
|
||||
| kvm2 provider | `internal/provider/kvm2` | library | — | the same backend, built clean-room from the provider contract as a second type: per-RPC child spans, pool-space quota, per-call budgets, its own metadata namespace, credentials re-read on every dial |
|
||||
| Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events |
|
||||
| Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex |
|
||||
| Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` |
|
||||
@@ -168,6 +169,39 @@ providerID = hosts/<host>/domains/<name>; GC identity lives in the domain's
|
||||
<metadata> element (libvirt has no labels). All RPCs run under a context-
|
||||
bounded call: go-libvirt never times out on its own, so a cancelled reconcile
|
||||
closes the SSH connection to unblock it.
|
||||
|
||||
kvm2 provider (internal/provider/kvm2/) - the same transport, a second type built from the contract
|
||||
prov.Create ──► placement checked against the catalogue (Permanent before any dial)
|
||||
DomainLookupByName: exists → uid guard (foreign uid = Permanent), ─┐ every step
|
||||
start if needed, wait for running, done │ idempotent,
|
||||
else: StoragePoolGetInfo precheck: image (if uncached) + diskSizeGb │ under a
|
||||
+ 16 MiB must fit, else ErrQuotaExceeded (reconciler │ per-name lock
|
||||
waits 5m) │ shared with
|
||||
ensure base image: marker present = cached; otherwise GET │ Delete
|
||||
streamed into StorageVolUpload on a DEDICATED connection, │
|
||||
StoragePoolRefresh, marker written last │
|
||||
overlay volume (exists = reuse) │
|
||||
seed ISO (always rebuilt) │
|
||||
DomainDefineXML (<metadata> in kvm2's own namespace: │
|
||||
managed, uid, createdAt) + DomainCreate, poll until │
|
||||
running (2m cap) ─┘
|
||||
prov.Get ──► DomainLookupByName + DomainGetMetadata + DomainGetState
|
||||
+ DomainInterfaceAddresses (guest agent, then ARP; first routable IPv4)
|
||||
running without IP = Provisioning; nostate = Provisioning;
|
||||
only ErrNoDomain is NotFound
|
||||
prov.Delete ──► destroy (not running = fine) → undefine (nvram, managed save, snapshots)
|
||||
→ delete overlay + seed by name; unconfigured host = logged no-op
|
||||
prov.ListByTag ─► ConnectListAllDomains on every host, keep domains whose kvm2
|
||||
<metadata> says managed=true; one host down fails the sweep,
|
||||
one unreadable domain is skipped
|
||||
Check ──► per host, on a FRESH connection: version, hostname, pool lookup,
|
||||
pool info, close; never the pooled one
|
||||
providerID = hosts/<host>/domains/<name>. Every RPC has its own budget (dial 10s,
|
||||
RPC 30s, DomainCreate 90s, seed upload 60s, image transfer 20m): a call that
|
||||
outlives its ctx or budget kills the pooled connection (the only way to unblock
|
||||
go-libvirt), so other calls in flight on that host fail once with Transient and
|
||||
retry; the next call redials. Each RPC is a child span kvm2.<Method> under
|
||||
provider.*, plus kvm2.dial and kvm2.download.
|
||||
```
|
||||
|
||||
The reconciler never watches provider-side resources (Pods or GCP VMs).
|
||||
@@ -343,6 +377,15 @@ incoming trace context):
|
||||
- `provider.create|get|delete|list` (client spans, `provider.WithTracing`
|
||||
decorator wired outermost around `WithMetrics`); the GCP SDK's own
|
||||
otelhttp transport contributes HTTP child spans automatically.
|
||||
- kvm2 child spans: under `provider.*`, one `kvm2.<Method>` client span per
|
||||
libvirt RPC (for example `kvm2.DomainDefineXML`, attributes `kvm2.host`,
|
||||
`kvm2.domain`, `kvm2.pool`, `kvm2.volume`, `kvm2.libvirt_code` on
|
||||
failure), `kvm2.dial` around the SSH handshake and libvirt connect, and
|
||||
`kvm2.download` around an image fetch; under `providerhealth.check`,
|
||||
`kvm2.dial` and the four `kvm2.<Method>` spans of a check. They come
|
||||
from a span decorator over the seam, applied once in the dialer, so no
|
||||
RPC is unspanned. The `libvirt call` log line of an RPC is written
|
||||
inside its span. The `kvm` provider has no child spans.
|
||||
- Kubernetes API calls become client child spans via a wrapped
|
||||
`rest.Config` transport — gated on an existing parent span, so informer
|
||||
list/watch long-polls and leader-election renewals never create root
|
||||
@@ -368,11 +411,12 @@ the slog handler's ctx.
|
||||
Answers "can this cluster reach each provider's backend right now?"
|
||||
without waiting for a Proxy to fail. A provider opts in by implementing
|
||||
`provider.Checker`: `Check(ctx) []TargetStatus` returns one verdict per
|
||||
target and never fails as a whole. All three providers do:
|
||||
target and never fails as a whole. All four providers do:
|
||||
|
||||
| Provider | Target | What one check does |
|
||||
|---|---|---|
|
||||
| kvm | each `kvm.hosts` entry | over the pooled SSH+libvirt connection: libvirt version and hostname, then the configured pool must exist and be running; reports pool capacity and free space ([details](kvm-provider.md#health-check)) |
|
||||
| kvm2 | each `kvm2.hosts` entry | on a fresh SSH+libvirt connection dialled for the check and closed after it: libvirt version and hostname, then the configured pool must exist and be running; `details` carry `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB` and `poolAvailableGiB` ([details](kvm2-provider.md#health-check)) |
|
||||
| kubernetes | `apiserver` | lists at most one managed Pod cluster-wide: the verb and scope `ListByTag` and GC rely on |
|
||||
| gcp | the project | one page of an aggregated instance list with `maxResults=1`: credentials, Compute API enabled, `compute.instances.list` |
|
||||
|
||||
@@ -613,3 +657,48 @@ they read as choices, not accidents. Chronological by build step.
|
||||
- **A provider name that is not a valid object name** gets metrics but no
|
||||
`ProviderHealth` object, with a warning at start. Tightening config
|
||||
validation instead would crash-loop existing deployments on upgrade.
|
||||
- **kvm2 is a second provider type built clean-room from the contract,**
|
||||
not a rewrite of `kvm` in place. Its spec
|
||||
(`docs/kvm2-provider-spec.md`) and code were written from
|
||||
[provider-contract.md](provider-contract.md), the kvm user guide and
|
||||
go-libvirt alone, never from `internal/provider/kvm/`, as a test of
|
||||
whether the contract is enough to build a provider. Both types ship,
|
||||
share the `KVMConfig` block shape and the `kvm-ssh` Secret, and may
|
||||
share a host.
|
||||
- **kvm2 image downloads run on a dedicated connection.** A cancelled or
|
||||
overlong call kills the connection it ran on (the only way to unblock
|
||||
go-libvirt), so a 435 MB upload on the pooled connection could die
|
||||
because a poll for another Proxy was cancelled. The transfer dials its
|
||||
own connection and closes it afterwards; the cheap checks and the marker
|
||||
stay on the pooled one.
|
||||
- **kvm2 health checks dial fresh connections**, the opposite of the kvm
|
||||
decision above. With credentials re-read on every dial a fresh
|
||||
connection does pick up a rotated Secret, and a check that runs out of
|
||||
time then closes only its own connection, so it cannot disrupt a
|
||||
reconcile. The cost is one sshd session per host per round.
|
||||
- **A cancelled kvm2 call kills the pooled connection.** Every RPC has a
|
||||
budget; when a call outlives its ctx or budget the adapter closes the
|
||||
socket and the SSH client, returns Transient, and marks the connection
|
||||
dead so the next call redials. Other calls in flight on that host fail
|
||||
once with Transient and their callers retry. Documented in the guide
|
||||
rather than avoided: the alternative, leaving a hung RPC blocked
|
||||
forever, pins a worker.
|
||||
- **kvm2 marks domains in its own metadata namespace**, so it never lists
|
||||
or adopts domains the `kvm` provider made. Switching an entry from `kvm`
|
||||
to `kvm2` means deleting its Proxies first; the two types can share a
|
||||
host without one collecting the other's VMs.
|
||||
- **kvm2 turns pool free space into the quota signal.** libvirt reports a
|
||||
full pool as an ordinary error, which would retry on the fast backoff.
|
||||
Before creating any volume, `Create` compares `StoragePoolGetInfo`'s
|
||||
available bytes with the image (if uncached) plus `diskSizeGb` plus 16
|
||||
MiB for the seed, and returns `ErrQuotaExceeded` below that, so the
|
||||
reconciler waits 5 minutes instead.
|
||||
- **kvm2 classifies SSH authentication and host-key failures as
|
||||
Permanent.** A key missing from `authorized_keys` or a host key missing
|
||||
from or different in known_hosts needs an admin; kvm retried these
|
||||
forever. A missing key or known_hosts file stays Transient, because the
|
||||
optional Secret may land later.
|
||||
- **kvm2 re-reads the SSH key and known_hosts on every dial**, so rotating
|
||||
the Secret needs no restart; nothing is cached. The constructor only
|
||||
checks that the files exist, to warn early without failing the provider
|
||||
set.
|
||||
|
||||
@@ -0,0 +1,597 @@
|
||||
# The kvm2 provider
|
||||
|
||||
Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. It is the
|
||||
second-generation libvirt provider, built clean-room from
|
||||
[provider-contract.md](provider-contract.md) and the design in
|
||||
[kvm2-provider-spec.md](kvm2-provider-spec.md), and it ships side by side
|
||||
with the first-generation `kvm` provider: both types can be configured at
|
||||
once, both take the same config block shape, and both read the same
|
||||
`kvm-ssh` Secret. What kvm2 adds over kvm: a child span per libvirt RPC,
|
||||
a pool-space precheck that maps to the reconciler's quota wait, SSH
|
||||
credentials re-read on every dial, image downloads and health checks on
|
||||
connections of their own, an adoption guard on the owning UID, and SSH
|
||||
authentication failures that stop retrying.
|
||||
|
||||
Like kvm, the operator reaches each host over SSH, forwards the libvirt
|
||||
unix socket through that connection and speaks libvirt's own RPC over it,
|
||||
so the hosts need nothing beyond `sshd` and `libvirtd` and the operator
|
||||
image carries no libvirt libraries. VMs boot a stock distro cloud image
|
||||
that the operator downloads into the host's storage pool on first use,
|
||||
with the Proxy's cloud-init delivered on a NoCloud seed ISO built
|
||||
in-process.
|
||||
|
||||
## Hypervisor prerequisites
|
||||
|
||||
Per host, once:
|
||||
|
||||
1. **libvirt and KVM** (`libvirtd` running, `/dev/kvm` present).
|
||||
2. **An SSH user in the `libvirt` group**, so it may open
|
||||
`/var/run/libvirt/libvirt-sock`. The operator authenticates with a
|
||||
private key; nothing else on the host is touched. The socket is used
|
||||
as `qemu:///system`, so no further host permissions are needed.
|
||||
3. **A bridge to a network with DHCP** (for example `br0` bridged onto
|
||||
the LAN). Each VM gets one virtio NIC on it and takes its IP from that
|
||||
network's DHCP. The operator's pod network and the crawler clients
|
||||
must route to those IPs: the health engine probes `http://<ip>:<port>`
|
||||
through the proxy and clients get the same address from the discovery
|
||||
API. Nothing on the host filters the port; the provider does not use
|
||||
`spec.port` at all.
|
||||
4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
|
||||
with room for one base image per distro (a few hundred MB) plus the
|
||||
per-VM overlays (`diskSizeGb` each, thin-provisioned). The pool must be
|
||||
running: a stopped pool fails the health check with `PoolNotActive` and
|
||||
fails `Create` permanently.
|
||||
5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu,
|
||||
`edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default with secure boot
|
||||
off (libvirt picks the OVMF binary itself); a domain start fails with
|
||||
"no firmware found" without it. Only `firmware: bios` images skip this.
|
||||
6. **Outbound internet from the VMs**, in practice: a vendor-data script
|
||||
installs `qemu-guest-agent` from the distro mirrors on first boot, and
|
||||
the agent is how the operator learns the VM's address. Without it the
|
||||
operator falls back to the host's ARP table, which only sees traffic
|
||||
addressed to the host itself, so on a plain bridge a VM without the
|
||||
agent stays `Provisioning`.
|
||||
|
||||
Check from the operator's point of view before wiring it in:
|
||||
|
||||
```sh
|
||||
ssh novakj@beelink virsh -c qemu:///system list --all # user can drive libvirt
|
||||
ssh novakj@beelink virsh -c qemu:///system pool-info default
|
||||
ssh novakj@beelink ip link show br0
|
||||
```
|
||||
|
||||
## Cluster-side setup
|
||||
|
||||
1. Create the SSH Secret from a key pair dedicated to the operator and a
|
||||
`known_hosts` file holding every host's key. It is the same `kvm-ssh`
|
||||
Secret the `kvm` provider uses; one Secret serves both types, and if
|
||||
you already have it there is nothing to do here.
|
||||
|
||||
```sh
|
||||
# once: a key pair for the operator; the .pub goes on every host
|
||||
ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator
|
||||
ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub novakj@beelink
|
||||
|
||||
# once per host: its host key, under the address the *operator* will
|
||||
# dial (hosts[].address), which may differ from what your laptop uses
|
||||
ssh-keyscan -H beelink.tail8c69c.ts.net >> ~/.ssh/kvm_known_hosts
|
||||
|
||||
# the Secret: the PRIVATE key (not the .pub) plus known_hosts
|
||||
kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
|
||||
--from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
|
||||
--from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
|
||||
```
|
||||
|
||||
`make kvm-ssh-secret` runs that last command as an apply, so it also
|
||||
updates an existing Secret; override the paths with
|
||||
`KVM_SSH_KEY=... KVM_KNOWN_HOSTS=...`. Putting the `.pub` in the
|
||||
Secret is the classic mistake: the target's `kvm-ssh-secret` check
|
||||
refuses a file without `PRIVATE KEY` in it, and the provider itself
|
||||
would fail every call permanently with a "parsing ssh key file" error.
|
||||
Without `known_hosts`, set `insecureIgnoreHostKey: true` and accept
|
||||
that a MITM could hand you a fake hypervisor.
|
||||
|
||||
`config/manager/manager.yaml` mounts the Secret (optional) at
|
||||
`/etc/proxy-operator/kvm` as two files, `id_ed25519` and
|
||||
`known_hosts`, which is what `sshKeyFile` and `knownHostsFile` in the
|
||||
provider config point at. The deployment starts fine without the
|
||||
Secret, even with a kvm2 provider configured: `New` only checks that
|
||||
the files exist and logs a `WARNING:` line when one does not; the
|
||||
provider's proxies fail transiently and recover once the Secret lands.
|
||||
|
||||
**Rotating the key or a host key needs no restart.** Both files are
|
||||
read and parsed on every dial (`loadCredentials` in
|
||||
`internal/provider/kvm2/ssh.go`); nothing is cached. Re-run the
|
||||
command or the make target, and the next connection, health check
|
||||
included, uses the new material. Connections that are already open
|
||||
stay open until they drop.
|
||||
|
||||
2. Add a provider entry to the providers config
|
||||
(`config/manager/providers_config.yaml` in-cluster; full sample in
|
||||
`config/samples/providers-config.yaml`). The type is `kvm2` and the
|
||||
block is `kvm2:`; its fields are those of the `kvm` block:
|
||||
|
||||
```yaml
|
||||
- name: kvm2-home
|
||||
type: kvm2
|
||||
kvm2:
|
||||
sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
|
||||
knownHostsFile: /etc/proxy-operator/kvm/known_hosts
|
||||
images:
|
||||
debian-13:
|
||||
url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
|
||||
machineTypes:
|
||||
small: {vcpus: 1, memoryMiB: 1024}
|
||||
hosts:
|
||||
- name: beelink
|
||||
address: beelink.tail8c69c.ts.net:22
|
||||
user: novakj
|
||||
bridge: br0
|
||||
```
|
||||
|
||||
Validation messages name the block, so a mistake reads
|
||||
`kvm2.hosts[0]: bridge is required`, not `kvm.hosts`. The providers
|
||||
config is read once per process: adding a host or an image entry needs
|
||||
a manager restart; credentials do not.
|
||||
|
||||
3. Apply a Proxy (`config/samples/proxy_kvm2.yaml`) and watch it:
|
||||
|
||||
```sh
|
||||
kubectl get px -w
|
||||
# NAME MODE PROVIDER PHASE IP HEALTHY
|
||||
# proxy-kvm2-sample Managed kvm2-home Provisioning
|
||||
# proxy-kvm2-sample Managed kvm2-home Running 192.168.1.73
|
||||
# proxy-kvm2-sample Managed kvm2-home Ready 192.168.1.73 True
|
||||
```
|
||||
|
||||
The first Proxy for a given image on a given host sits in
|
||||
`Provisioning` for the length of the image download; the operator log
|
||||
shows `fetching base image into host pool` with the URL and size and
|
||||
`base image cached` with the size, duration and digest.
|
||||
|
||||
## Config reference
|
||||
|
||||
The block is `kvm2:` and its Go type is the shared `KVMConfig` in
|
||||
`internal/provider/config_kvm.go`, so the fields, defaults and validation
|
||||
are the same as for `kvm`. Defaults are applied by `withDefaults` in
|
||||
`internal/provider/kvm2/kvm2.go`.
|
||||
|
||||
| Field | Required | Default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `sshKeyFile` | yes | | private key used for every host; OpenSSH or PEM, parsed on every dial |
|
||||
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against, read on every dial |
|
||||
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely; `New` logs a `WARNING:` line when it is on |
|
||||
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it. Also what the pool-space precheck reserves per VM |
|
||||
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on |
|
||||
| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table |
|
||||
| `images.<key>` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only: it becomes the cached volume name `base-<key>.<format>` |
|
||||
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw). The server must send `Content-Length` |
|
||||
| `images.<key>.sha256` | | | expected digest, 64 hex characters; verified after download, a change triggers a re-download |
|
||||
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
|
||||
| `images.<key>.firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition |
|
||||
| `machineTypes.<key>` | at least one | | `{vcpus, memoryMiB}` size profile, both greater than zero |
|
||||
| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique. Letters, digits, dots, dashes and underscores only: it becomes part of the providerID `hosts/<name>/domains/<vm>` |
|
||||
| `hosts[].address` | yes | | SSH endpoint, `host:port` |
|
||||
| `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
|
||||
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
|
||||
| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds |
|
||||
| `hosts[].bridge` | yes | | host bridge for the VM's NIC |
|
||||
|
||||
## Placement
|
||||
|
||||
All three fields are checked before any host is dialled. An unknown value
|
||||
is `ErrPermanent`, which the reconciler latches as `Failed` with the
|
||||
message below until the spec changes.
|
||||
|
||||
| Proxy field | Meaning for kvm2 | Error when unknown |
|
||||
|---|---|---|
|
||||
| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required | `placement.zone "x" is not in kvm2.hosts`; with several hosts and no zone, `placement.zone is required when kvm2.hosts lists more than one host` |
|
||||
| `placement.machineType` | key in `machineTypes` | `placement.machineType "x" is not in kvm2.machineTypes` |
|
||||
| `placement.image` | key in `images`, or an `http(s)://` URL used directly with the defaults (`qcow2`, `uefi`) | `placement.image "x" is neither a key in kvm2.images nor an http(s) URL` |
|
||||
| `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | |
|
||||
|
||||
The messages come from `Create` in `internal/provider/kvm2/create.go`,
|
||||
`host` in `kvm2.go` and `resolveImage` in `image.go`. Fix the spec (which
|
||||
replaces the VM) or the config. Catalogue edits affect only new VMs: the
|
||||
spec hash covers the placement values, not what the config maps them to.
|
||||
|
||||
## What lands on the host
|
||||
|
||||
For a Proxy whose deterministic name is `proxy-abc…`, on the chosen
|
||||
host's pool:
|
||||
|
||||
| Object | Name | Lifetime |
|
||||
|---|---|---|
|
||||
| base image | `base-<key>.<format>` (for a URL image, `<key>` is the first 16 hex characters of the URL's sha256) | cached; survives Proxy deletion |
|
||||
| image marker | `base-<key>.<format>.sha256`, a 64-byte raw volume holding the digest | with the base image |
|
||||
| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base file, `diskSizeGb` capacity | with the Proxy |
|
||||
| seed ISO | `proxy-abc…-seed.iso`, raw | with the Proxy |
|
||||
| domain | `proxy-abc…` | with the Proxy |
|
||||
|
||||
The names are built in `internal/provider/kvm2/volume.go`.
|
||||
|
||||
The domain (`buildDomainXML` in `xml.go`) is a `q35` machine with a
|
||||
host-passthrough CPU, ACPI and APIC, memory in MiB and vcpus from the
|
||||
machine type, the overlay on `vda` (virtio) and the seed as a read-only
|
||||
SATA cdrom on `sda`, one virtio NIC on the bridge, a serial console on a
|
||||
pty, the guest-agent channel `org.qemu.guest_agent.0`, a virtio RNG fed
|
||||
from `/dev/urandom`, and no graphics. Both disks are `type="file"` with
|
||||
the paths `StorageVolGetPath` returned: AppArmor derives QEMU's profile
|
||||
from those paths, and volume-type disks fail with permission denied on
|
||||
stock Debian and Ubuntu hosts. UEFI images get `<os firmware="efi">` with
|
||||
the `secure-boot` feature disabled; BIOS images get neither.
|
||||
|
||||
Its `<metadata>` carries one element in kvm2's own namespace,
|
||||
`https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2`
|
||||
(`metadataURI` in `metadata.go`):
|
||||
|
||||
```xml
|
||||
<instance xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2">
|
||||
<managed>true</managed>
|
||||
<uid>0f3a…</uid>
|
||||
<createdAt>2026-09-16T17:42:11Z</createdAt>
|
||||
</instance>
|
||||
```
|
||||
|
||||
That is how orphan GC recognises its own VMs, since libvirt has no
|
||||
labels, and how a repeat `Create` knows whether an existing domain of the
|
||||
same name belongs to this Proxy: a different non-empty `uid` is a
|
||||
permanent error naming both UIDs; an empty one is adopted. The namespace
|
||||
is kvm2's own, so the `kvm` provider's domains are invisible to it (see
|
||||
*Migrating from kvm*).
|
||||
|
||||
The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled
|
||||
`cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
|
||||
for byte**: any format cloud-init accepts works, including a Jinja
|
||||
template, a MIME archive, a shell script or a gzipped payload; the
|
||||
operator never parses or re-renders it. `meta-data` holds
|
||||
`instance-id: <name>` and `local-hostname: <name>`, so a reboot is never
|
||||
taken for a new instance. `vendor-data` carries the guest-agent
|
||||
bootstrap. The ISO is staged in a temp directory under `/tmp` (the
|
||||
manager's 64Mi emptyDir), assembled in memory (tens of KiB) and uploaded
|
||||
as a raw volume; the staging directory is removed before the call
|
||||
returns. The seed is rebuilt on every `Create` attempt, because a
|
||||
previous attempt may have been cut off mid-upload and a truncated seed
|
||||
would boot a VM without cloud-init.
|
||||
|
||||
The guest agent rides along in `vendor-data` as a **shell script**
|
||||
(`guestAgentScript` in `seed.go`), not a cloud-config, on purpose:
|
||||
cloud-init merges a vendor-data cloud-config under the user-data with
|
||||
mergers that keep the user-data's value for every top-level key both
|
||||
define and never append lists, so a Proxy carrying its own `packages:`
|
||||
would silently lose the agent. Scripts bypass the merge. The script is
|
||||
idempotent, picks whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk`
|
||||
the image has, enables the service with `systemctl` or OpenRC, and logs
|
||||
into `/var/log/cloud-init-output.log`. With `vendorData` also configured,
|
||||
the two travel as separate parts of a MIME multipart (`text/plain` for
|
||||
your document, `text/x-shellscript` for the script), with the
|
||||
`MIME-Version` header first so cloud-init detects it. With
|
||||
`installGuestAgent: false` the file is your `vendorData` alone, or empty.
|
||||
|
||||
## The image cache
|
||||
|
||||
The base image is downloaded **by the operator, through libvirt**, into
|
||||
the host's pool the first time a Proxy asks for it on that host. Nothing
|
||||
is pre-placed on the hypervisor and the operator needs no filesystem
|
||||
access there. The bytes stream from the URL through the operator into
|
||||
`StorageVolUpload` with a running sha256; nothing is buffered
|
||||
(`fetchImage` and `uploadImage` in `internal/provider/kvm2/image.go`).
|
||||
|
||||
The upload runs on a **dedicated connection** to the host, dialled for
|
||||
the transfer and closed after it, not on the pooled connection the other
|
||||
operations share. A cancelled or overlong call kills the connection it
|
||||
ran on (see *Timeouts and connections*), and a poll for some other Proxy
|
||||
being cancelled must not cut off a 20-minute upload.
|
||||
|
||||
The download needs a `Content-Length`: the volume's capacity is declared
|
||||
before the upload starts. The request sends `Accept-Encoding: identity`
|
||||
so a transparently gzipped body cannot hide the length. A response
|
||||
without it, or with length 0, is a permanent error naming the URL, and so
|
||||
is any 4xx status; 429 and 5xx are transient and retried.
|
||||
|
||||
The marker volume is written **last**, on the pooled connection, after
|
||||
`StoragePoolRefresh` has made libvirt read the file's real format. Its
|
||||
presence is what marks a base complete: a crash mid-download leaves a
|
||||
base volume without a marker, which the next attempt deletes and redoes
|
||||
(`cachedImage`). A re-download also happens when the marker's content is
|
||||
not a 64-character hex digest, or when `sha256` is configured and differs
|
||||
from it. Otherwise the cached copy is used for as long as it exists, even
|
||||
if the URL's content changed upstream: a `latest` URL is pinned at the
|
||||
moment of first fetch, per host. Concurrent Creates for the same image
|
||||
on the same host queue on a per-(host, volume) lock; the first fetches,
|
||||
the rest find the marker.
|
||||
|
||||
To evict manually:
|
||||
|
||||
```sh
|
||||
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256
|
||||
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2
|
||||
```
|
||||
|
||||
**Warning: do not change an image entry's `url` or `sha256`, and do not
|
||||
evict its base volume, while Proxies using that image exist on the
|
||||
host.** The overlays back onto the base file; deleting or replacing it
|
||||
corrupts every VM that uses it. Delete those Proxies first, or add the
|
||||
new image under a new key and move Proxies over by changing their
|
||||
`placement.image` (which replaces the VM).
|
||||
|
||||
## Quota: the pool-space precheck
|
||||
|
||||
Before creating any volume, `Create` reads `StoragePoolGetInfo` and
|
||||
compares the pool's free space with what it is about to commit
|
||||
(`checkPoolSpace` and `neededBytes` in `image.go`): the image's
|
||||
`Content-Length` when the base is not cached yet (found with a `HEAD`
|
||||
request, or a `GET` whose body is closed at once when the server refuses
|
||||
`HEAD`), plus `diskSizeGb` in bytes for the overlay, plus 16 MiB for the
|
||||
seed. Below that it returns `ErrQuotaExceeded` with a message naming the
|
||||
pool, the bytes available and the bytes needed. The reconciler answers
|
||||
quota with a wait, not a failure: the Proxy shows `Provisioned=False`
|
||||
with reason `QuotaExceeded` and is retried after `QuotaRetry` (5
|
||||
minutes), off the error backoff curve and never `Failed`. Free up the
|
||||
pool or lower `diskSizeGb` and the next attempt goes through.
|
||||
|
||||
The precheck is the only quota signal; libvirt itself reports a full pool
|
||||
as an ordinary error, which would be transient. The health check exposes
|
||||
the same number as `poolAvailableGiB`, so a pool that is filling up shows
|
||||
there before Creates start waiting.
|
||||
|
||||
A pool that is not running fails the precheck permanently (`storage pool
|
||||
<pool> is not running`): start it with `virsh pool-start` and
|
||||
`virsh pool-autostart`.
|
||||
|
||||
## Migrating from `kvm`
|
||||
|
||||
The `kvm2` block is the `kvm` block under another name, so an entry ports
|
||||
by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The
|
||||
`kvm-ssh` Secret is shared; nothing changes on the hosts.
|
||||
|
||||
What does not port is the VMs. kvm2 marks its domains in its own metadata
|
||||
namespace and neither lists nor adopts domains the `kvm` provider made:
|
||||
`ListByTag` skips them, so orphan GC never sees them, and a Proxy's
|
||||
`spec.provider` is immutable, so an existing Proxy cannot be moved to the
|
||||
new entry anyway. Do it in this order:
|
||||
|
||||
1. Delete the Proxies on the `kvm` entry and wait for them to go; the
|
||||
`kvm` provider removes their domains and volumes.
|
||||
2. Change the entry (or add a second one under a new name and keep the
|
||||
old one for a while; both types may share a host).
|
||||
3. Restart the manager, then recreate the Proxies with the new
|
||||
`spec.provider`.
|
||||
|
||||
A `kvm` domain left behind on a host is invisible to kvm2 forever; clean
|
||||
it up on the host with `virsh undefine --nvram <name>` and
|
||||
`virsh vol-delete` for its `<name>.qcow2` and `<name>-seed.iso`.
|
||||
|
||||
## Logging
|
||||
|
||||
The provider logs through controller-runtime's logger, named `kvm2`, with
|
||||
`provider` (the configured name) on every line, and `host` and `name`
|
||||
(the domain) on every line about an instance. Every line inside a traced
|
||||
operation also carries `traceID` and `spanID`.
|
||||
|
||||
- `--zap-log-level=debug` (logr V(1)):
|
||||
- one line per operation outcome: `kvm2 domain ensured running` with the
|
||||
`id`, `kvm2 domain fetched` with `state`, `ip` and `uid`,
|
||||
`kvm2 domain not found`, `kvm2 domain deleted`, `kvm2 domains listed`
|
||||
with `hosts` and `managed` counts, `kvm2 host checked` with `ready`,
|
||||
`reason` and `latency`; plus the steps inside a Create
|
||||
(`kvm2 domain already defined`, `overlay volume created`,
|
||||
`seed volume uploaded` with `bytes`, `domain defined` with `uid` and
|
||||
`firmware`, `domain started`, `fetching base image into host pool`,
|
||||
`base image cached`);
|
||||
- **one `libvirt call` line per libvirt RPC**, written by the decorator
|
||||
in `internal/provider/kvm2/observe.go` inside that RPC's span: `rpc`
|
||||
(the go-libvirt method name, for example `DomainLookupByName`), the
|
||||
identifiers it was given (`domain`, `pool`, `volume`), `duration`, and
|
||||
on failure `error` plus `libvirtCode` and `libvirtError` (the code's
|
||||
name, for example `ErrNoDomain`). A Get is four or five RPCs; a
|
||||
Create with a cached image is around twenty;
|
||||
- connection events from `internal/provider/kvm2/hosts.go`:
|
||||
`libvirt host connected` with `address`, `user` and `duration`, and
|
||||
`libvirt host connect failed` with the same plus `error` and `stage`
|
||||
(`credentials`, `tcp`, `ssh` or `socket`);
|
||||
- `kvm2 backend call failed` with `op` and the raw `error` once per
|
||||
failed operation, before the error is returned;
|
||||
- a line per domain that `ListByTag` skips, and a line when the image
|
||||
marker forces a refetch.
|
||||
- `--zap-log-level=2`: the `libvirt call` lines also carry the domain and
|
||||
volume `xml`, the `metadata` element, the interface `addresses`, the
|
||||
listed `domains`, the pool `state` and sizes and the volume `path`; the
|
||||
`domain defined` line carries the `xml`; `kvm2 create resolved` logs the
|
||||
resolved parameters (pool, bridge, machine type, sizes, image volume,
|
||||
firmware) and `cloudInitBytes`; `kvm2 listed instance` logs each listed
|
||||
domain.
|
||||
- Never logged at any level: the cloud-init content, the SSH key, the
|
||||
known_hosts content, and upload or download bodies. The seed and image
|
||||
uploads report `offset` and `length` only. There is no raw wire tier
|
||||
like the GCP provider's V(5); go-libvirt exposes no hook for it.
|
||||
|
||||
## Tracing
|
||||
|
||||
With an OTLP exporter configured ([architecture.md
|
||||
§10](architecture.md#10-tracing-internaltracing)), every backend call is
|
||||
a child span of the operation span the decorators open
|
||||
(`provider.create`, `provider.get`, `provider.delete`, `provider.list`)
|
||||
or of the monitor's `providerhealth.check`:
|
||||
|
||||
| Span | Around | Attributes |
|
||||
|---|---|---|
|
||||
| `kvm2.<Method>`, for example `kvm2.DomainDefineXML` | one libvirt RPC (`observe.go`) | `kvm2.host`, and the identifiers passed: `kvm2.domain`, `kvm2.pool`, `kvm2.volume`; on failure the status is Error and `kvm2.libvirt_code` carries the libvirt error number |
|
||||
| `kvm2.dial` | the SSH handshake and the libvirt connect (`hosts.go`) | `kvm2.host`, `kvm2.address`, `kvm2.user` |
|
||||
| `kvm2.download` | one image fetch, from the HTTP request to the marker (`image.go`) | `kvm2.host`, `kvm2.volume`, `kvm2.url_host`, `kvm2.sha256_configured`, and `kvm2.bytes` once the response arrived |
|
||||
|
||||
All spans are client spans. No attribute carries cloud-init, key material
|
||||
or a URL beyond its host. The `libvirt call` log line for an RPC is
|
||||
written inside that RPC's span, so its `spanID` points at the span.
|
||||
|
||||
## Timeouts and connections
|
||||
|
||||
No caller puts a deadline on the context, so every backend call carries
|
||||
its own budget (`defaultBudgets` in `internal/provider/kvm2/backend.go`,
|
||||
applied by `call`):
|
||||
|
||||
| Call | Budget |
|
||||
|---|---|
|
||||
| TCP connect plus SSH handshake plus libvirt connect | 10 s |
|
||||
| ordinary RPC | 30 s |
|
||||
| `DomainCreate` | 90 s |
|
||||
| `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s |
|
||||
| `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier |
|
||||
| wait for running after `DomainCreate` | 2 m, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`) |
|
||||
| the `HEAD` request of the pool-space precheck | 30 s |
|
||||
| the courtesy `ConnectClose` when a connection is closed | 5 s |
|
||||
|
||||
go-libvirt's calls take no context and block, so the only way to abandon
|
||||
one is to close the connection under it. That is what a cancelled or
|
||||
overlong call does: the adapter kills the connection (socket and SSH
|
||||
client), returns `ErrTransient` wrapping the context or deadline error,
|
||||
and marks the connection dead so the next call redials. The consequence:
|
||||
**other RPCs in flight on that host's pooled connection fail once with
|
||||
`ErrTransient`** and their callers retry on a fresh connection. In the
|
||||
reconciler that is one requeue with backoff; in GC it is one skipped
|
||||
sweep.
|
||||
|
||||
Two callers are kept off the pooled connection for that reason
|
||||
(`dialFresh` in `hosts.go`): the image upload, because a 435 MB transfer
|
||||
must not die because a poll for another Proxy was cancelled, and the
|
||||
health check, because a check that runs out of the monitor's timeout must
|
||||
close only its own connection.
|
||||
|
||||
Otherwise there is one pooled SSH plus libvirt connection per host,
|
||||
dialled lazily on the first call that needs it and shared by every
|
||||
operation for that host (go-libvirt multiplexes RPCs on one connection).
|
||||
Concurrent callers wait for one dial and share the result; waiting ends
|
||||
with the caller's context. A `keepalive@openssh.com` request every 15 s
|
||||
marks the connection dead when it fails or gets no reply within another
|
||||
15 s, as does go-libvirt noticing the socket is gone; the next call
|
||||
redials. Create and Delete for the same domain name are serialised on a
|
||||
per-name lock, so two Deletes (GC and the reconciler) never race on
|
||||
destroy and undefine, and a Delete cannot interleave with a Create's
|
||||
steps; Get and ListByTag take no lock.
|
||||
|
||||
## Health check
|
||||
|
||||
The operator checks every configured host every `--provider-check-interval`
|
||||
(default 1m) and publishes the result in the `ProviderHealth` object named
|
||||
after the provider (`kubectl get ph kvm2-home -o yaml`) and in the
|
||||
`proxy_operator_provider_target_up{provider,target}` gauge. One target per
|
||||
`hosts[]` entry, in config order, all checked concurrently.
|
||||
|
||||
Each check dials a **fresh connection** to the host and closes it when
|
||||
done (`probeHost` in `internal/provider/kvm2/check.go`); the pooled
|
||||
connection is never touched, so a check that runs out of time cannot fail
|
||||
a reconcile's call in flight, and a fresh dial picks up a rotated Secret
|
||||
at once. The sequence is the SSH and libvirt handshakes, then four RPCs:
|
||||
`ConnectGetLibVersion`, `ConnectGetHostname`, `StoragePoolLookupByName`
|
||||
and `StoragePoolGetInfo` on the host's `pool`. The cost per host per
|
||||
round is one sshd session and those four calls.
|
||||
|
||||
Every target's `details` carry `user` and `pool`. As facts arrive they
|
||||
gain `libvirtVersion` (decoded `major.minor.micro`), `hostname`,
|
||||
`poolState` (virsh's word: `running`, `inactive`, ...),
|
||||
`poolCapacityGiB` and `poolAvailableGiB`, so a target that fails late in
|
||||
the sequence still says what the host answered before. The free space is
|
||||
the number the quota precheck compares against.
|
||||
|
||||
A failing target says where the check broke (`checkReason` in `check.go`):
|
||||
|
||||
| Reason | Meaning | Look at |
|
||||
|---|---|---|
|
||||
| `Reachable` | ready | |
|
||||
| `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host |
|
||||
| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
|
||||
| `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) |
|
||||
| `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group |
|
||||
| `Timeout` | the connection was up, but the monitor's `--provider-check-timeout` (default 15s) or an RPC's budget ran out before the check finished | a hung `libvirtd` or a saturated link |
|
||||
| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
|
||||
| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
|
||||
| `APIError` | any other libvirt error | the message carries libvirt's own text |
|
||||
|
||||
## Error classes
|
||||
|
||||
Every error is classified once, in `classify` in
|
||||
`internal/provider/kvm2/errors.go`, with errors that carry a class from
|
||||
their source (validation, the image fetch, the dial) keeping it. The
|
||||
reconciler latches `ErrPermanent` as `Failed` until the spec changes,
|
||||
waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
|
||||
`Get`, and retries everything else with backoff.
|
||||
|
||||
| Class | What lands there |
|
||||
|---|---|
|
||||
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
|
||||
| `ErrQuotaExceeded` | the pool-space precheck |
|
||||
| `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
|
||||
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring |
|
||||
|
||||
Error strings name paths, hosts, volumes and libvirt's own text. They
|
||||
never contain key bytes, the known_hosts content or cloud-init.
|
||||
|
||||
## Known gaps
|
||||
|
||||
- **Create blocks**, including the first image download on each host,
|
||||
holding one of the three reconcile workers for its duration (LAN
|
||||
speed, a few hundred MB, once per image per host). Other Proxies keep
|
||||
reconciling on the remaining workers. Warm the cache with one Proxy per
|
||||
image per host before creating many.
|
||||
- **Volumes without a domain are invisible to GC.** A crash between
|
||||
creating the overlay or seed and defining the domain leaves volumes
|
||||
that `ListByTag` cannot see (it lists domains). The same Proxy's next
|
||||
Create reuses the overlay and rebuilds the seed, and its Delete removes
|
||||
both by name even with no domain, but a Proxy CR that is gone for good
|
||||
leaves them for `virsh vol-list` to find.
|
||||
- **A host removed from the config orphans its VMs.** `Delete` for a
|
||||
providerID naming an unconfigured host is a logged no-op success, so
|
||||
the Proxy finishes deleting instead of pinning its finalizer, but the
|
||||
domain and its volumes stay on the hypervisor and orphan GC cannot see
|
||||
them either (it only sweeps configured hosts). Delete the Proxies before
|
||||
retiring a host, or clean up afterwards with `virsh undefine --nvram`
|
||||
plus `virsh vol-delete`. `Get` fails permanently for such a providerID
|
||||
on purpose: a typo in `hosts` must not look like a missing VM and
|
||||
trigger a rebuild elsewhere.
|
||||
- **No adoption of `kvm` domains.** See *Migrating from kvm*.
|
||||
- **A cancelled call drops the host's pooled connection once**, so other
|
||||
RPCs in flight on that host fail once with `ErrTransient` and retry.
|
||||
The image upload and the health check are isolated on their own
|
||||
connections.
|
||||
- **`Content-Length` is required** for image downloads; a mirror that
|
||||
only chunks is rejected permanently. Point the catalogue at one that
|
||||
sends the length.
|
||||
- **A paused or pmsuspended domain cannot be adopted.** A repeat `Create`
|
||||
for a domain in that state calls `DomainCreate`, which libvirt refuses
|
||||
as "already running"; the provider then waits for the domain to report
|
||||
running, the 2-minute wait expires, and the call fails transiently on
|
||||
every retry. Resume the domain on the host (`virsh resume <name>`) or
|
||||
delete the Proxy.
|
||||
- **Parser text in credential errors.** A key or known_hosts file that
|
||||
does not parse fails with the parser's own message, which can quote a
|
||||
PEM header type or a malformed known_hosts token. Key bytes are never
|
||||
quoted.
|
||||
- **The sample cloud-init is an open proxy.**
|
||||
`config/samples/proxy_kvm2.yaml` installs Squid with
|
||||
`http_access allow all`; restrict it before exposing the bridge network
|
||||
to anything but the cluster.
|
||||
- **IP discovery needs the agent or guest traffic to the host.** A VM on
|
||||
a bridge with no DHCP never gets an address and stays `Provisioning`
|
||||
forever; check `virsh console <name>` on the host. The same symptom
|
||||
with a booted, addressed VM means the guest agent never came up: check
|
||||
`systemctl status qemu-guest-agent`, `cloud-init status --long` and the
|
||||
vendor script's output in `/var/log/cloud-init-output.log` (a VM without
|
||||
outbound internet cannot install it). A user-data that disables
|
||||
vendor-data (`vendor_data: {enabled: false}`) also removes the agent, by
|
||||
design. `virsh domifaddr <name> --source agent` and `--source arp` show
|
||||
what the operator sees, in that order.
|
||||
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
|
||||
([Debian bug #1111240](https://bugs.debian.org/1111240)), which is why
|
||||
`firmware` defaults to `uefi`; the same image boots under UEFI with the
|
||||
serial console intact. Images that need `firmware: bios` are the
|
||||
exception.
|
||||
|
||||
## Testing
|
||||
|
||||
Unit tests run over an in-memory fake of the libvirt seam and an
|
||||
in-process SSH server. The optional integration test against a real host
|
||||
is described in
|
||||
[testing.md](testing.md#the-kvm2-integration-test-real-hypervisor); run it
|
||||
with `make test-integration-kvm2`. `TestKVM2Check_realHost` runs the
|
||||
health check against the same host and creates nothing:
|
||||
`go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/`.
|
||||
@@ -11,7 +11,7 @@ Plan: [docs/plans/2026-09-16-1333-kvm2-provider.md](../plans/2026-09-16-1333-kvm
|
||||
- [x] Step 6 — The four contract methods
|
||||
- [x] Step 7 — Checker and the CRD comment
|
||||
- [x] Step 8 — Deployment files, Makefile, integration test
|
||||
- [ ] Step 9 — Documentation
|
||||
- [x] Step 9 — Documentation
|
||||
- [ ] Step 10 — Milestone review
|
||||
- [ ] Step 11 — Real-host validation, changelog, MR
|
||||
|
||||
@@ -318,3 +318,23 @@ make -n test-integration-kvm2
|
||||
```
|
||||
|
||||
All clean. Attestation: "files read outside the allowed set: none".
|
||||
|
||||
## Step 9 — Documentation
|
||||
|
||||
2026-09-16 19:54. Eighth subagent step: `docs/kvm2-provider.md` (sixteen
|
||||
sections modelled on the kvm guide), a `kvm2` column in the three tables of
|
||||
`docs/providers-existing.md` plus a kvm2 section and renumbering, the
|
||||
components row, §5 block, §10 span bullet, §11 row and eight Decisions
|
||||
bullets in `docs/architecture.md`, a README row and paragraph, and the
|
||||
guide in the `docs/providers.md` index. The agent verified each claim
|
||||
against the package rather than the spec and dropped a draft sentence
|
||||
claiming kvm and kvm2 share the base-image cache, because checking it would
|
||||
have meant reading v1's volume names.
|
||||
|
||||
Checks re-run by the orchestrator: no em-dash in the guide, no line
|
||||
references, each of the 19 axis codes appears once in the appendix table
|
||||
with seven cells, every link and anchor across the seven touched documents
|
||||
resolves. The one added em-dash is the `Leader-elected` cell of the new
|
||||
components-table row, where the table's own convention wins.
|
||||
|
||||
Attestation: "files read outside the allowed set: none".
|
||||
|
||||
+267
-68
@@ -1,13 +1,17 @@
|
||||
# The existing providers
|
||||
|
||||
This is the appendix to [provider-contract.md](provider-contract.md): how the
|
||||
three providers that exist today (kubernetes, gcp, kvm) meet the contract,
|
||||
which profile each one chose on every axis, and what is still missing. It is
|
||||
reference material for whoever writes the next provider, not a user guide;
|
||||
the user guides are [kvm-provider.md](kvm-provider.md),
|
||||
four providers that exist today (kubernetes, gcp, kvm, kvm2) meet the
|
||||
contract, which profile each one chose on every axis, and what is still
|
||||
missing. It is reference material for whoever writes the next provider, not
|
||||
a user guide; the user guides are [kvm-provider.md](kvm-provider.md),
|
||||
[kvm2-provider.md](kvm2-provider.md),
|
||||
[gcp-vm-validation.md](gcp-vm-validation.md) and
|
||||
[gcp-in-specific-project.md](gcp-in-specific-project.md). The README's
|
||||
"Implemented providers" table stays the short public summary.
|
||||
"Implemented providers" table stays the short public summary. kvm2 is the
|
||||
one provider built from the contract itself (clean-room, from
|
||||
[kvm2-provider-spec.md](kvm2-provider-spec.md), without reading kvm's code),
|
||||
so its column is also a record of what the contract produced.
|
||||
|
||||
The per-provider call flows are drawn in
|
||||
[architecture.md §5](architecture.md#5-what-provider-calls-do-in-the-outside-world),
|
||||
@@ -23,59 +27,59 @@ Code references are a file path plus a function or symbol name.
|
||||
One row per axis of contract §4, in the same order, with the profile name
|
||||
and the symbol that shows it.
|
||||
|
||||
| Axis | kubernetes | gcp | kvm |
|
||||
|---|---|---|---|
|
||||
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) |
|
||||
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) |
|
||||
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
|
||||
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) |
|
||||
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) |
|
||||
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) |
|
||||
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) |
|
||||
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) |
|
||||
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) |
|
||||
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` |
|
||||
| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning |
|
||||
| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none |
|
||||
| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
|
||||
| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) |
|
||||
| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) |
|
||||
| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection |
|
||||
| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier |
|
||||
| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans |
|
||||
| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` |
|
||||
| Axis | kubernetes | gcp | kvm | kvm2 |
|
||||
|---|---|---|---|---|
|
||||
| `readiness` | immediate: a Pod is scheduled within seconds (`stateFromPod`) | provisioning-window: minutes of boot (`mapState`) | provisioning-window: boot plus, on first use, an image download (`imageCache.ensure`) | provisioning-window: boot plus, on first use, an image download (`ensureImage`) |
|
||||
| `create-return` | submit: returns after `client.Create` | submit: `Insert` is never awaited, `Operation.Wait` is not called | **blocking**: waits through `DomainCreate` (`ensureRunning`), and the first image fetch runs inside (`imageCache.fetch`) | **blocking**: waits through `DomainCreate` and a 2-minute poll for running (`ensureRunning`, `waitRunning`), and the first image fetch runs inside (`fetchImage`) |
|
||||
| `delete-return` | returns-early: `client.Delete`, Pod terminates on its own | returns-early: `Delete` submits an operation each time | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) | waits: destroy, undefine and volume removal complete before return (`removeDomain`, `removeVolumes`) |
|
||||
| `ip-source` | assigned-at-create: `PodIP` (`instanceFromPod`) | assigned-at-create: the external NAT IP of the first NIC (`toInstance`) | discovered: ARP table first, then the guest agent, first routable IPv4 (`address`, `firstIPv4`) | discovered: the guest agent first, then the ARP table, first routable IPv4 (`addressSources`, `address`, `firstIPv4`) |
|
||||
| `gc-mark` | native-labels: Pod labels (`buildPod`) | native-labels: instance labels (`buildInsertRequest`) | written-metadata: a namespaced `<metadata>` element written in `DomainDefineXML` (`metadata.go`) | written-metadata: an element in kvm2's own namespace written in `DomainDefineXML` (`metadataURI`, `newInstanceMetadata`) |
|
||||
| `created-at` | backend-timestamp: `CreationTimestamp` | backend-timestamp: `CreationTimestamp`, parse error gives zero | written-at-create: `createdAt` in the metadata element (`newInstanceMetadata`) | written-at-create: `createdAt` in the metadata element, parse error gives zero (`newInstanceMetadata`, `createdAt`) |
|
||||
| `list-partial` | n/a: one call, all namespaces | omit-unreachable: `ReturnPartialSuccess` on `AggregatedList` | fail-whole per host, one unreadable domain skipped (`ListByTag`, `managedInstance`) | fail-whole per host, one unreadable domain skipped unless ctx ended (`ListByTag`) |
|
||||
| `quota` | absent: a quota 403 is Permanent (`classify`) | detected: 429, and 403 with reason `quotaExceeded` or `rateLimitExceeded` (`classify`, `hasReason`) | absent (`classify`) | detected: a pool-space precheck before any volume is created (`checkPoolSpace`, `neededBytes`) |
|
||||
| `not-found` | status-404: `apierrors.IsNotFound` | status-404: `isNotFound` | error-code: `ErrNoDomain`, `ErrNoStorageVol` (`hasCode`) | error-code: `ErrNoDomain`, `ErrNoStorageVol`, Transient inside Create (`classifyLibvirt`) |
|
||||
| `primary-key` | name-is-key: `namespace/name` | name-is-key: `zones/<zone>/instances/<name>` | name-is-key: `hosts/<host>/domains/<name>` | name-is-key: `hosts/<host>/domains/<name>`, with a UID guard on adoption (`create`) |
|
||||
| `provisioning-state` | native: `PodPending` | native: `PROVISIONING`, `STAGING`, `REPAIRING` | create-blocks: a domain is defined and started in one `Create`; `nostate` maps to Provisioning | create-blocks: defined, started and polled to running in one `Create`; `nostate` maps to Provisioning (`mapState`) |
|
||||
| `async-ops` | none | task-handle, but the handle is discarded: an insert that fails after acceptance is never read | none | none |
|
||||
| `secondary-resources` | none | cascade: `AutoDelete` on the boot disk (`buildInsertRequest`) | explicit: overlay and seed removed by name (`removeVolumes`); base image kept | explicit: overlay and seed removed by name (`removeVolumes`); base image kept |
|
||||
| `credentials` | workload-identity: in-cluster config or kubeconfig (`NewWithTransportWrapper`) | environment: Application Default Credentials | mounted-files: SSH key and known_hosts from the optional `kvm-ssh` Secret, read lazily (`newLazySSH`) | mounted-files: the same `kvm-ssh` Secret, read on every dial and never cached (`loadCredentials`) |
|
||||
| `client-kind` | vendor-sdk: controller-runtime `client.Client`, no seam interface | vendor-sdk: `cloud.google.com/go/compute` behind `instancesAPI` | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind `libvirtAPI` (`conn.go`) | rpc-over-ssh: `go-libvirt` over an SSH-forwarded unix socket, behind the ctx-first `backend` seam (`backend.go`) |
|
||||
| `timeouts` | ctx-only: no `rest.Config.Timeout` is set | ctx-only: SDK defaults, with SDK retries on Get and List | per-call for the dial only (`defaultHandshakeTimeout` in `dialSSH`); RPCs are ctx-only and a cancelled `call` closes the connection | per-call: a budget around the dial, every RPC, the uploads and the download (`defaultBudgets`, `call`); a cancelled or overlong call kills the pooled connection |
|
||||
| `logging` | none | tiered: V(1) outcomes and `logAPIError`, V(2) detail, V(5) wire log (`wirelog.go`) | tiered: V(1) outcomes and one `libvirt call` line per RPC (`trace.go`), V(2) XML and addresses; no wire tier | tiered: V(1) outcomes and one `libvirt call` line per RPC (`observe.go`), V(2) XML and addresses; no wire tier |
|
||||
| `child-spans` | transport-wrapper: `tracing.RestConfigWrapper()` on its own `rest.Config`, parent-gated | sdk-instrumentation: the SDK adds `otelhttp` itself; not parent-gated, but every call site has a parent | **none**: `trace.go` is a log decorator, not spans | span-decorator: `kvm2.<Method>` per RPC (`observe`), plus `kvm2.dial` (`libvirtDialer`) and `kvm2.download` (`fetchImage`) |
|
||||
| `backend-test` | cluster-e2e: kind (`test/e2e`) | manual-runbook: [gcp-vm-validation.md](gcp-vm-validation.md) | integration-tag: `kvm_integration_test.go`, `make test-integration-kvm` | integration-tag: `kvm2_integration_test.go`, `make test-integration-kvm2` |
|
||||
|
||||
## 2. Other differences
|
||||
|
||||
Rows that are not profile axes but that a new provider author compares
|
||||
against.
|
||||
|
||||
| Feature | kubernetes | gcp | kvm |
|
||||
|---|---|---|---|
|
||||
| Create verifies the adopted instance's UID | no | no | no |
|
||||
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image |
|
||||
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
|
||||
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no |
|
||||
| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped |
|
||||
| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a |
|
||||
| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes |
|
||||
| Checker target | `apiserver` | the project | each host, concurrently |
|
||||
| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` |
|
||||
| Credential rotation without restart | n/a | via ADC | no: cached after first successful load |
|
||||
| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) |
|
||||
| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` |
|
||||
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
|
||||
| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN |
|
||||
| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself |
|
||||
| Feature | kubernetes | gcp | kvm | kvm2 |
|
||||
|---|---|---|---|---|
|
||||
| Create verifies the adopted instance's UID | no | no | no | **yes**: a different non-empty UID in the metadata is Permanent, an empty one is adopted (`create`) |
|
||||
| Placement used | none | Zone, MachineType, Image | Zone (host), MachineType, Image | Zone (host), MachineType, Image |
|
||||
| CloudInit delivered | **no**: ignored | yes: `user-data` metadata | yes: NoCloud seed ISO built in-process (`buildSeedISO`) | yes: NoCloud seed ISO built in-process (`buildSeedISO`) |
|
||||
| Port used | yes: Squid config and containerPort (`squidConf`) | no: firewall is manual | no | no |
|
||||
| States produced | Provisioning, Running, Terminated | all four | Provisioning, Running, Stopped | Provisioning, Running, Stopped |
|
||||
| Deleting instance still reports its state | yes: a Pod being deleted reports its phase | n/a | n/a | n/a: Delete waits, and a concurrent Get sees the domain until it is undefined |
|
||||
| Error.Provider is the configured name | **no**: hardcoded `"kubernetes"` | yes | yes | yes (`wrapErr`) |
|
||||
| Checker target | `apiserver` | the project | each host, concurrently | each host, concurrently, on a fresh connection |
|
||||
| Checker Details | none | none | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacity`, `poolAvailable` | `user`, `pool`, `hostname`, `libvirtVersion`, `poolState`, `poolCapacityGiB`, `poolAvailableGiB`, filled in as far as the check got |
|
||||
| Credential rotation without restart | n/a | via ADC | no: cached after first successful load | yes: both files read on every dial (`loadCredentials`) |
|
||||
| Connection handling | client-go | SDK client, never closed | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostPool`) | one pooled SSH plus libvirt connection per host, 15s keepalive (`hostTable`); dedicated connections for the image upload and the check (`dialFresh`) |
|
||||
| Unit-test double | controller-runtime fake client | `fakeAPI` behind `instancesAPI`, plus `httptest` for the real SDK client in `check_test.go` | `fakeAPI` behind `libvirtAPI`, plus an in-process `x/crypto/ssh` server in `conn_test.go` | `fakeBackend` behind `backend` (`fake_test.go`), plus an in-process `x/crypto/ssh` server (`sshserver_test.go`) and go-libvirt's own mock for the adapter |
|
||||
| Proxy software | generated Squid config (`pod.go`) | from `cloudInit`; the image must run cloud-init | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) | from `cloudInit`, plus a vendor-data script that installs the guest agent (`seed.go`) |
|
||||
| IP reachable from the operator | Pod network, so in-cluster only (not from `make run-dev` on a laptop) | external IP, once a firewall rule allows the cluster's egress IP | LAN address on the host's bridge; the operator needs a route to that LAN | LAN address on the host's bridge; the operator needs a route to that LAN |
|
||||
| Distinct egress IP per proxy | no: the cluster's egress | yes: ephemeral external IP | whatever the host's LAN routes through; not per VM by itself | whatever the host's LAN routes through; not per VM by itself |
|
||||
|
||||
## 3. Observability status
|
||||
|
||||
| Layer | kubernetes | gcp | kvm |
|
||||
|---|---|---|---|
|
||||
| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators |
|
||||
| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** |
|
||||
| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) |
|
||||
| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no |
|
||||
| Layer | kubernetes | gcp | kvm | kvm2 |
|
||||
|---|---|---|---|---|
|
||||
| 1. Operation spans and metrics | yes: decorators | yes: decorators | yes: decorators | yes: decorators |
|
||||
| 2. Child spans per backend call | yes: `tracing.RestConfigWrapper()`, parent-gated (`cmd/main.go`, `NewWithTransportWrapper`) | yes: the SDK's own `otelhttp`; not parent-gated | **no** | yes: `kvm2.<Method>` from the `observe` decorator, `kvm2.dial`, `kvm2.download`; always under the caller's span, since every call carries the caller's ctx |
|
||||
| 3. Logs carry `traceID` and `spanID` | n/a: the package does not log | yes, including the V(5) wire log | yes: every `libvirt call` line at V(1) | yes: every `libvirt call` line at V(1), written inside its RPC's span |
|
||||
| Tested against a real collector | yes: the kind e2e checks `provider.create` and `provider.delete` in Tempo (`test/e2e/tracing_test.go`), operation spans only | no | no | no; the integration test asserts the parent-child topology in an in-memory recorder (`assertLifecycleSpans`) |
|
||||
|
||||
## 4. kubernetes (`internal/provider/kubernetes`)
|
||||
|
||||
@@ -329,18 +333,205 @@ the author's view.
|
||||
tries ARP first; it says `host` and `name` are on every log line, but
|
||||
the check and list paths lack them.
|
||||
|
||||
## 7. Gaps shared by the existing providers
|
||||
## 7. kvm2 (`internal/provider/kvm2`)
|
||||
|
||||
- **Adoption is not verified.** None of the three checks `LabelUID` when
|
||||
`Create` finds an existing instance with the right name.
|
||||
- **The Create comment does not match kvm.** The interface comment in
|
||||
`internal/provider/provider.go` says `Create` returns as soon as the
|
||||
request is submitted; kvm blocks until the domain has started.
|
||||
The full user guide is [kvm2-provider.md](kvm2-provider.md) and the design
|
||||
it was built from is [kvm2-provider-spec.md](kvm2-provider-spec.md); this
|
||||
section is the author's view. Nothing in the package was taken from
|
||||
`internal/provider/kvm`; where the two agree it is because both follow the
|
||||
same libvirt.
|
||||
|
||||
- **Package and seam.** `kvm2.go` (`Provider`, `New`, `withDefaults`,
|
||||
`host`, `wrapErr`, the per-name lock), `create.go`, `get.go`,
|
||||
`delete.go`, `list.go` (the four methods), `backend.go` (the seam, the
|
||||
production adapter and the budgets), `ssh.go` (credentials, the forwarded
|
||||
socket, keepalive), `hosts.go` (the pooled table and the production
|
||||
dialer), `observe.go` (the span and log decorator), `image.go` (cache,
|
||||
download, quota precheck), `seed.go`, `xml.go`, `volume.go`,
|
||||
`metadata.go`, `id.go`, `state.go`, `errors.go`, `check.go`. The seam is
|
||||
`backend`: the twenty go-libvirt calls the provider uses, each with `ctx`
|
||||
first (go-libvirt's own methods take none and block), plus `Close`.
|
||||
`libvirtBackend` adapts `*libvirt.Libvirt` and runs every call through
|
||||
`call`, which applies the budget and the abandon policy. The `dialer`
|
||||
seam separates the SSH transport: `newProvider` injects one for tests,
|
||||
`newLibvirtDialer` is production. The `observe` decorator is applied
|
||||
once, in `libvirtDialer.dial`, so provider code never holds an
|
||||
undecorated backend.
|
||||
- **Config.** `ProviderConfig.KVM2`, the same `KVMConfig` type as kvm,
|
||||
validated by `validateKVMBlock` with block name `kvm2` so messages read
|
||||
`kvm2.hosts`. Defaults (`diskSizeGb` 10, socket, pool, image format and
|
||||
firmware) are filled by `withDefaults`. `New` never dials; it checks that
|
||||
the credential files exist and warns when one does not, or when
|
||||
`insecureIgnoreHostKey` is on.
|
||||
- **ID.** `formatProviderID(host, name)` gives `hosts/<host>/domains/<name>`;
|
||||
`parseProviderID` accepts exactly four segments with the literal
|
||||
`hosts` and `domains` and non-empty host and name, else `ErrPermanent`.
|
||||
- **Create.** `Create` validates `Zone` (`host`; optional with one host),
|
||||
`MachineType` (a catalogue key) and `Image` (`resolveImage`: a catalogue
|
||||
key or an http(s) URL with qcow2 and uefi defaults) before any dial, and
|
||||
logs the resolved parameters with `cloudInitBytes` at V(2). Then, under
|
||||
the per-name lock (`lockName`) and on the pooled connection
|
||||
(`hostTable.get`), `create`: `DomainLookupByName`; found means the
|
||||
adoption guard (a non-empty metadata UID that differs from `req.UID` is
|
||||
Permanent) and `ensureRunning`. Otherwise `StoragePoolLookupByName`,
|
||||
`imageSizeIfUncached` (marker lookup, then `HEAD` or a closed `GET`) and
|
||||
`checkPoolSpace` with `neededBytes` (image if uncached, plus
|
||||
`diskSizeGb`, plus 16 MiB), then `ensureImage` (`cachedImage` on the
|
||||
pooled connection, `fetchImage` streaming into `uploadImage` on a
|
||||
`dialFresh` connection, `StoragePoolRefresh`, `writeMarker` last),
|
||||
`ensureOverlay` (`ErrStorageVolExist` means reuse), `writeSeed` (delete,
|
||||
build, upload; always rebuilt), two `StorageVolGetPath`, `buildDomainXML`
|
||||
with file-type disks and the metadata element, `DomainDefineXML`,
|
||||
`ensureRunning` (`DomainGetState`; `DomainCreate` with
|
||||
`ErrOperationInvalid` as already running; `waitRunning` polls every
|
||||
500 ms for up to 2 minutes, Transient on expiry). `Create` returns only
|
||||
when the domain is running.
|
||||
- **Get and state.** `mapState(state, ip)`:
|
||||
|
||||
| libvirt state | IP | Provider state |
|
||||
|---|---|---|
|
||||
| nostate | any | Provisioning |
|
||||
| running, blocked | empty | Provisioning |
|
||||
| running, blocked | set | Running |
|
||||
| paused, shutdown, shutoff, crashed, pmsuspended, anything else | any | Stopped |
|
||||
|
||||
`get`: `DomainLookupByName` (only `ErrNoDomain` reaches NotFound),
|
||||
`readMetadata` (`DomainGetMetadata` with `metadataURI`;
|
||||
`ErrNoDomainMetadata` means unmanaged with an empty UID; an element that
|
||||
does not parse is Permanent), `inspect` (`DomainGetState`, then for an
|
||||
active domain `address`). `addressSources` asks the guest agent first,
|
||||
then ARP; a source that fails is skipped at V(2), only ctx ending is an
|
||||
error, so a cancelled Get is Transient and not a false Provisioning.
|
||||
`firstIPv4` drops loopback, link-local and unspecified addresses and an
|
||||
address whose type tag and literal disagree. `UID` and `CreatedAt` are
|
||||
read back on every call; an unparsable `createdAt` gives the zero time.
|
||||
- **Delete.** Under the per-name lock: `removeDomain` (`DomainDestroy`,
|
||||
tolerating not running and gone; `DomainUndefineFlags` with NVRAM,
|
||||
managed save and snapshot metadata) and `removeVolumes` (overlay and seed
|
||||
by name through `deleteVolumeIfPresent`; a missing pool is a no-op). A
|
||||
host no longer in the config is a logged no-op success. Nothing is left
|
||||
when nil is returned.
|
||||
- **ListByTag.** `ConnectListAllDomains` (active and inactive) on every
|
||||
host in config order; `readMetadata` and `managed` keep kvm2's own
|
||||
domains; `inspect` fills state and address. A domain whose metadata or
|
||||
state cannot be read is skipped at V(1), unless ctx ended, which fails
|
||||
the call. A host that cannot be dialled or listed fails the whole call.
|
||||
- **Errors.** `classify(op, err)`: an error already carrying a class keeps
|
||||
it; libvirt codes through `classifyLibvirt` (`ErrNoDomain` and
|
||||
`ErrNoStorageVol` NotFound, Transient when `op == "create"`;
|
||||
`ErrNoStoragePool`, `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`,
|
||||
`ErrAuthFailed`, `ErrAccessDenied` Permanent; the rest Transient);
|
||||
`knownhosts.KeyError` and the `unable to authenticate` text from
|
||||
`x/crypto/ssh` Permanent; everything else Transient, including a missing
|
||||
credential file. `permanentf` and `transientf` classify at the source:
|
||||
placement, XML builders, a parse failure of the key or known_hosts, an
|
||||
image 4xx, a missing or zero `Content-Length`, a sha256 mismatch, a pool
|
||||
that is not running (Permanent); an image 429 or 5xx, a short body, the
|
||||
running wait expiring (Transient). `checkPoolSpace` is the only source of
|
||||
`ErrQuotaExceeded`. `logAPIError` writes `kvm2 backend call failed` with
|
||||
`libvirtErrorFields`.
|
||||
- **Check.** `Check` runs `checkHost` per host concurrently, each through
|
||||
`probeHost` on a `dialFresh` connection that is closed when the probe
|
||||
ends; the pooled table is never consulted. `ConnectGetLibVersion`,
|
||||
`ConnectGetHostname`, `StoragePoolLookupByName`, `StoragePoolGetInfo`;
|
||||
a pool that is not running is `poolNotActiveError`. Details are filled
|
||||
as facts arrive (`user` and `pool` always; `libvirtVersion`, `hostname`,
|
||||
`poolState`, `poolCapacityGiB`, `poolAvailableGiB`). `checkReason`: a
|
||||
`dialError` is judged by its stage (`credentials` gives
|
||||
`CredentialsUnavailable`; `ssh` with a `knownhosts.KeyError` or the auth
|
||||
text gives `Unauthorized`; any other stage, including a dial that ran
|
||||
out of time, gives `Unreachable`); then `context.DeadlineExceeded` or
|
||||
`Canceled` `Timeout`, `ErrAccessDenied` or `ErrAuthFailed` `Forbidden`,
|
||||
`ErrNoStoragePool` `PoolNotFound`, any other libvirt code `APIError`,
|
||||
anything else `Unreachable`; `PoolNotActive` from the pool state.
|
||||
- **Connections.** `hostTable` keeps one pooled connection per host,
|
||||
dialled lazily under a per-host one-slot semaphore so waiting ends with
|
||||
the caller's ctx; a dead backend (`liveness`) is closed and redialled on
|
||||
the next `get`. `dialSSHSocket` runs credentials, TCP, the SSH handshake
|
||||
and the `direct-streamlocal` channel to the socket, each failure a
|
||||
`dialError` with its stage; `libvirtDialer.connect` bounds the whole
|
||||
sequence plus the libvirt `Connect` with the dial budget and starts the
|
||||
`keepalive` (15 s, `keepalive@openssh.com`) and a watcher on go-libvirt's
|
||||
`Disconnected`. `call` runs each blocking go-libvirt call in a goroutine
|
||||
and waits on the result, ctx and a budget timer; abandoning a call
|
||||
(`link.markDead`) closes the socket and the SSH client, returns Transient
|
||||
wrapping the ctx or deadline error, and every other call waiting on that
|
||||
link fails once with Transient. `loadCredentials` runs on every dial.
|
||||
- **Logging.** Logger from the context, named `kvm2`, with `provider`,
|
||||
and `host` and `name` on lines about an instance. V(1): one line per
|
||||
operation outcome (`kvm2 domain ensured running`, `kvm2 domain fetched`,
|
||||
`kvm2 domain deleted`, `kvm2 domains listed`, `kvm2 host checked`), one
|
||||
`libvirt call` line per RPC from `observe` (rpc name, identifiers,
|
||||
duration, error fields), the connection events `libvirt host connected`
|
||||
and `libvirt host connect failed` with `stage`, and `kvm2 backend call
|
||||
failed`. V(2): XML, metadata, addresses, listed names, pool sizes, the
|
||||
resolved Create parameters. Upload and download bodies, cloud-init and
|
||||
key material are never logged.
|
||||
- **Tracing.** `observe.call` starts `kvm2.<Method>` per RPC with
|
||||
`kvm2.host` and the identifiers (`kvm2.domain`, `kvm2.pool`,
|
||||
`kvm2.volume`), sets `kvm2.libvirt_code` on failure and writes the
|
||||
`libvirt call` line inside the span; `libvirtDialer.dial` starts
|
||||
`kvm2.dial` (`kvm2.address`, `kvm2.user`); `fetchImage` starts
|
||||
`kvm2.download` (`kvm2.url_host`, `kvm2.sha256_configured`,
|
||||
`kvm2.bytes`). Both the decorator and the dialer take `tracing.Option`
|
||||
so tests inject a recorder; `New` leaves them on the global provider.
|
||||
- **Tests.** Pure builders (`xml_test.go`, `volume_test.go`,
|
||||
`seed_test.go`, `id_test.go`, `metadata_test.go`, `state_test.go`);
|
||||
`errors_test.go` (the classify table with real SSH error strings and a
|
||||
real `knownhosts.KeyError`); `backend_test.go` (the adapter over
|
||||
go-libvirt's own mock through a `net.Pipe`: deadline kills the
|
||||
connection, cancel unblocks, killed by another caller); `hosts_test.go`;
|
||||
`ssh_test.go` over `sshserver_test.go` (every dial stage, host-key
|
||||
policies, key material never in errors or logs); `image_test.go`
|
||||
(marker rules, streaming asserted by counting bytes, dedicated upload
|
||||
connection); `create_test.go`, `get_test.go`, `delete_test.go`,
|
||||
`list_test.go`, `kvm2_test.go`, `provider_test.go` (the contract §11
|
||||
list, named test by test in the spec's entry 21); `observe_test.go`
|
||||
(every seam method spanned and logged exactly once, by reflection);
|
||||
`check_test.go` (the reason table, never touches the pool). The double is
|
||||
`fakeBackend` in `fake_test.go`, whose `twin` shares state under its own
|
||||
call log so a test can tell the pooled connection from a dedicated one.
|
||||
Real backend: `kvm2_integration_test.go` behind `//go:build integration`,
|
||||
driven by `KVM2_TEST_*` variables and `make test-integration-kvm2`,
|
||||
described in
|
||||
[testing.md](testing.md#the-kvm2-integration-test-real-hypervisor).
|
||||
- **Deployment.** The `kvm-ssh` Secret and its mount are shared with kvm
|
||||
(`config/manager/manager.yaml` says so), `make kvm-ssh-secret` is reused,
|
||||
a commented `kvm2-home` entry in both providers configs,
|
||||
`config/samples/proxy_kvm2.yaml` (zone `beelink`, matching the shipped
|
||||
config), the `kvm2` type in the `status.type` comment of
|
||||
`api/v1alpha1/providerhealth_types.go`. No RBAC.
|
||||
- **Known gaps** (see also [kvm2-provider.md § Known gaps](kvm2-provider.md#known-gaps)).
|
||||
- `Create` blocks, including the first image download on each host.
|
||||
- GC blind spots: volumes without a domain are invisible to GC, and so are
|
||||
VMs on a host removed from the config, which `Delete` treats as done.
|
||||
- No adoption of kvm's domains: a switched entry needs its Proxies
|
||||
deleted first.
|
||||
- Cancelling one operation cuts off the others on that host once: a
|
||||
cancelled `call` kills the pooled connection.
|
||||
- `Content-Length` is required for image downloads.
|
||||
- A paused or pmsuspended domain cannot be adopted by a repeat `Create`:
|
||||
`DomainCreate` refuses, `waitRunning` expires, Transient on every retry.
|
||||
- Credential parse errors carry the parser's own text, which can quote a
|
||||
PEM header type or a malformed known_hosts token (never key bytes).
|
||||
- No reused-overlay check: an existing overlay is reused without checking
|
||||
its backing image or size.
|
||||
- The sample cloud-init installs Squid with `http_access allow all`.
|
||||
|
||||
## 8. Gaps shared by the existing providers
|
||||
|
||||
- **Adoption is not verified in three of the four.** kubernetes, gcp and
|
||||
kvm adopt any existing instance with the right name without checking
|
||||
`LabelUID`; kvm2 returns `ErrPermanent` when the UID differs.
|
||||
- **The Create comment does not match kvm and kvm2.** The interface comment
|
||||
in `internal/provider/provider.go` says `Create` returns as soon as the
|
||||
request is submitted; both libvirt providers block until the domain has
|
||||
started.
|
||||
- **`Stopped` and `Terminated` are produced inconsistently.** kubernetes
|
||||
never produces Stopped, kvm never produces Terminated, and the reconciler
|
||||
treats them the same.
|
||||
never produces Stopped, kvm and kvm2 never produce Terminated, and the
|
||||
reconciler treats them the same.
|
||||
- **Different `ListByTag` partial-failure policies.** gcp leaves failed
|
||||
parts out, kvm fails the whole list. Both are safe.
|
||||
parts out, kvm and kvm2 fail the whole list. Both are safe.
|
||||
- **Stale `mock` comments** in `internal/provider/provider.go`,
|
||||
`internal/provider/registry/registry.go` and `api/v1alpha1/proxy_types.go`
|
||||
(and so the generated CRD's `spec.provider` description).
|
||||
@@ -351,33 +542,41 @@ the author's view.
|
||||
is still open.
|
||||
- **Tracing.** kvm has no child spans, so a kvm trace shows `provider.create`
|
||||
and nothing under it; correlation works only through `traceID` on its V(1)
|
||||
log lines. Only kubernetes is tested against a real collector, and that
|
||||
test checks only the operation spans; nothing checks that gcp's SDK spans
|
||||
or the kubernetes API spans nest under `provider.*`.
|
||||
log lines. kvm2 has them, but only an in-memory recorder checks their
|
||||
topology. Only kubernetes is tested against a real collector, and that
|
||||
test checks only the operation spans; nothing checks that gcp's SDK spans,
|
||||
the kubernetes API spans or kvm2's spans nest under `provider.*` in a
|
||||
real collector.
|
||||
|
||||
## 8. Proposed features
|
||||
## 9. Proposed features
|
||||
|
||||
Listed only. Nothing below is designed or implemented.
|
||||
Listed only. Nothing below is designed or implemented, except where a
|
||||
bullet says kvm2 already does it.
|
||||
|
||||
- **Child spans in kvm:** extend the `tracingAPI` decorator in
|
||||
`kvm/trace.go`, which already wraps every libvirt call, to also start a
|
||||
span per RPC, and add spans for the SSH dial and the image download. The
|
||||
seam's methods take no `ctx` today (`hostPool.call` hands `fn` only the
|
||||
API), so the span context has to be plumbed in.
|
||||
- **Tempo e2e checks for child spans**, and for gcp and kvm through their
|
||||
real backend runbooks.
|
||||
API), so the span context has to be plumbed in. kvm2's `observe.go` over
|
||||
its ctx-first seam is the shape this would take.
|
||||
- **Tempo e2e checks for child spans**, and for gcp, kvm and kvm2 through
|
||||
their real backend runbooks.
|
||||
- **A shared conformance suite:** for example `internal/provider/providertest`,
|
||||
which runs the contract §11 tests against any provider plus a fake
|
||||
backend, so a new provider gets the contract tests for free.
|
||||
- **Verify the UID when `Create` adopts an instance.** Return `ErrPermanent`
|
||||
on a mismatch instead of adopting an instance that belongs to someone else.
|
||||
kvm2 does this; the other three do not.
|
||||
- **Report asynchronous create failures:** gcp keeps the operation name, and
|
||||
`Get` reports the failure instead of NotFound.
|
||||
- **Capacity or quota signals** for kvm, such as pool free space and host
|
||||
memory, mapped to `ErrQuotaExceeded`.
|
||||
memory, mapped to `ErrQuotaExceeded`. kvm2 has the pool-space precheck;
|
||||
host memory is still unchecked in both.
|
||||
- **GC for secondary resources:** mark volumes and similar resources, and
|
||||
let `ListByTag` or a second listing method cover them.
|
||||
- **Reload credentials and config without a restart.**
|
||||
- **Reload credentials and config without a restart.** kvm2 re-reads its
|
||||
SSH material on every dial; the providers config is still read once per
|
||||
process for every type.
|
||||
- **Checker `Details`** for kubernetes (server version) and gcp (quota
|
||||
headroom).
|
||||
- **Internal-IP mode** for gcp, and a firewall rule managed per provider.
|
||||
|
||||
+5
-3
@@ -13,11 +13,13 @@ the existing ones happen to work.
|
||||
a new provider's design answers, one entry per decision the contract leaves
|
||||
open. Copy it to `docs/<type>-provider-spec.md`, fill it in, and hand it to
|
||||
the implementer with the other two files.
|
||||
- **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp and
|
||||
kvm meet the contract, a table mapping each of them to every profile, their
|
||||
per-provider reference notes, shared gaps and proposed features.
|
||||
- **[providers-existing.md](providers-existing.md)**: how kubernetes, gcp,
|
||||
kvm and kvm2 meet the contract, a table mapping each of them to every
|
||||
profile, their per-provider reference notes, shared gaps and proposed
|
||||
features.
|
||||
|
||||
Per-provider user guides: [kvm-provider.md](kvm-provider.md),
|
||||
[kvm2-provider.md](kvm2-provider.md),
|
||||
[gcp-vm-validation.md](gcp-vm-validation.md) and
|
||||
[gcp-in-specific-project.md](gcp-in-specific-project.md).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user