kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fcbf8ddc91b7ef6ffb1db418100d13c7d4aaccac
egress-proxies-operator/docs/kvm2-provider.md
T

598 lines
34 KiB
Markdown
Raw Blame History

# The kvm2 provider
Runs Managed proxies as VMs on your own libvirt/KVM hypervisors. It is the
second-generation libvirt provider, built clean-room from
[provider-contract.md](provider-contract.md) and the design in
[kvm2-provider-spec.md](kvm2-provider-spec.md), and it ships side by side
with the first-generation `kvm` provider: both types can be configured at
once, both take the same config block shape, and both read the same
`kvm-ssh` Secret. What kvm2 adds over kvm: a child span per libvirt RPC,
a pool-space precheck that maps to the reconciler's quota wait, SSH
credentials re-read on every dial, image downloads and health checks on
connections of their own, an adoption guard on the owning UID, and SSH
authentication failures that stop retrying.
Like kvm, the operator reaches each host over SSH, forwards the libvirt
unix socket through that connection and speaks libvirt's own RPC over it,
so the hosts need nothing beyond `sshd` and `libvirtd` and the operator
image carries no libvirt libraries. VMs boot a stock distro cloud image
that the operator downloads into the host's storage pool on first use,
with the Proxy's cloud-init delivered on a NoCloud seed ISO built
in-process.
## Hypervisor prerequisites
Per host, once:
1. **libvirt and KVM** (`libvirtd` running, `/dev/kvm` present).
2. **An SSH user in the `libvirt` group**, so it may open
`/var/run/libvirt/libvirt-sock`. The operator authenticates with a
private key; nothing else on the host is touched. The socket is used
as `qemu:///system`, so no further host permissions are needed.
3. **A bridge to a network with DHCP** (for example `br0` bridged onto
the LAN). Each VM gets one virtio NIC on it and takes its IP from that
network's DHCP. The operator's pod network and the crawler clients
must route to those IPs: the health engine probes `http://<ip>:<port>`
through the proxy and clients get the same address from the discovery
API. Nothing on the host filters the port; the provider does not use
`spec.port` at all.
4. **A storage pool** (default `default`, usually `/var/lib/libvirt/images`)
with room for one base image per distro (a few hundred MB) plus the
per-VM overlays (`diskSizeGb` each, thin-provisioned). The pool must be
running: a stopped pool fails the health check with `PoolNotActive` and
fails `Create` permanently.
5. **UEFI firmware for guests**: the `ovmf` package on Debian/Ubuntu,
`edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default with secure boot
off (libvirt picks the OVMF binary itself); a domain start fails with
"no firmware found" without it. Only `firmware: bios` images skip this.
6. **Outbound internet from the VMs**, in practice: a vendor-data script
installs `qemu-guest-agent` from the distro mirrors on first boot, and
the agent is how the operator learns the VM's address. Without it the
operator falls back to the host's ARP table, which only sees traffic
addressed to the host itself, so on a plain bridge a VM without the
agent stays `Provisioning`.
Check from the operator's point of view before wiring it in:
```sh
ssh novakj@beelink virsh -c qemu:///system list --all # user can drive libvirt
ssh novakj@beelink virsh -c qemu:///system pool-info default
ssh novakj@beelink ip link show br0
```
## Cluster-side setup
1. Create the SSH Secret from a key pair dedicated to the operator and a
`known_hosts` file holding every host's key. It is the same `kvm-ssh`
Secret the `kvm` provider uses; one Secret serves both types, and if
you already have it there is nothing to do here.
```sh
# once: a key pair for the operator; the .pub goes on every host
ssh-keygen -t ed25519 -f ~/.ssh/proxy-operator_ed25519 -N '' -C proxy-operator
ssh-copy-id -i ~/.ssh/proxy-operator_ed25519.pub novakj@beelink
# once per host: its host key, under the address the *operator* will
# dial (hosts[].address), which may differ from what your laptop uses
ssh-keyscan -H beelink.tail8c69c.ts.net >> ~/.ssh/kvm_known_hosts
# the Secret: the PRIVATE key (not the .pub) plus known_hosts
kubectl -n egress-proxies-operator-system create secret generic kvm-ssh \
--from-file=id_ed25519=$HOME/.ssh/proxy-operator_ed25519 \
--from-file=known_hosts=$HOME/.ssh/kvm_known_hosts
```
`make kvm-ssh-secret` runs that last command as an apply, so it also
updates an existing Secret; override the paths with
`KVM_SSH_KEY=... KVM_KNOWN_HOSTS=...`. Putting the `.pub` in the
Secret is the classic mistake: the target's `kvm-ssh-secret` check
refuses a file without `PRIVATE KEY` in it, and the provider itself
would fail every call permanently with a "parsing ssh key file" error.
Without `known_hosts`, set `insecureIgnoreHostKey: true` and accept
that a MITM could hand you a fake hypervisor.
`config/manager/manager.yaml` mounts the Secret (optional) at
`/etc/proxy-operator/kvm` as two files, `id_ed25519` and
`known_hosts`, which is what `sshKeyFile` and `knownHostsFile` in the
provider config point at. The deployment starts fine without the
Secret, even with a kvm2 provider configured: `New` only checks that
the files exist and logs a `WARNING:` line when one does not; the
provider's proxies fail transiently and recover once the Secret lands.
**Rotating the key or a host key needs no restart.** Both files are
read and parsed on every dial (`loadCredentials` in
`internal/provider/kvm2/ssh.go`); nothing is cached. Re-run the
command or the make target, and the next connection, health check
included, uses the new material. Connections that are already open
stay open until they drop.
2. Add a provider entry to the providers config
(`config/manager/providers_config.yaml` in-cluster; full sample in
`config/samples/providers-config.yaml`). The type is `kvm2` and the
block is `kvm2:`; its fields are those of the `kvm` block:
```yaml
- name: kvm2-home
type: kvm2
kvm2:
sshKeyFile: /etc/proxy-operator/kvm/id_ed25519
knownHostsFile: /etc/proxy-operator/kvm/known_hosts
images:
debian-13:
url: https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2
machineTypes:
small: {vcpus: 1, memoryMiB: 1024}
hosts:
- name: beelink
address: beelink.tail8c69c.ts.net:22
user: novakj
bridge: br0
```
Validation messages name the block, so a mistake reads
`kvm2.hosts[0]: bridge is required`, not `kvm.hosts`. The providers
config is read once per process: adding a host or an image entry needs
a manager restart; credentials do not.
3. Apply a Proxy (`config/samples/proxy_kvm2.yaml`) and watch it:
```sh
kubectl get px -w
# NAME MODE PROVIDER PHASE IP HEALTHY
# proxy-kvm2-sample Managed kvm2-home Provisioning
# proxy-kvm2-sample Managed kvm2-home Running 192.168.1.73
# proxy-kvm2-sample Managed kvm2-home Ready 192.168.1.73 True
```
The first Proxy for a given image on a given host sits in
`Provisioning` for the length of the image download; the operator log
shows `fetching base image into host pool` with the URL and size and
`base image cached` with the size, duration and digest.
## Config reference
The block is `kvm2:` and its Go type is the shared `KVMConfig` in
`internal/provider/config_kvm.go`, so the fields, defaults and validation
are the same as for `kvm`. Defaults are applied by `withDefaults` in
`internal/provider/kvm2/kvm2.go`.
| Field | Required | Default | Meaning |
|---|---|---|---|
| `sshKeyFile` | yes | | private key used for every host; OpenSSH or PEM, parsed on every dial |
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against, read on every dial |
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely; `New` logs a `WARNING:` line when it is on |
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it. Also what the pool-space precheck reserves per VM |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on |
| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table |
| `images.<key>` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only: it becomes the cached volume name `base-<key>.<format>` |
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw). The server must send `Content-Length` |
| `images.<key>.sha256` | | | expected digest, 64 hex characters; verified after download, a change triggers a re-download |
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
| `images.<key>.firmware` | | `uefi` | `uefi` (OVMF, secure boot off) or `bios` (SeaBIOS) for images without an EFI system partition |
| `machineTypes.<key>` | at least one | | `{vcpus, memoryMiB}` size profile, both greater than zero |
| `hosts[].name` | yes | | the `placement.zone` value that selects this host; unique. Letters, digits, dots, dashes and underscores only: it becomes part of the providerID `hosts/<name>/domains/<vm>` |
| `hosts[].address` | yes | | SSH endpoint, `host:port` |
| `hosts[].user` | yes | | SSH user, member of the `libvirt` group |
| `hosts[].socket` | | `/var/run/libvirt/libvirt-sock` | libvirt socket path on the host |
| `hosts[].pool` | | `default` | storage pool for base images, overlays and seeds |
| `hosts[].bridge` | yes | | host bridge for the VM's NIC |
## Placement
All three fields are checked before any host is dialled. An unknown value
is `ErrPermanent`, which the reconciler latches as `Failed` with the
message below until the spec changes.
| Proxy field | Meaning for kvm2 | Error when unknown |
|---|---|---|
| `placement.zone` | host name from `hosts`. May be omitted when exactly one host is configured; otherwise required | `placement.zone "x" is not in kvm2.hosts`; with several hosts and no zone, `placement.zone is required when kvm2.hosts lists more than one host` |
| `placement.machineType` | key in `machineTypes` | `placement.machineType "x" is not in kvm2.machineTypes` |
| `placement.image` | key in `images`, or an `http(s)://` URL used directly with the defaults (`qcow2`, `uefi`) | `placement.image "x" is neither a key in kvm2.images nor an http(s) URL` |
| `placement.region` | ignored. It is still part of the spec hash, so changing it replaces the VM and does nothing else | |
The messages come from `Create` in `internal/provider/kvm2/create.go`,
`host` in `kvm2.go` and `resolveImage` in `image.go`. Fix the spec (which
replaces the VM) or the config. Catalogue edits affect only new VMs: the
spec hash covers the placement values, not what the config maps them to.
## What lands on the host
For a Proxy whose deterministic name is `proxy-abc…`, on the chosen
host's pool:
| Object | Name | Lifetime |
|---|---|---|
| base image | `base-<key>.<format>` (for a URL image, `<key>` is the first 16 hex characters of the URL's sha256) | cached; survives Proxy deletion |
| image marker | `base-<key>.<format>.sha256`, a 64-byte raw volume holding the digest | with the base image |
| overlay disk | `proxy-abc….qcow2`, qcow2 backed by the base file, `diskSizeGb` capacity | with the Proxy |
| seed ISO | `proxy-abc…-seed.iso`, raw | with the Proxy |
| domain | `proxy-abc…` | with the Proxy |
The names are built in `internal/provider/kvm2/volume.go`.
The domain (`buildDomainXML` in `xml.go`) is a `q35` machine with a
host-passthrough CPU, ACPI and APIC, memory in MiB and vcpus from the
machine type, the overlay on `vda` (virtio) and the seed as a read-only
SATA cdrom on `sda`, one virtio NIC on the bridge, a serial console on a
pty, the guest-agent channel `org.qemu.guest_agent.0`, a virtio RNG fed
from `/dev/urandom`, and no graphics. Both disks are `type="file"` with
the paths `StorageVolGetPath` returned: AppArmor derives QEMU's profile
from those paths, and volume-type disks fail with permission denied on
stock Debian and Ubuntu hosts. UEFI images get `<os firmware="efi">` with
the `secure-boot` feature disabled; BIOS images get neither.
Its `<metadata>` carries one element in kvm2's own namespace,
`https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2`
(`metadataURI` in `metadata.go`):
```xml
<instance xmlns="https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/kvm2">
<managed>true</managed>
<uid>0f3a…</uid>
<createdAt>2026-09-16T17:42:11Z</createdAt>
</instance>
```
That is how orphan GC recognises its own VMs, since libvirt has no
labels, and how a repeat `Create` knows whether an existing domain of the
same name belongs to this Proxy: a different non-empty `uid` is a
permanent error naming both UIDs; an empty one is adopted. The namespace
is kvm2's own, so the `kvm` provider's domains are invisible to it (see
*Migrating from kvm*).
The seed (`buildSeedISO` in `seed.go`) is an ISO 9660 image labelled
`cidata` with three files. `user-data` is the Proxy's `cloudInit`, **byte
for byte**: any format cloud-init accepts works, including a Jinja
template, a MIME archive, a shell script or a gzipped payload; the
operator never parses or re-renders it. `meta-data` holds
`instance-id: <name>` and `local-hostname: <name>`, so a reboot is never
taken for a new instance. `vendor-data` carries the guest-agent
bootstrap. The ISO is staged in a temp directory under `/tmp` (the
manager's 64Mi emptyDir), assembled in memory (tens of KiB) and uploaded
as a raw volume; the staging directory is removed before the call
returns. The seed is rebuilt on every `Create` attempt, because a
previous attempt may have been cut off mid-upload and a truncated seed
would boot a VM without cloud-init.
The guest agent rides along in `vendor-data` as a **shell script**
(`guestAgentScript` in `seed.go`), not a cloud-config, on purpose:
cloud-init merges a vendor-data cloud-config under the user-data with
mergers that keep the user-data's value for every top-level key both
define and never append lists, so a Proxy carrying its own `packages:`
would silently lose the agent. Scripts bypass the merge. The script is
idempotent, picks whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk`
the image has, enables the service with `systemctl` or OpenRC, and logs
into `/var/log/cloud-init-output.log`. With `vendorData` also configured,
the two travel as separate parts of a MIME multipart (`text/plain` for
your document, `text/x-shellscript` for the script), with the
`MIME-Version` header first so cloud-init detects it. With
`installGuestAgent: false` the file is your `vendorData` alone, or empty.
## The image cache
The base image is downloaded **by the operator, through libvirt**, into
the host's pool the first time a Proxy asks for it on that host. Nothing
is pre-placed on the hypervisor and the operator needs no filesystem
access there. The bytes stream from the URL through the operator into
`StorageVolUpload` with a running sha256; nothing is buffered
(`fetchImage` and `uploadImage` in `internal/provider/kvm2/image.go`).
The upload runs on a **dedicated connection** to the host, dialled for
the transfer and closed after it, not on the pooled connection the other
operations share. A cancelled or overlong call kills the connection it
ran on (see *Timeouts and connections*), and a poll for some other Proxy
being cancelled must not cut off a 20-minute upload.
The download needs a `Content-Length`: the volume's capacity is declared
before the upload starts. The request sends `Accept-Encoding: identity`
so a transparently gzipped body cannot hide the length. A response
without it, or with length 0, is a permanent error naming the URL, and so
is any 4xx status; 429 and 5xx are transient and retried.
The marker volume is written **last**, on the pooled connection, after
`StoragePoolRefresh` has made libvirt read the file's real format. Its
presence is what marks a base complete: a crash mid-download leaves a
base volume without a marker, which the next attempt deletes and redoes
(`cachedImage`). A re-download also happens when the marker's content is
not a 64-character hex digest, or when `sha256` is configured and differs
from it. Otherwise the cached copy is used for as long as it exists, even
if the URL's content changed upstream: a `latest` URL is pinned at the
moment of first fetch, per host. Concurrent Creates for the same image
on the same host queue on a per-(host, volume) lock; the first fetches,
the rest find the marker.
To evict manually:
```sh
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2.sha256
ssh novakj@beelink virsh -c qemu:///system vol-delete --pool default base-debian-13.qcow2
```
**Warning: do not change an image entry's `url` or `sha256`, and do not
evict its base volume, while Proxies using that image exist on the
host.** The overlays back onto the base file; deleting or replacing it
corrupts every VM that uses it. Delete those Proxies first, or add the
new image under a new key and move Proxies over by changing their
`placement.image` (which replaces the VM).
## Quota: the pool-space precheck
Before creating any volume, `Create` reads `StoragePoolGetInfo` and
compares the pool's free space with what it is about to commit
(`checkPoolSpace` and `neededBytes` in `image.go`): the image's
`Content-Length` when the base is not cached yet (found with a `HEAD`
request, or a `GET` whose body is closed at once when the server refuses
`HEAD`), plus `diskSizeGb` in bytes for the overlay, plus 16 MiB for the
seed. Below that it returns `ErrQuotaExceeded` with a message naming the
pool, the bytes available and the bytes needed. The reconciler answers
quota with a wait, not a failure: the Proxy shows `Provisioned=False`
with reason `QuotaExceeded` and is retried after `QuotaRetry` (5
minutes), off the error backoff curve and never `Failed`. Free up the
pool or lower `diskSizeGb` and the next attempt goes through.
The precheck is the only quota signal; libvirt itself reports a full pool
as an ordinary error, which would be transient. The health check exposes
the same number as `poolAvailableGiB`, so a pool that is filling up shows
there before Creates start waiting.
A pool that is not running fails the precheck permanently (`storage pool
<pool> is not running`): start it with `virsh pool-start` and
`virsh pool-autostart`.
## Migrating from `kvm`
The `kvm2` block is the `kvm` block under another name, so an entry ports
by changing `type: kvm` to `type: kvm2` and `kvm:` to `kvm2:`. The
`kvm-ssh` Secret is shared; nothing changes on the hosts.
What does not port is the VMs. kvm2 marks its domains in its own metadata
namespace and neither lists nor adopts domains the `kvm` provider made:
`ListByTag` skips them, so orphan GC never sees them, and a Proxy's
`spec.provider` is immutable, so an existing Proxy cannot be moved to the
new entry anyway. Do it in this order:
1. Delete the Proxies on the `kvm` entry and wait for them to go; the
`kvm` provider removes their domains and volumes.
2. Change the entry (or add a second one under a new name and keep the
old one for a while; both types may share a host).
3. Restart the manager, then recreate the Proxies with the new
`spec.provider`.
A `kvm` domain left behind on a host is invisible to kvm2 forever; clean
it up on the host with `virsh undefine --nvram <name>` and
`virsh vol-delete` for its `<name>.qcow2` and `<name>-seed.iso`.
## Logging
The provider logs through controller-runtime's logger, named `kvm2`, with
`provider` (the configured name) on every line, and `host` and `name`
(the domain) on every line about an instance. Every line inside a traced
operation also carries `traceID` and `spanID`.
- `--zap-log-level=debug` (logr V(1)):
- one line per operation outcome: `kvm2 domain ensured running` with the
`id`, `kvm2 domain fetched` with `state`, `ip` and `uid`,
`kvm2 domain not found`, `kvm2 domain deleted`, `kvm2 domains listed`
with `hosts` and `managed` counts, `kvm2 host checked` with `ready`,
`reason` and `latency`; plus the steps inside a Create
(`kvm2 domain already defined`, `overlay volume created`,
`seed volume uploaded` with `bytes`, `domain defined` with `uid` and
`firmware`, `domain started`, `fetching base image into host pool`,
`base image cached`);
- **one `libvirt call` line per libvirt RPC**, written by the decorator
in `internal/provider/kvm2/observe.go` inside that RPC's span: `rpc`
(the go-libvirt method name, for example `DomainLookupByName`), the
identifiers it was given (`domain`, `pool`, `volume`), `duration`, and
on failure `error` plus `libvirtCode` and `libvirtError` (the code's
name, for example `ErrNoDomain`). A Get is four or five RPCs; a
Create with a cached image is around twenty;
- connection events from `internal/provider/kvm2/hosts.go`:
`libvirt host connected` with `address`, `user` and `duration`, and
`libvirt host connect failed` with the same plus `error` and `stage`
(`credentials`, `tcp`, `ssh` or `socket`);
- `kvm2 backend call failed` with `op` and the raw `error` once per
failed operation, before the error is returned;
- a line per domain that `ListByTag` skips, and a line when the image
marker forces a refetch.
- `--zap-log-level=2`: the `libvirt call` lines also carry the domain and
volume `xml`, the `metadata` element, the interface `addresses`, the
listed `domains`, the pool `state` and sizes and the volume `path`; the
`domain defined` line carries the `xml`; `kvm2 create resolved` logs the
resolved parameters (pool, bridge, machine type, sizes, image volume,
firmware) and `cloudInitBytes`; `kvm2 listed instance` logs each listed
domain.
- Never logged at any level: the cloud-init content, the SSH key, the
known_hosts content, and upload or download bodies. The seed and image
uploads report `offset` and `length` only. There is no raw wire tier
like the GCP provider's V(5); go-libvirt exposes no hook for it.
## Tracing
With an OTLP exporter configured ([architecture.md
§10](architecture.md#10-tracing-internaltracing)), every backend call is
a child span of the operation span the decorators open
(`provider.create`, `provider.get`, `provider.delete`, `provider.list`)
or of the monitor's `providerhealth.check`:
| Span | Around | Attributes |
|---|---|---|
| `kvm2.<Method>`, for example `kvm2.DomainDefineXML` | one libvirt RPC (`observe.go`) | `kvm2.host`, and the identifiers passed: `kvm2.domain`, `kvm2.pool`, `kvm2.volume`; on failure the status is Error and `kvm2.libvirt_code` carries the libvirt error number |
| `kvm2.dial` | the SSH handshake and the libvirt connect (`hosts.go`) | `kvm2.host`, `kvm2.address`, `kvm2.user` |
| `kvm2.download` | one image fetch, from the HTTP request to the marker (`image.go`) | `kvm2.host`, `kvm2.volume`, `kvm2.url_host`, `kvm2.sha256_configured`, and `kvm2.bytes` once the response arrived |
All spans are client spans. No attribute carries cloud-init, key material
or a URL beyond its host. The `libvirt call` log line for an RPC is
written inside that RPC's span, so its `spanID` points at the span.
## Timeouts and connections
No caller puts a deadline on the context, so every backend call carries
its own budget (`defaultBudgets` in `internal/provider/kvm2/backend.go`,
applied by `call`):
| Call | Budget |
|---|---|
| TCP connect plus SSH handshake plus libvirt connect | 10 s |
| ordinary RPC | 30 s |
| `DomainCreate` | 90 s |
| `StorageVolUpload` of at most 16 MiB (the seed, the marker) | 60 s |
| `StorageVolUpload` of an image, `StorageVolDownload`, and the whole image fetch from HTTP request to marker | 20 m; the caller's ctx may end it earlier |
| wait for running after `DomainCreate` | 2 m, polled every 500 ms (`defaultRunningWait` and `defaultRunningPoll` in `create.go`) |
| the `HEAD` request of the pool-space precheck | 30 s |
| the courtesy `ConnectClose` when a connection is closed | 5 s |
go-libvirt's calls take no context and block, so the only way to abandon
one is to close the connection under it. That is what a cancelled or
overlong call does: the adapter kills the connection (socket and SSH
client), returns `ErrTransient` wrapping the context or deadline error,
and marks the connection dead so the next call redials. The consequence:
**other RPCs in flight on that host's pooled connection fail once with
`ErrTransient`** and their callers retry on a fresh connection. In the
reconciler that is one requeue with backoff; in GC it is one skipped
sweep.
Two callers are kept off the pooled connection for that reason
(`dialFresh` in `hosts.go`): the image upload, because a 435 MB transfer
must not die because a poll for another Proxy was cancelled, and the
health check, because a check that runs out of the monitor's timeout must
close only its own connection.
Otherwise there is one pooled SSH plus libvirt connection per host,
dialled lazily on the first call that needs it and shared by every
operation for that host (go-libvirt multiplexes RPCs on one connection).
Concurrent callers wait for one dial and share the result; waiting ends
with the caller's context. A `keepalive@openssh.com` request every 15 s
marks the connection dead when it fails or gets no reply within another
15 s, as does go-libvirt noticing the socket is gone; the next call
redials. Create and Delete for the same domain name are serialised on a
per-name lock, so two Deletes (GC and the reconciler) never race on
destroy and undefine, and a Delete cannot interleave with a Create's
steps; Get and ListByTag take no lock.
## Health check
The operator checks every configured host every `--provider-check-interval`
(default 1m) and publishes the result in the `ProviderHealth` object named
after the provider (`kubectl get ph kvm2-home -o yaml`) and in the
`proxy_operator_provider_target_up{provider,target}` gauge. One target per
`hosts[]` entry, in config order, all checked concurrently.
Each check dials a **fresh connection** to the host and closes it when
done (`probeHost` in `internal/provider/kvm2/check.go`); the pooled
connection is never touched, so a check that runs out of time cannot fail
a reconcile's call in flight, and a fresh dial picks up a rotated Secret
at once. The sequence is the SSH and libvirt handshakes, then four RPCs:
`ConnectGetLibVersion`, `ConnectGetHostname`, `StoragePoolLookupByName`
and `StoragePoolGetInfo` on the host's `pool`. The cost per host per
round is one sshd session and those four calls.
Every target's `details` carry `user` and `pool`. As facts arrive they
gain `libvirtVersion` (decoded `major.minor.micro`), `hostname`,
`poolState` (virsh's word: `running`, `inactive`, ...),
`poolCapacityGiB` and `poolAvailableGiB`, so a target that fails late in
the sequence still says what the host answered before. The free space is
the number the quota precheck compares against.
A failing target says where the check broke (`checkReason` in `check.go`):
| Reason | Meaning | Look at |
|---|---|---|
| `Reachable` | ready | |
| `CredentialsUnavailable` | the key or known_hosts file is missing or does not parse | the `kvm-ssh` Secret (see *Cluster-side setup*); fixed in the cluster, not on the host |
| `Unreachable` | the TCP connect failed or ran out of the 10 s dial budget, the SSH transport failed for a reason other than authentication, the forwarded libvirt socket was refused, or the link died mid-check | DNS, Tailscale or a firewall between the cluster and `address`; `libvirtd` running and the `socket` path |
| `Unauthorized` | the SSH server rejected the key, or the host key is missing from known_hosts or has changed | `user`'s `authorized_keys`; the known_hosts entry (re-run `ssh-keyscan` and update the Secret) |
| `Forbidden` | SSH worked, but libvirt refused the user (`ErrAccessDenied` or `ErrAuthFailed`) | `user` in the `libvirt` group |
| `Timeout` | the connection was up, but the monitor's `--provider-check-timeout` (default 15s) or an RPC's budget ran out before the check finished | a hung `libvirtd` or a saturated link |
| `PoolNotFound` | there is no pool named `pool` | `virsh pool-list --all` |
| `PoolNotActive` | the pool exists but is not running | `virsh pool-start <pool>` and `virsh pool-autostart <pool>` |
| `APIError` | any other libvirt error | the message carries libvirt's own text |
## Error classes
Every error is classified once, in `classify` in
`internal/provider/kvm2/errors.go`, with errors that carry a class from
their source (validation, the image fetch, the dial) keeping it. The
reconciler latches `ErrPermanent` as `Failed` until the spec changes,
waits 5 minutes on `ErrQuotaExceeded`, recreates on `ErrNotFound` from
`Get`, and retries everything else with backoff.
| Class | What lands there |
|---|---|
| `ErrPermanent` | placement values not in the catalogue; a malformed providerID; a host no longer in the config (from `Get`; `Delete` treats it as done); libvirt `ErrXMLError`, `ErrNoSupport`, `ErrInvalidArg`, `ErrAuthFailed`, `ErrAccessDenied` and `ErrNoStoragePool`; a pool that is not running; SSH authentication refused (`unable to authenticate`); a host key missing from known_hosts or changed (`knownhosts.KeyError`); a key or known_hosts file that exists but does not parse; a domain of the same name owned by another UID; corrupt metadata on a domain; an image URL answering 4xx, a response without `Content-Length` or with length 0, a sha256 that does not match the configured one |
| `ErrQuotaExceeded` | the pool-space precheck |
| `ErrNotFound` | libvirt `ErrNoDomain` or `ErrNoStorageVol` from `Get`, `Delete` or `ListByTag`; inside `Create` the same codes are `ErrTransient`, because the contract forbids NotFound from Create |
| `ErrTransient` | everything else: a missing key or known_hosts file (the Secret has not landed yet), TCP and network errors, context cancellation, an abandoned or budget-exceeded call, a connection killed by another caller, libvirt `ErrOperationInvalid` and every unrecognised libvirt code, an image URL answering 429 or 5xx or ending the body early, the wait for running expiring |
Error strings name paths, hosts, volumes and libvirt's own text. They
never contain key bytes, the known_hosts content or cloud-init.
## Known gaps
- **Create blocks**, including the first image download on each host,
holding one of the three reconcile workers for its duration (LAN
speed, a few hundred MB, once per image per host). Other Proxies keep
reconciling on the remaining workers. Warm the cache with one Proxy per
image per host before creating many.
- **Volumes without a domain are invisible to GC.** A crash between
creating the overlay or seed and defining the domain leaves volumes
that `ListByTag` cannot see (it lists domains). The same Proxy's next
Create reuses the overlay and rebuilds the seed, and its Delete removes
both by name even with no domain, but a Proxy CR that is gone for good
leaves them for `virsh vol-list` to find.
- **A host removed from the config orphans its VMs.** `Delete` for a
providerID naming an unconfigured host is a logged no-op success, so
the Proxy finishes deleting instead of pinning its finalizer, but the
domain and its volumes stay on the hypervisor and orphan GC cannot see
them either (it only sweeps configured hosts). Delete the Proxies before
retiring a host, or clean up afterwards with `virsh undefine --nvram`
plus `virsh vol-delete`. `Get` fails permanently for such a providerID
on purpose: a typo in `hosts` must not look like a missing VM and
trigger a rebuild elsewhere.
- **No adoption of `kvm` domains.** See *Migrating from kvm*.
- **A cancelled call drops the host's pooled connection once**, so other
RPCs in flight on that host fail once with `ErrTransient` and retry.
The image upload and the health check are isolated on their own
connections.
- **`Content-Length` is required** for image downloads; a mirror that
only chunks is rejected permanently. Point the catalogue at one that
sends the length.
- **A paused or pmsuspended domain cannot be adopted.** A repeat `Create`
for a domain in that state calls `DomainCreate`, which libvirt refuses
as "already running"; the provider then waits for the domain to report
running, the 2-minute wait expires, and the call fails transiently on
every retry. Resume the domain on the host (`virsh resume <name>`) or
delete the Proxy.
- **Parser text in credential errors.** A key or known_hosts file that
does not parse fails with the parser's own message, which can quote a
PEM header type or a malformed known_hosts token. Key bytes are never
quoted.
- **The sample cloud-init is an open proxy.**
`config/samples/proxy_kvm2.yaml` installs Squid with
`http_access allow all`; restrict it before exposing the bridge network
to anything but the cluster.
- **IP discovery needs the agent or guest traffic to the host.** A VM on
a bridge with no DHCP never gets an address and stays `Provisioning`
forever; check `virsh console <name>` on the host. The same symptom
with a booted, addressed VM means the guest agent never came up: check
`systemctl status qemu-guest-agent`, `cloud-init status --long` and the
vendor script's output in `/var/log/cloud-init-output.log` (a VM without
outbound internet cannot install it). A user-data that disables
vendor-data (`vendor_data: {enabled: false}`) also removes the agent, by
design. `virsh domifaddr <name> --source agent` and `--source arp` show
what the operator sees, in that order.
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
([Debian bug #1111240](https://bugs.debian.org/1111240)), which is why
`firmware` defaults to `uefi`; the same image boots under UEFI with the
serial console intact. Images that need `firmware: bios` are the
exception.
## Testing
Unit tests run over an in-memory fake of the libvirt seam and an
in-process SSH server. The optional integration test against a real host
is described in
[testing.md](testing.md#the-kvm2-integration-test-real-hypervisor); run it
with `make test-integration-kvm2`. `TestKVM2Check_realHost` runs the
health check against the same host and creates nothing:
`go test -tags=integration -count=1 -v -run TestKVM2Check ./internal/provider/kvm2/`.
Reference in New Issue View Git Blame Copy Permalink