kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge the qemu-guest-agent bootstrap into each Proxy's cloud-config instead of vendor-data

Browse Source 
In the kind cluster two KVM Proxies with real cloud-configs booted fine
but never left Provisioning: the guest agent was not installed, so the
provider could not learn the IP. cloud-init merges vendor-data under
user-data with its default mergers, which keep the user-data's value
for any key both define and never append lists, so a cloud-config with
its own `packages:` silently dropped the vendor-data's packages and
runcmd. The integration test passed only because its user-data was
empty.

withGuestAgent now appends qemu-guest-agent to `packages` and the
enable command to `runcmd` of the Proxy's own cloud-config (skipping
entries already present) and re-renders it. Vendor-data is no longer
defaulted; `vendorData` is plain extra data. User-data that is not a
cloud-config document (script, #include, MIME) still gets the bootstrap
as vendor-data. A cloud-config that is not a YAML mapping, or whose
packages/runcmd is not a list, fails the Proxy as ErrPermanent.

Validated in-cluster: both Proxies Ready and healthy in two minutes, 204
through squid from the LAN. Docs, sample comment, CHANGELOG and the
plan execution log (Run 7) updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-10 21:06:54 +00:00
co-authored by Claude Fable 5.1
parent 220967660d
commit 366736f53f
9 changed files with 382 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/kvm-provider.md
+27 -7
View File
@@ -27,11 +27,13 @@ Per host, once:
`edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default (libvirt picks
the OVMF binary itself); a domain start fails with "no firmware
found" without it. Only `firmware: bios` images skip this.
6. Optionally, **outbound internet from the VMs**: the default
vendor-data installs `qemu-guest-agent` from the distro mirrors on
first boot. Without it the VM still works — the operator falls back to
the host's ARP table to find the IP, which fills in once the guest
sends traffic.
6. Optionally, **outbound internet from the VMs**: the operator adds
`qemu-guest-agent` to every Proxy's cloud-config (`packages` plus a
`runcmd` that enables it), so it is installed from the distro mirrors
on first boot. Without it the VM still works — the operator falls back
to the host's ARP table to find the IP, which fills in once the guest
sends traffic — but note the ARP table only sees traffic to the host
itself, so on a plain bridge the agent is in practice required.
Check from the operator's point of view before wiring it in:
@@ -129,7 +131,7 @@ ssh virt@nas1.home.lan ip link show br0
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against |
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely |
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it |
| `vendorData` | | installs `qemu-guest-agent` | cloud-init vendor-data placed on every seed next to the Proxy's user-data |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data; cloud-init drops any top-level key here that the user-data also defines. The guest-agent bootstrap is not configured here, see *What lands on the host* |
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) |
| `images.<key>.sha256` | | | expected digest; verified after download, a change triggers a re-download |
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
@@ -174,6 +176,21 @@ console, the guest-agent channel, a virtio RNG and no graphics. Its
creation time) — that is how orphan GC recognises its own VMs, since
libvirt has no labels.
The seed's `user-data` is the Proxy's `cloudInit` with the guest-agent
bootstrap **merged into the document**: `qemu-guest-agent` appended to
`packages` and `[systemctl, enable, --now, qemu-guest-agent]` to `runcmd`
(neither duplicated if already there; other keys untouched, though the
YAML is re-rendered). It is not shipped as vendor-data because cloud-init
merges vendor-data under user-data with its default mergers, which keep
the user-data's value for any key both define and never append lists — a
cloud-config with its own `packages:` would silently lose the agent and
the Proxy would sit in `Provisioning` forever. Only when the user-data is
not a cloud-config document (a shell script, `#include`, MIME archive)
does the bootstrap go to `vendor-data`, and then only if `vendorData` is
not configured. A cloud-config that does not parse as a YAML mapping, or
whose `packages`/`runcmd` is not a list, fails the Proxy permanently
rather than booting a VM without its config.
## The image cache
The base image is downloaded **by the operator, through libvirt**, into
@@ -254,7 +271,10 @@ The provider logs through controller-runtime's logger, named `kvm`, with
under *The image cache*.
- **IP discovery needs either guest traffic or the agent.** A VM on a
bridge with no DHCP never gets an address and stays `Provisioning`
forever; check `virsh console`.
forever; check `virsh console`. The same symptom with a booted, addressed
VM means the guest agent never came up: check `systemctl status
qemu-guest-agent` and `cloud-init status --long` in the guest (a VM
without outbound internet cannot install it).
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
([Debian bug #1111240](https://bugs.debian.org/1111240), open, in
`grub2`): trixie's image ships `terminal_output gfxterm serial` in its
docs/plans-executions/2026-09-04-1404-kvm-provider.md
+73
View File
@@ -557,3 +557,76 @@ on the host it needs `virsh undefine --nvram` plus its two volumes by
hand. The `ListByTag` sweep costs one `virDomainGetMetadata` per
domain on the host (21 here, ~0.4 ms each over the tunnel), which is
fine at this scale but is the call to watch on a busy hypervisor.
### Run 7: in-cluster, two defects the integration test could not see
The e2e pass ran the operator inside the kind cluster
(`egress-proxies-operator:kvm-dev`, built with `make docker-build`,
loaded with `kind load docker-image`, deployed via the kustomize
`config/manager` with the `kvm-home` provider pointing at beelink) and
two Proxies with real cloud-configs: one plain squid, one with a debug
login user. Both defects come from the difference between the test
harness and the real thing, not from libvirt.
**Read-only root filesystem.** The manager runs with
`readOnlyRootFilesystem: true`, and `iso9660.NewWriter` stages the seed
in a temp directory, so the first Create failed with `creating iso
writer: read-only file system`. Fix: a 64 MiB `emptyDir` on `/tmp` in
`config/manager/manager.yaml`. Side finding: the error is classified
`ErrPermanent` by `writeSeed`, yet `hostPool.call` treated any
non-libvirt error as a transport failure and dropped the SSH connection,
so every retry redialled. `call` now leaves the connection alone for
errors already carrying a provider sentinel (`isClassified`), with a
test.
**The guest agent never got installed.** Both VMs booted (confirmed on
the console by hand) but sat in `Provisioning` for 25 minutes: the
trace showed `virDomainInterfaceAddresses` from the agent source failing
with `ErrAgentUnresponsive` on every poll and the ARP source empty. The
integration test had passed with `CloudInit: "#cloud-config\n"`; the
real Proxies have `packages: [squid]` and `runcmd:`. cloud-init merges
vendor-data under user-data with its default mergers, which keep the
user-data's value for any key both define and never append lists, so the
vendor-data's `packages`/`runcmd` were dropped wholesale. Confirmed
locally with the dev box's cloud-init 26.1 before changing anything:
```sh
python3 - <<'EOF'
from cloudinit import mergers
user = {"packages": ["squid"], "runcmd": ["systemctl restart squid"]}
vendor = {"packages": ["qemu-guest-agent"], "runcmd": [["systemctl", "enable", "--now", "qemu-guest-agent"]]}
print(mergers.construct(mergers.default_mergers()).merge(dict(user), dict(vendor)))
EOF
# {'packages': ['squid'], 'runcmd': ['systemctl restart squid']}
```
Fix: `withGuestAgent` in `seed.go` merges the bootstrap into the Proxy's
cloud-config itself (append to `packages` and `runcmd`, skip if present,
re-render with `sigs.k8s.io/yaml`), so vendor-data is no longer defaulted
and `vendorData` is plain extra data. User-data that is not a cloud-config
document (script, `#include`, MIME) still gets the bootstrap as
vendor-data since there is nothing to collide with; a cloud-config that
does not parse as a mapping fails the Proxy as `ErrPermanent`. The
rendered document for the squid Proxy passed `cloud-init schema
--config-file`. The logging test's fixture (`#cloud-config` + a bare
sentinel line) is not a mapping and had to become `hostname: <sentinel>`.
Rebuilt, reloaded, restarted, deleted and recreated both Proxies: both
`Ready`/`HEALTHY True` in 2 min 1 s from `kubectl apply`, IP via the
agent 110 s after `virDomainCreate` (the guest installs squid and the
agent from the mirrors first), and `curl -x http://<ip>:3128
https://www.gstatic.com/generate_204` from the dev box returns 204
through both.
Worth noting: the `#cloud-config` round-trip drops comments and reorders
keys alphabetically; semantics are preserved (numbers, booleans and
block scalars survive, checked in tests) but a user diffing the seed
against their CR will see a different document. Vendor-data supplied via
`vendorData` is subject to the same cloud-init merge rule and is now
documented as such. The kind pod also restarted twice during the session
from "Pod sandbox changed" (the node briefly lost the API server), which
is unrelated to the operator. Still uncommitted on purpose: the
`kustomization.yaml` image override (`egress-proxies-operator:kvm-dev`)
and the `beelink` host entries in `config/manager/providers_config.yaml`
and `config/samples/providers-config.yaml`, which are this machine's dev
setup rather than the repo's defaults.