kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge the qemu-guest-agent bootstrap into each Proxy's cloud-config instead of vendor-data

Browse Source 
In the kind cluster two KVM Proxies with real cloud-configs booted fine
but never left Provisioning: the guest agent was not installed, so the
provider could not learn the IP. cloud-init merges vendor-data under
user-data with its default mergers, which keep the user-data's value
for any key both define and never append lists, so a cloud-config with
its own `packages:` silently dropped the vendor-data's packages and
runcmd. The integration test passed only because its user-data was
empty.

withGuestAgent now appends qemu-guest-agent to `packages` and the
enable command to `runcmd` of the Proxy's own cloud-config (skipping
entries already present) and re-renders it. Vendor-data is no longer
defaulted; `vendorData` is plain extra data. User-data that is not a
cloud-config document (script, #include, MIME) still gets the bootstrap
as vendor-data. A cloud-config that is not a YAML mapping, or whose
packages/runcmd is not a list, fails the Proxy as ErrPermanent.

Validated in-cluster: both Proxies Ready and healthy in two minutes, 204
through squid from the LAN. Docs, sample comment, CHANGELOG and the
plan execution log (Run 7) updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-10 21:06:54 +00:00
co-authored by Claude Fable 5.1
parent 220967660d
commit 366736f53f
9 changed files with 382 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/kvm-provider.md
+27 -7
View File
@@ -27,11 +27,13 @@ Per host, once:
`edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default (libvirt picks
the OVMF binary itself); a domain start fails with "no firmware
found" without it. Only `firmware: bios` images skip this.
6. Optionally, **outbound internet from the VMs**: the default
vendor-data installs `qemu-guest-agent` from the distro mirrors on
first boot. Without it the VM still works — the operator falls back to
the host's ARP table to find the IP, which fills in once the guest
sends traffic.
6. Optionally, **outbound internet from the VMs**: the operator adds
`qemu-guest-agent` to every Proxy's cloud-config (`packages` plus a
`runcmd` that enables it), so it is installed from the distro mirrors
on first boot. Without it the VM still works — the operator falls back
to the host's ARP table to find the IP, which fills in once the guest
sends traffic — but note the ARP table only sees traffic to the host
itself, so on a plain bridge the agent is in practice required.
Check from the operator's point of view before wiring it in:
@@ -129,7 +131,7 @@ ssh virt@nas1.home.lan ip link show br0
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against |
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely |
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it |
| `vendorData` | | installs `qemu-guest-agent` | cloud-init vendor-data placed on every seed next to the Proxy's user-data |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data; cloud-init drops any top-level key here that the user-data also defines. The guest-agent bootstrap is not configured here, see *What lands on the host* |
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) |
| `images.<key>.sha256` | | | expected digest; verified after download, a change triggers a re-download |
| `images.<key>.format` | | `qcow2` | `qcow2` or `raw` |
@@ -174,6 +176,21 @@ console, the guest-agent channel, a virtio RNG and no graphics. Its
creation time) — that is how orphan GC recognises its own VMs, since
libvirt has no labels.
The seed's `user-data` is the Proxy's `cloudInit` with the guest-agent
bootstrap **merged into the document**: `qemu-guest-agent` appended to
`packages` and `[systemctl, enable, --now, qemu-guest-agent]` to `runcmd`
(neither duplicated if already there; other keys untouched, though the
YAML is re-rendered). It is not shipped as vendor-data because cloud-init
merges vendor-data under user-data with its default mergers, which keep
the user-data's value for any key both define and never append lists — a
cloud-config with its own `packages:` would silently lose the agent and
the Proxy would sit in `Provisioning` forever. Only when the user-data is
not a cloud-config document (a shell script, `#include`, MIME archive)
does the bootstrap go to `vendor-data`, and then only if `vendorData` is
not configured. A cloud-config that does not parse as a YAML mapping, or
whose `packages`/`runcmd` is not a list, fails the Proxy permanently
rather than booting a VM without its config.
## The image cache
The base image is downloaded **by the operator, through libvirt**, into
@@ -254,7 +271,10 @@ The provider logs through controller-runtime's logger, named `kvm`, with
under *The image cache*.
- **IP discovery needs either guest traffic or the agent.** A VM on a
bridge with no DHCP never gets an address and stays `Provisioning`
forever; check `virsh console`.
forever; check `virsh console`. The same symptom with a booted, addressed
VM means the guest agent never came up: check `systemctl status
qemu-guest-agent` and `cloud-init status --long` in the guest (a VM
without outbound internet cannot install it).
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
([Debian bug #1111240](https://bugs.debian.org/1111240), open, in
`grub2`): trixie's image ships `terminal_output gfxterm serial` in its