kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Install the guest agent from a vendor-data script instead of editing the user's cloud-config

Browse Source 
Merging the bootstrap into each Proxy's cloud-config was the wrong
mechanism twice over. Detection only matched a literal #cloud-config
first line, so a jinja template or a MIME archive still fell back to
vendor-data, where cloud-init's default mergers drop it — the exact
failure the merge existed to prevent. And the YAML round-trip re-typed
unquoted scalars: version: 1.10 reached the guest as 1.1, comments were
dropped and keys reordered.

Neither is worth patching, because vendor-data delivered as a shell
script bypasses the merger entirely: cloud-init hands vendor-data to its
own handler chain, writes scripts to a separate directory and runs them
from cc_scripts_vendor, after the user's packages and before the user's
runcmd. The merge could not have been rescued anyway — merge_how in
user-data governs only the first source, and merge_how in vendor-data is
discarded before the document is stored.

The Proxy's cloud-init now reaches the seed byte for byte in any format.
installGuestAgent (default true) turns the script off; a configured
vendorData travels beside it as a MIME multipart, declared text/plain so
cloud-init re-sniffs it and this code never classifies user content.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-10 22:42:55 +00:00
co-authored by Claude Opus 5
parent 10ca71331e
commit c23def0051
8 changed files with 323 additions and 261 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/kvm-provider.md
+33 -24
View File
@@ -27,13 +27,12 @@ Per host, once:
`edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default (libvirt picks
the OVMF binary itself); a domain start fails with "no firmware
found" without it. Only `firmware: bios` images skip this.
6. Optionally, **outbound internet from the VMs**: the operator adds
`qemu-guest-agent` to every Proxy's cloud-config (`packages` plus a
`runcmd` that enables it), so it is installed from the distro mirrors
on first boot. Without it the VM still works — the operator falls back
to the host's ARP table to find the IP, which fills in once the guest
sends traffic — but note the ARP table only sees traffic to the host
itself, so on a plain bridge the agent is in practice required.
6. Optionally, **outbound internet from the VMs**: a vendor-data script
installs `qemu-guest-agent` from the distro mirrors on first boot.
Without it the VM still works — the operator falls back to the host's
ARP table to find the IP, which fills in once the guest sends traffic —
but the ARP table only sees traffic to the host itself, so on a plain
bridge the agent is in practice required.
Check from the operator's point of view before wiring it in:
@@ -136,7 +135,8 @@ ssh virt@nas1.home.lan ip link show br0
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against |
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely |
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data; cloud-init drops any top-level key here that the user-data also defines. The guest-agent bootstrap is not configured here, see *What lands on the host* |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on |
| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table |
| `images.<key>` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only — it becomes the cached volume name `base-<key>.<format>` |
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) |
| `images.<key>.sha256` | | | expected digest; verified after download, a change triggers a re-download |
@@ -182,20 +182,27 @@ console, the guest-agent channel, a virtio RNG and no graphics. Its
creation time) — that is how orphan GC recognises its own VMs, since
libvirt has no labels.
The seed's `user-data` is the Proxy's `cloudInit` with the guest-agent
bootstrap **merged into the document**: `qemu-guest-agent` appended to
`packages` and `[systemctl, enable, --now, qemu-guest-agent]` to `runcmd`
(neither duplicated if already there; other keys untouched, though the
YAML is re-rendered). It is not shipped as vendor-data because cloud-init
merges vendor-data under user-data with its default mergers, which keep
the user-data's value for any key both define and never append lists — a
cloud-config with its own `packages:` would silently lose the agent and
the Proxy would sit in `Provisioning` forever. Only when the user-data is
not a cloud-config document (a shell script, `#include`, MIME archive)
does the bootstrap go to `vendor-data`, and then only if `vendorData` is
not configured. A cloud-config that does not parse as a YAML mapping, or
whose `packages`/`runcmd` is not a list, fails the Proxy permanently
rather than booting a VM without its config.
The seed's `user-data` is the Proxy's `cloudInit`, **byte for byte**. Any
format cloud-init accepts works, including a Jinja template, a MIME
archive, a shell script or a gzipped payload; the operator never parses or
re-renders it.
The guest agent rides along in `vendor-data`, as a **shell script** rather
than a cloud-config. That is the only shape cloud-init cannot drop:
vendor-data cloud-config is merged under the user's own document with
mergers that keep the user-data's value for every key both define and never
append lists, so a Proxy carrying its own `packages:` silently lost the
agent and sat in `Provisioning` forever. The strategy cannot be overridden
from either document — `merge_how` in user-data governs only the first
source, and `merge_how` in vendor-data is consumed and discarded before the
document is stored. A vendor-data script skips all of it: cloud-init gives
vendor-data its own handler chain, writes scripts to a separate directory
and runs them from `cc_scripts_vendor`, after the user's packages are
installed and before the user's `runcmd`. The script is idempotent, picks
whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk` the image has, and
logs into `/var/log/cloud-init-output.log`. Set `installGuestAgent: false`
to leave it out. When `vendorData` is also configured, the two travel as
separate parts of a MIME multipart vendor-data.
## The image cache
@@ -287,8 +294,10 @@ The provider logs through controller-runtime's logger, named `kvm`, with
bridge with no DHCP never gets an address and stays `Provisioning`
forever; check `virsh console`. The same symptom with a booted, addressed
VM means the guest agent never came up: check `systemctl status
qemu-guest-agent` and `cloud-init status --long` in the guest (a VM
without outbound internet cannot install it).
qemu-guest-agent`, `cloud-init status --long` and the vendor script's
output in `/var/log/cloud-init-output.log` (a VM without outbound
internet cannot install it). A user-data that disables vendor-data
(`vendor_data: {enabled: false}`) also removes the agent, by design.
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
([Debian bug #1111240](https://bugs.debian.org/1111240), open, in
`grub2`): trixie's image ships `terminal_output gfxterm serial` in its