kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Install the guest agent from a vendor-data script instead of editing the user's cloud-config

Browse Source 
Merging the bootstrap into each Proxy's cloud-config was the wrong
mechanism twice over. Detection only matched a literal #cloud-config
first line, so a jinja template or a MIME archive still fell back to
vendor-data, where cloud-init's default mergers drop it — the exact
failure the merge existed to prevent. And the YAML round-trip re-typed
unquoted scalars: version: 1.10 reached the guest as 1.1, comments were
dropped and keys reordered.

Neither is worth patching, because vendor-data delivered as a shell
script bypasses the merger entirely: cloud-init hands vendor-data to its
own handler chain, writes scripts to a separate directory and runs them
from cc_scripts_vendor, after the user's packages and before the user's
runcmd. The merge could not have been rescued anyway — merge_how in
user-data governs only the first source, and merge_how in vendor-data is
discarded before the document is stored.

The Proxy's cloud-init now reaches the seed byte for byte in any format.
installGuestAgent (default true) turns the script off; a configured
vendorData travels beside it as a MIME multipart, declared text/plain so
cloud-init re-sniffs it and this code never classifies user content.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-10 22:42:55 +00:00
co-authored by Claude Opus 5
parent 10ca71331e
commit c23def0051
8 changed files with 323 additions and 261 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/kvm-provider.md
+33 -24
View File
@@ -27,13 +27,12 @@ Per host, once:
`edk2-ovmf` on Fedora/RHEL. VMs boot UEFI by default (libvirt picks
the OVMF binary itself); a domain start fails with "no firmware
found" without it. Only `firmware: bios` images skip this.
6. Optionally, **outbound internet from the VMs**: the operator adds
`qemu-guest-agent` to every Proxy's cloud-config (`packages` plus a
`runcmd` that enables it), so it is installed from the distro mirrors
on first boot. Without it the VM still works — the operator falls back
to the host's ARP table to find the IP, which fills in once the guest
sends traffic — but note the ARP table only sees traffic to the host
itself, so on a plain bridge the agent is in practice required.
6. Optionally, **outbound internet from the VMs**: a vendor-data script
installs `qemu-guest-agent` from the distro mirrors on first boot.
Without it the VM still works — the operator falls back to the host's
ARP table to find the IP, which fills in once the guest sends traffic —
but the ARP table only sees traffic to the host itself, so on a plain
bridge the agent is in practice required.
Check from the operator's point of view before wiring it in:
@@ -136,7 +135,8 @@ ssh virt@nas1.home.lan ip link show br0
| `knownHostsFile` | unless `insecureIgnoreHostKey` | | OpenSSH `known_hosts` to verify host keys against |
| `insecureIgnoreHostKey` | | `false` | skip host-key verification entirely |
| `diskSizeGb` | | `10` | capacity of each VM's overlay disk; cloud-init's growpart expands the root fs into it |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data; cloud-init drops any top-level key here that the user-data also defines. The guest-agent bootstrap is not configured here, see *What lands on the host* |
| `vendorData` | | | extra cloud-init vendor-data placed on every seed next to the Proxy's user-data. Beware that cloud-init drops any top-level key of a vendor-data **cloud-config** that the user-data also defines. Shipped as one part of a MIME multipart alongside the agent script when `installGuestAgent` is on |
| `installGuestAgent` | | `true` | add the vendor-data script that installs and starts `qemu-guest-agent`. Turn off for images that ship it already or cannot reach a package mirror; IP discovery then depends on the host's ARP table |
| `images.<key>` | | | the key names the image in `placement.image`. Letters, digits, dots, dashes and underscores only — it becomes the cached volume name `base-<key>.<format>` |
| `images.<key>.url` | yes per entry | | http(s) URL of a cloud image (qcow2 or raw) |
| `images.<key>.sha256` | | | expected digest; verified after download, a change triggers a re-download |
@@ -182,20 +182,27 @@ console, the guest-agent channel, a virtio RNG and no graphics. Its
creation time) — that is how orphan GC recognises its own VMs, since
libvirt has no labels.
The seed's `user-data` is the Proxy's `cloudInit` with the guest-agent
bootstrap **merged into the document**: `qemu-guest-agent` appended to
`packages` and `[systemctl, enable, --now, qemu-guest-agent]` to `runcmd`
(neither duplicated if already there; other keys untouched, though the
YAML is re-rendered). It is not shipped as vendor-data because cloud-init
merges vendor-data under user-data with its default mergers, which keep
the user-data's value for any key both define and never append lists — a
cloud-config with its own `packages:` would silently lose the agent and
the Proxy would sit in `Provisioning` forever. Only when the user-data is
not a cloud-config document (a shell script, `#include`, MIME archive)
does the bootstrap go to `vendor-data`, and then only if `vendorData` is
not configured. A cloud-config that does not parse as a YAML mapping, or
whose `packages`/`runcmd` is not a list, fails the Proxy permanently
rather than booting a VM without its config.
The seed's `user-data` is the Proxy's `cloudInit`, **byte for byte**. Any
format cloud-init accepts works, including a Jinja template, a MIME
archive, a shell script or a gzipped payload; the operator never parses or
re-renders it.
The guest agent rides along in `vendor-data`, as a **shell script** rather
than a cloud-config. That is the only shape cloud-init cannot drop:
vendor-data cloud-config is merged under the user's own document with
mergers that keep the user-data's value for every key both define and never
append lists, so a Proxy carrying its own `packages:` silently lost the
agent and sat in `Provisioning` forever. The strategy cannot be overridden
from either document — `merge_how` in user-data governs only the first
source, and `merge_how` in vendor-data is consumed and discarded before the
document is stored. A vendor-data script skips all of it: cloud-init gives
vendor-data its own handler chain, writes scripts to a separate directory
and runs them from `cc_scripts_vendor`, after the user's packages are
installed and before the user's `runcmd`. The script is idempotent, picks
whichever of `apt-get`, `dnf`, `yum`, `zypper` or `apk` the image has, and
logs into `/var/log/cloud-init-output.log`. Set `installGuestAgent: false`
to leave it out. When `vendorData` is also configured, the two travel as
separate parts of a MIME multipart vendor-data.
## The image cache
@@ -287,8 +294,10 @@ The provider logs through controller-runtime's logger, named `kvm`, with
bridge with no DHCP never gets an address and stays `Provisioning`
forever; check `virsh console`. The same symptom with a booted, addressed
VM means the guest agent never came up: check `systemctl status
qemu-guest-agent` and `cloud-init status --long` in the guest (a VM
without outbound internet cannot install it).
qemu-guest-agent`, `cloud-init status --long` and the vendor script's
output in `/var/log/cloud-init-output.log` (a VM without outbound
internet cannot install it). A user-data that disables vendor-data
(`vendor_data: {enabled: false}`) also removes the agent, by design.
- **Debian 13 cloud images hang in GRUB under BIOS on a headless VM**
([Debian bug #1111240](https://bugs.debian.org/1111240), open, in
`grub2`): trixie's image ships `terminal_output gfxterm serial` in its
docs/plans-executions/2026-09-10-2222-kvm-review-fixes.md
+65 -1
View File
@@ -5,7 +5,7 @@ Plan: [docs/plans/2026-09-10-2222-kvm-review-fixes.md](../plans/2026-09-10-2222-
- [x] Step 1 — Unwedge deletion (finding 1)
- [x] Step 2 — Validate names that become identifiers (finding 2)
- [x] Step 3 — Read the SSH material lazily (finding 3)
- [ ] Step 4 — Guest agent via vendor-data script (findings 4 and 5)
- [x] Step 4 — Guest agent via vendor-data script (findings 4 and 5)
- [ ] Step 5 — Validate, record, push
## Step 0 — plan committed
@@ -98,6 +98,70 @@ a rotated Secret needs a manager restart. `TestNew_badKeyPathFailsFast` was
inverted rather than deleted, so the old contract stays visible in the diff
next to the new one.
## Step 4 — Guest agent via vendor-data script
The review filed two findings against `withGuestAgent` — detection too
narrow, and YAML re-typing on the round-trip. Reading cloud-init 26.1's
sources while planning turned both into one finding: the operator should
not be editing the user's document at all.
Vendor-data gets its own handler chain (`stages.py:_default_vendordata_handlers`),
with separate script and cloud-config paths. A vendor-data **script** is
therefore written to the vendor script directory and run by
`cc_scripts_vendor`, never touching the config merger that drops colliding
keys. On the final-stage module list it lands after
`package_update_upgrade_install` and before the user's `runcmd`, so the
agent now comes up *earlier* than the old `runcmd` append did.
That also closed the door the previous approach had left ajar. The merge
cannot be rescued from inside either document: `merge_how` in user-data
governs only the first source, and `merge_how` in vendor-data is popped by
`dict_extract_mergers` and discarded before the document is written. There
was no fix that kept the merge.
`withGuestAgent`, `isCloudConfig` and `appendUnique` are gone, along with
seed.go's `sigs.k8s.io/yaml` dependency. `installGuestAgent` (default true)
turns the script off. When `vendorData` is also configured the two ship as
a MIME multipart, with the user's document declared `text/plain` — a type
cloud-init re-sniffs from the content's own header line, so this code never
classifies what the user wrote.
Before deleting the old path, its damage was captured rather than taken on
trust. Round-tripping a plausible cloud-config through it:
```sh
# temporary test in the package, deleted afterwards
go test -run TestZZProbeRetyping -v ./internal/provider/kvm/
# in: version: 1.10
# out: version: 1.1
```
`1.10` arriving as `1.1` is worse than the reviewer's `passwd` example,
since nothing about it looks wrong in the seed. Comments were dropped and
keys reordered on the same pass.
The generated vendor-data was then checked against the cloud-init actually
installed on this machine, rather than against the docs:
```sh
python3 -c 'from cloudinit import user_data, handlers; ...'
# vendor-script: multipart=False sniffed=text/x-shellscript
# vendor-multipart: multipart=True part 1 text/plain -> text/cloud-config
# part 2 text/x-shellscript
```
and the script itself through `sh -n`.
Worth noting: the script has to choose a package manager where cloud-init's
`packages:` key was distro-agnostic. It covers apt, dnf, yum, zypper and
apk, and both systemd and OpenRC for the enable step, but an image outside
that set now needs `installGuestAgent: false` and the agent baked in. The
seed's user-data is byte-identical to the Proxy's `cloudInit` from here on,
which is what the new `TestBuildSeedISO_userDataIsNeverRewritten` pins
across seven formats. One consequence worth remembering: a user-data that
sets `vendor_data: {enabled: false}` disables the agent, and cloud-init
documents that as intended behaviour.
Worth noting: the review filed this as a KVM finding, but the fix is mostly
in the controller, because KVM is the only provider that resolves a
providerID segment against operator config at all. GCP and Kubernetes hand