From c3a581af9522d49d9ca362dc8ac84e3799a12def Mon Sep 17 00:00:00 2001 From: kacerr Date: Fri, 18 Sep 2026 07:45:16 +0000 Subject: [PATCH] samples: a kvm2 crawl pool, not just the single-proxy sample proxy_kvm2.yaml shows one VM; nothing showed the shape a crawler actually leases from. Add a five-Proxy pool on the kvm2-home entry: four share one labelled cloud-init Secret, and the fifth carries its own inline block because it runs squid on 8080 and spec.port has to match the http_port a shared Secret would pin. It stays out of the samples kustomization so `apply -k config/samples` doesn't boot a pool along with the one-per-provider samples. All six objects validate against the installed CRD under --dry-run=server. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 14 ++ config/samples/kustomization.yaml | 3 + config/samples/proxy_kvm2_crawl_pool.yaml | 190 ++++++++++++++++++++++ 3 files changed, 207 insertions(+) create mode 100644 config/samples/proxy_kvm2_crawl_pool.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d6ba6d..f4387db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,19 @@ # Changelog +## 2026-09-18 07:44 UTC — a kvm2 crawl-pool sample + +Until now the only kvm2 manifest was the one-VM `config/samples/proxy_kvm2.yaml`, which +left the multi-proxy shape — the one a crawler actually leases from — undocumented. + +- Added `config/samples/proxy_kvm2_crawl_pool.yaml`: five Managed Proxies on the `kvm2-home` + entry, four sharing one labelled cloud-init Secret and one inline (it runs squid on 8080, + so it cannot share a Secret that pins `http_port 3128`). Covers the `tier` attribute the + discovery API selects on, a `medium`/`ubuntu-24.04` member with a raised `maxLeases`, and + a member with a tightened `healthCheck`. +- Kept out of `config/samples/kustomization.yaml` on purpose so `apply -k` doesn't boot a + whole pool; the exclusion is commented there next to the existing `providers-config.yaml` + one. Validated with `kubectl apply --dry-run=server`. + ## 2026-09-17 20:36 UTC — kvm2: the four high findings from the code review, and its first run on a real hypervisor A scoped review of the kvm2 provider (`review/REPORT.md`: 52 findings, 4 high, 20 medium, diff --git a/config/samples/kustomization.yaml b/config/samples/kustomization.yaml index c252b84..518981b 100644 --- a/config/samples/kustomization.yaml +++ b/config/samples/kustomization.yaml @@ -1,6 +1,9 @@ ## Append samples of your project ## # providers-config.yaml is deliberately absent: it is a sample # --providers-config file, not a Kubernetes manifest. +# proxy_kvm2_crawl_pool.yaml is deliberately absent too: it is a whole pool +# of VMs, so it stays an explicit `kubectl apply -f` rather than something +# `apply -k` boots along with the one-per-provider samples. resources: - proxy_kubernetes.yaml - proxy_gcp.yaml diff --git a/config/samples/proxy_kvm2_crawl_pool.yaml b/config/samples/proxy_kvm2_crawl_pool.yaml new file mode 100644 index 0000000..4294b86 --- /dev/null +++ b/config/samples/proxy_kvm2_crawl_pool.yaml @@ -0,0 +1,190 @@ +# A pool of Managed crawl proxies on the kvm2 provider — the multi-proxy +# counterpart to the single proxy_kvm2.yaml sample. Everything here boots on +# the one host the kvm2 entry configures (beelink); the proxies differ in the +# attributes crawlers select on, in size, and in image. +# +# Requires the same prerequisites as proxy_kvm2.yaml: a providers-config +# entry named "kvm2-home" and the kvm-ssh Secret that the kvm and kvm2 +# providers share (docs/kvm2-provider.md). +# +# Deliberately NOT in kustomization.yaml: `kubectl apply -k config/samples` +# would boot every VM in here at once. Apply it on its own when you want the +# pool: +# +# kubectl apply -f config/samples/proxy_kvm2_crawl_pool.yaml +# +# Crawlers reach these through the discovery API by attribute, e.g. +# `GET /v1/proxies?attr.purpose=crawl&healthy=true`, or by leasing one with +# the same selector. Attribute matching is exact equality, so keep the keys +# consistent across the pool and vary only the values you want to select on. +# +# The first Proxy for an image on the host waits for that image to download; +# the rest reuse the cached copy. debian-13 and ubuntu-24.04 are separate +# downloads, so the ubuntu member below is the slow one on a cold host. +--- +# One cloud-init Secret shared by the whole pool: the proxies below point at +# it with secretRef instead of each carrying its own copy of the squid +# config. The crawl.example.com/cloud-init label is mandatory — the manager's +# cache only holds Secrets that carry it, so an unlabelled Secret is +# invisible to the operator and the Proxy never resolves. +# +# Editing this Secret changes the resolved cloud-init of every Proxy that +# references it, and that replaces each of their VMs. That is the trade for +# sharing one definition: convenient for a config rollout, disruptive if you +# only meant to touch one member. Give a member its own Secret (or an inline +# block) when it needs to diverge. +apiVersion: v1 +kind: Secret +metadata: + name: crawl-squid-cloud-init + labels: + crawl.example.com/cloud-init: "true" +type: Opaque +stringData: + # The default key name; SecretKeySelector.key defaults to user-data, so + # the proxies below omit it. + user-data: | + #cloud-config + packages: + - squid + write_files: + - path: /etc/squid/conf.d/proxy-operator.conf + content: | + http_port 3128 + http_access allow all + via off + forwarded_for off + runcmd: + - systemctl restart squid + # Optional: log in with `ssh debian@` (or `ubuntu@` on the + # ubuntu-24.04 member) to look around; `virsh console ` on the + # host works too. + # ssh_authorized_keys: + # - ssh-ed25519 AAAA... you@laptop +--- +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: crawl-home-1 +spec: + mode: Managed + provider: kvm2-home + placement: + zone: beelink + machineType: small + image: debian-13 + port: 3128 + cloudInit: + secretRef: + name: crawl-squid-cloud-init + attributes: + geo: home + purpose: crawl + tier: general +--- +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: crawl-home-2 +spec: + mode: Managed + provider: kvm2-home + placement: + zone: beelink + machineType: small + image: debian-13 + port: 3128 + cloudInit: + secretRef: + name: crawl-squid-cloud-init + attributes: + geo: home + purpose: crawl + tier: general +--- +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: crawl-home-3 +spec: + mode: Managed + provider: kvm2-home + placement: + zone: beelink + machineType: small + image: debian-13 + port: 3128 + cloudInit: + secretRef: + name: crawl-squid-cloud-init + attributes: + geo: home + purpose: crawl + tier: general +--- +# A heavier member: two vCPUs and 2 GiB from the medium machine type, a +# different base image, and room for more concurrent leases. tier=heavy is +# what a crawler selects on to land here rather than on the general members. +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: crawl-home-heavy-1 +spec: + mode: Managed + provider: kvm2-home + placement: + zone: beelink + machineType: medium + image: ubuntu-24.04 + port: 3128 + cloudInit: + secretRef: + name: crawl-squid-cloud-init + attributes: + geo: home + purpose: crawl + tier: heavy + maxLeases: 10 +--- +# A member that probes harder than the pool default (30s interval, 3 failures +# to go unhealthy): a tighter loop takes a flapping proxy out of rotation +# sooner, at the cost of more probe traffic. Everything under healthCheck is +# defaulted, so set only what you want to change. +# +# This one also runs squid on a non-default port, which is why it carries its +# own inline cloud-init: spec.port and the http_port in the config have to +# agree, and the shared Secret above pins 3128. +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: crawl-home-probe-fast-1 +spec: + mode: Managed + provider: kvm2-home + placement: + zone: beelink + machineType: small + image: debian-13 + port: 8080 + cloudInit: + inline: | + #cloud-config + packages: + - squid + write_files: + - path: /etc/squid/conf.d/proxy-operator.conf + content: | + http_port 8080 + http_access allow all + via off + forwarded_for off + runcmd: + - systemctl restart squid + healthCheck: + intervalSeconds: 10 + timeoutSeconds: 3 + failureThreshold: 2 + attributes: + geo: home + purpose: crawl + tier: general