samples: a kvm2 crawl pool, not just the single-proxy sample

proxy_kvm2.yaml shows one VM; nothing showed the shape a crawler actually leases from. Add a five-Proxy pool on the kvm2-home entry: four share one labelled cloud-init Secret, and the fifth carries its own inline block because it runs squid on 8080 and spec.port has to match the http_port a shared Secret would pin. It stays out of the samples kustomization so `apply -k config/samples` doesn't boot a pool along with the one-per-provider samples. All six objects validate against the installed CRD under --dry-run=server. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 07:45:16 +00:00
parent 5928c3fe75
commit c3a581af95
3 changed files with 207 additions and 0 deletions
@@ -1,5 +1,19 @@
 # Changelog

+## 2026-09-18 07:44 UTC — a kvm2 crawl-pool sample
+
+Until now the only kvm2 manifest was the one-VM `config/samples/proxy_kvm2.yaml`, which
+left the multi-proxy shape — the one a crawler actually leases from — undocumented.
+
+- Added `config/samples/proxy_kvm2_crawl_pool.yaml`: five Managed Proxies on the `kvm2-home`
+  entry, four sharing one labelled cloud-init Secret and one inline (it runs squid on 8080,
+  so it cannot share a Secret that pins `http_port 3128`). Covers the `tier` attribute the
+  discovery API selects on, a `medium`/`ubuntu-24.04` member with a raised `maxLeases`, and
+  a member with a tightened `healthCheck`.
+- Kept out of `config/samples/kustomization.yaml` on purpose so `apply -k` doesn't boot a
+  whole pool; the exclusion is commented there next to the existing `providers-config.yaml`
+  one. Validated with `kubectl apply --dry-run=server`.
+
 ## 2026-09-17 20:36 UTC — kvm2: the four high findings from the code review, and its first run on a real hypervisor

 A scoped review of the kvm2 provider (`review/REPORT.md`: 52 findings, 4 high, 20 medium,
@@ -1,6 +1,9 @@
 ## Append samples of your project ##
 # providers-config.yaml is deliberately absent: it is a sample
 # --providers-config file, not a Kubernetes manifest.
+# proxy_kvm2_crawl_pool.yaml is deliberately absent too: it is a whole pool
+# of VMs, so it stays an explicit `kubectl apply -f` rather than something
+# `apply -k` boots along with the one-per-provider samples.
 resources:
 - proxy_kubernetes.yaml
 - proxy_gcp.yaml
@@ -0,0 +1,190 @@
+# A pool of Managed crawl proxies on the kvm2 provider — the multi-proxy
+# counterpart to the single proxy_kvm2.yaml sample. Everything here boots on
+# the one host the kvm2 entry configures (beelink); the proxies differ in the
+# attributes crawlers select on, in size, and in image.
+#
+# Requires the same prerequisites as proxy_kvm2.yaml: a providers-config
+# entry named "kvm2-home" and the kvm-ssh Secret that the kvm and kvm2
+# providers share (docs/kvm2-provider.md).
+#
+# Deliberately NOT in kustomization.yaml: `kubectl apply -k config/samples`
+# would boot every VM in here at once. Apply it on its own when you want the
+# pool:
+#
+#     kubectl apply -f config/samples/proxy_kvm2_crawl_pool.yaml
+#
+# Crawlers reach these through the discovery API by attribute, e.g.
+# `GET /v1/proxies?attr.purpose=crawl&healthy=true`, or by leasing one with
+# the same selector. Attribute matching is exact equality, so keep the keys
+# consistent across the pool and vary only the values you want to select on.
+#
+# The first Proxy for an image on the host waits for that image to download;
+# the rest reuse the cached copy. debian-13 and ubuntu-24.04 are separate
+# downloads, so the ubuntu member below is the slow one on a cold host.
+---
+# One cloud-init Secret shared by the whole pool: the proxies below point at
+# it with secretRef instead of each carrying its own copy of the squid
+# config. The crawl.example.com/cloud-init label is mandatory — the manager's
+# cache only holds Secrets that carry it, so an unlabelled Secret is
+# invisible to the operator and the Proxy never resolves.
+#
+# Editing this Secret changes the resolved cloud-init of every Proxy that
+# references it, and that replaces each of their VMs. That is the trade for
+# sharing one definition: convenient for a config rollout, disruptive if you
+# only meant to touch one member. Give a member its own Secret (or an inline
+# block) when it needs to diverge.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: crawl-squid-cloud-init
+  labels:
+    crawl.example.com/cloud-init: "true"
+type: Opaque
+stringData:
+  # The default key name; SecretKeySelector.key defaults to user-data, so
+  # the proxies below omit it.
+  user-data: |
+    #cloud-config
+    packages:
+      - squid
+    write_files:
+      - path: /etc/squid/conf.d/proxy-operator.conf
+        content: |
+          http_port 3128
+          http_access allow all
+          via off
+          forwarded_for off
+    runcmd:
+      - systemctl restart squid
+    # Optional: log in with `ssh debian@<ip>` (or `ubuntu@<ip>` on the
+    # ubuntu-24.04 member) to look around; `virsh console <name>` on the
+    # host works too.
+    # ssh_authorized_keys:
+    #   - ssh-ed25519 AAAA... you@laptop
+---
+apiVersion: crawl.example.com/v1alpha1
+kind: Proxy
+metadata:
+  name: crawl-home-1
+spec:
+  mode: Managed
+  provider: kvm2-home
+  placement:
+    zone: beelink
+    machineType: small
+    image: debian-13
+  port: 3128
+  cloudInit:
+    secretRef:
+      name: crawl-squid-cloud-init
+  attributes:
+    geo: home
+    purpose: crawl
+    tier: general
+---
+apiVersion: crawl.example.com/v1alpha1
+kind: Proxy
+metadata:
+  name: crawl-home-2
+spec:
+  mode: Managed
+  provider: kvm2-home
+  placement:
+    zone: beelink
+    machineType: small
+    image: debian-13
+  port: 3128
+  cloudInit:
+    secretRef:
+      name: crawl-squid-cloud-init
+  attributes:
+    geo: home
+    purpose: crawl
+    tier: general
+---
+apiVersion: crawl.example.com/v1alpha1
+kind: Proxy
+metadata:
+  name: crawl-home-3
+spec:
+  mode: Managed
+  provider: kvm2-home
+  placement:
+    zone: beelink
+    machineType: small
+    image: debian-13
+  port: 3128
+  cloudInit:
+    secretRef:
+      name: crawl-squid-cloud-init
+  attributes:
+    geo: home
+    purpose: crawl
+    tier: general
+---
+# A heavier member: two vCPUs and 2 GiB from the medium machine type, a
+# different base image, and room for more concurrent leases. tier=heavy is
+# what a crawler selects on to land here rather than on the general members.
+apiVersion: crawl.example.com/v1alpha1
+kind: Proxy
+metadata:
+  name: crawl-home-heavy-1
+spec:
+  mode: Managed
+  provider: kvm2-home
+  placement:
+    zone: beelink
+    machineType: medium
+    image: ubuntu-24.04
+  port: 3128
+  cloudInit:
+    secretRef:
+      name: crawl-squid-cloud-init
+  attributes:
+    geo: home
+    purpose: crawl
+    tier: heavy
+  maxLeases: 10
+---
+# A member that probes harder than the pool default (30s interval, 3 failures
+# to go unhealthy): a tighter loop takes a flapping proxy out of rotation
+# sooner, at the cost of more probe traffic. Everything under healthCheck is
+# defaulted, so set only what you want to change.
+#
+# This one also runs squid on a non-default port, which is why it carries its
+# own inline cloud-init: spec.port and the http_port in the config have to
+# agree, and the shared Secret above pins 3128.
+apiVersion: crawl.example.com/v1alpha1
+kind: Proxy
+metadata:
+  name: crawl-home-probe-fast-1
+spec:
+  mode: Managed
+  provider: kvm2-home
+  placement:
+    zone: beelink
+    machineType: small
+    image: debian-13
+  port: 8080
+  cloudInit:
+    inline: |
+      #cloud-config
+      packages:
+        - squid
+      write_files:
+        - path: /etc/squid/conf.d/proxy-operator.conf
+          content: |
+            http_port 8080
+            http_access allow all
+            via off
+            forwarded_for off
+      runcmd:
+        - systemctl restart squid
+  healthCheck:
+    intervalSeconds: 10
+    timeoutSeconds: 3
+    failureThreshold: 2
+  attributes:
+    geo: home
+    purpose: crawl
+    tier: general