samples: a kvm2 crawl pool, not just the single-proxy sample
proxy_kvm2.yaml shows one VM; nothing showed the shape a crawler actually leases from. Add a five-Proxy pool on the kvm2-home entry: four share one labelled cloud-init Secret, and the fifth carries its own inline block because it runs squid on 8080 and spec.port has to match the http_port a shared Secret would pin. It stays out of the samples kustomization so `apply -k config/samples` doesn't boot a pool along with the one-per-provider samples. All six objects validate against the installed CRD under --dry-run=server. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,19 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## 2026-09-18 07:44 UTC — a kvm2 crawl-pool sample
|
||||||
|
|
||||||
|
Until now the only kvm2 manifest was the one-VM `config/samples/proxy_kvm2.yaml`, which
|
||||||
|
left the multi-proxy shape — the one a crawler actually leases from — undocumented.
|
||||||
|
|
||||||
|
- Added `config/samples/proxy_kvm2_crawl_pool.yaml`: five Managed Proxies on the `kvm2-home`
|
||||||
|
entry, four sharing one labelled cloud-init Secret and one inline (it runs squid on 8080,
|
||||||
|
so it cannot share a Secret that pins `http_port 3128`). Covers the `tier` attribute the
|
||||||
|
discovery API selects on, a `medium`/`ubuntu-24.04` member with a raised `maxLeases`, and
|
||||||
|
a member with a tightened `healthCheck`.
|
||||||
|
- Kept out of `config/samples/kustomization.yaml` on purpose so `apply -k` doesn't boot a
|
||||||
|
whole pool; the exclusion is commented there next to the existing `providers-config.yaml`
|
||||||
|
one. Validated with `kubectl apply --dry-run=server`.
|
||||||
|
|
||||||
## 2026-09-17 20:36 UTC — kvm2: the four high findings from the code review, and its first run on a real hypervisor
|
## 2026-09-17 20:36 UTC — kvm2: the four high findings from the code review, and its first run on a real hypervisor
|
||||||
|
|
||||||
A scoped review of the kvm2 provider (`review/REPORT.md`: 52 findings, 4 high, 20 medium,
|
A scoped review of the kvm2 provider (`review/REPORT.md`: 52 findings, 4 high, 20 medium,
|
||||||
|
|||||||
@@ -1,6 +1,9 @@
|
|||||||
## Append samples of your project ##
|
## Append samples of your project ##
|
||||||
# providers-config.yaml is deliberately absent: it is a sample
|
# providers-config.yaml is deliberately absent: it is a sample
|
||||||
# --providers-config file, not a Kubernetes manifest.
|
# --providers-config file, not a Kubernetes manifest.
|
||||||
|
# proxy_kvm2_crawl_pool.yaml is deliberately absent too: it is a whole pool
|
||||||
|
# of VMs, so it stays an explicit `kubectl apply -f` rather than something
|
||||||
|
# `apply -k` boots along with the one-per-provider samples.
|
||||||
resources:
|
resources:
|
||||||
- proxy_kubernetes.yaml
|
- proxy_kubernetes.yaml
|
||||||
- proxy_gcp.yaml
|
- proxy_gcp.yaml
|
||||||
|
|||||||
@@ -0,0 +1,190 @@
|
|||||||
|
# A pool of Managed crawl proxies on the kvm2 provider — the multi-proxy
|
||||||
|
# counterpart to the single proxy_kvm2.yaml sample. Everything here boots on
|
||||||
|
# the one host the kvm2 entry configures (beelink); the proxies differ in the
|
||||||
|
# attributes crawlers select on, in size, and in image.
|
||||||
|
#
|
||||||
|
# Requires the same prerequisites as proxy_kvm2.yaml: a providers-config
|
||||||
|
# entry named "kvm2-home" and the kvm-ssh Secret that the kvm and kvm2
|
||||||
|
# providers share (docs/kvm2-provider.md).
|
||||||
|
#
|
||||||
|
# Deliberately NOT in kustomization.yaml: `kubectl apply -k config/samples`
|
||||||
|
# would boot every VM in here at once. Apply it on its own when you want the
|
||||||
|
# pool:
|
||||||
|
#
|
||||||
|
# kubectl apply -f config/samples/proxy_kvm2_crawl_pool.yaml
|
||||||
|
#
|
||||||
|
# Crawlers reach these through the discovery API by attribute, e.g.
|
||||||
|
# `GET /v1/proxies?attr.purpose=crawl&healthy=true`, or by leasing one with
|
||||||
|
# the same selector. Attribute matching is exact equality, so keep the keys
|
||||||
|
# consistent across the pool and vary only the values you want to select on.
|
||||||
|
#
|
||||||
|
# The first Proxy for an image on the host waits for that image to download;
|
||||||
|
# the rest reuse the cached copy. debian-13 and ubuntu-24.04 are separate
|
||||||
|
# downloads, so the ubuntu member below is the slow one on a cold host.
|
||||||
|
---
|
||||||
|
# One cloud-init Secret shared by the whole pool: the proxies below point at
|
||||||
|
# it with secretRef instead of each carrying its own copy of the squid
|
||||||
|
# config. The crawl.example.com/cloud-init label is mandatory — the manager's
|
||||||
|
# cache only holds Secrets that carry it, so an unlabelled Secret is
|
||||||
|
# invisible to the operator and the Proxy never resolves.
|
||||||
|
#
|
||||||
|
# Editing this Secret changes the resolved cloud-init of every Proxy that
|
||||||
|
# references it, and that replaces each of their VMs. That is the trade for
|
||||||
|
# sharing one definition: convenient for a config rollout, disruptive if you
|
||||||
|
# only meant to touch one member. Give a member its own Secret (or an inline
|
||||||
|
# block) when it needs to diverge.
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: crawl-squid-cloud-init
|
||||||
|
labels:
|
||||||
|
crawl.example.com/cloud-init: "true"
|
||||||
|
type: Opaque
|
||||||
|
stringData:
|
||||||
|
# The default key name; SecretKeySelector.key defaults to user-data, so
|
||||||
|
# the proxies below omit it.
|
||||||
|
user-data: |
|
||||||
|
#cloud-config
|
||||||
|
packages:
|
||||||
|
- squid
|
||||||
|
write_files:
|
||||||
|
- path: /etc/squid/conf.d/proxy-operator.conf
|
||||||
|
content: |
|
||||||
|
http_port 3128
|
||||||
|
http_access allow all
|
||||||
|
via off
|
||||||
|
forwarded_for off
|
||||||
|
runcmd:
|
||||||
|
- systemctl restart squid
|
||||||
|
# Optional: log in with `ssh debian@<ip>` (or `ubuntu@<ip>` on the
|
||||||
|
# ubuntu-24.04 member) to look around; `virsh console <name>` on the
|
||||||
|
# host works too.
|
||||||
|
# ssh_authorized_keys:
|
||||||
|
# - ssh-ed25519 AAAA... you@laptop
|
||||||
|
---
|
||||||
|
apiVersion: crawl.example.com/v1alpha1
|
||||||
|
kind: Proxy
|
||||||
|
metadata:
|
||||||
|
name: crawl-home-1
|
||||||
|
spec:
|
||||||
|
mode: Managed
|
||||||
|
provider: kvm2-home
|
||||||
|
placement:
|
||||||
|
zone: beelink
|
||||||
|
machineType: small
|
||||||
|
image: debian-13
|
||||||
|
port: 3128
|
||||||
|
cloudInit:
|
||||||
|
secretRef:
|
||||||
|
name: crawl-squid-cloud-init
|
||||||
|
attributes:
|
||||||
|
geo: home
|
||||||
|
purpose: crawl
|
||||||
|
tier: general
|
||||||
|
---
|
||||||
|
apiVersion: crawl.example.com/v1alpha1
|
||||||
|
kind: Proxy
|
||||||
|
metadata:
|
||||||
|
name: crawl-home-2
|
||||||
|
spec:
|
||||||
|
mode: Managed
|
||||||
|
provider: kvm2-home
|
||||||
|
placement:
|
||||||
|
zone: beelink
|
||||||
|
machineType: small
|
||||||
|
image: debian-13
|
||||||
|
port: 3128
|
||||||
|
cloudInit:
|
||||||
|
secretRef:
|
||||||
|
name: crawl-squid-cloud-init
|
||||||
|
attributes:
|
||||||
|
geo: home
|
||||||
|
purpose: crawl
|
||||||
|
tier: general
|
||||||
|
---
|
||||||
|
apiVersion: crawl.example.com/v1alpha1
|
||||||
|
kind: Proxy
|
||||||
|
metadata:
|
||||||
|
name: crawl-home-3
|
||||||
|
spec:
|
||||||
|
mode: Managed
|
||||||
|
provider: kvm2-home
|
||||||
|
placement:
|
||||||
|
zone: beelink
|
||||||
|
machineType: small
|
||||||
|
image: debian-13
|
||||||
|
port: 3128
|
||||||
|
cloudInit:
|
||||||
|
secretRef:
|
||||||
|
name: crawl-squid-cloud-init
|
||||||
|
attributes:
|
||||||
|
geo: home
|
||||||
|
purpose: crawl
|
||||||
|
tier: general
|
||||||
|
---
|
||||||
|
# A heavier member: two vCPUs and 2 GiB from the medium machine type, a
|
||||||
|
# different base image, and room for more concurrent leases. tier=heavy is
|
||||||
|
# what a crawler selects on to land here rather than on the general members.
|
||||||
|
apiVersion: crawl.example.com/v1alpha1
|
||||||
|
kind: Proxy
|
||||||
|
metadata:
|
||||||
|
name: crawl-home-heavy-1
|
||||||
|
spec:
|
||||||
|
mode: Managed
|
||||||
|
provider: kvm2-home
|
||||||
|
placement:
|
||||||
|
zone: beelink
|
||||||
|
machineType: medium
|
||||||
|
image: ubuntu-24.04
|
||||||
|
port: 3128
|
||||||
|
cloudInit:
|
||||||
|
secretRef:
|
||||||
|
name: crawl-squid-cloud-init
|
||||||
|
attributes:
|
||||||
|
geo: home
|
||||||
|
purpose: crawl
|
||||||
|
tier: heavy
|
||||||
|
maxLeases: 10
|
||||||
|
---
|
||||||
|
# A member that probes harder than the pool default (30s interval, 3 failures
|
||||||
|
# to go unhealthy): a tighter loop takes a flapping proxy out of rotation
|
||||||
|
# sooner, at the cost of more probe traffic. Everything under healthCheck is
|
||||||
|
# defaulted, so set only what you want to change.
|
||||||
|
#
|
||||||
|
# This one also runs squid on a non-default port, which is why it carries its
|
||||||
|
# own inline cloud-init: spec.port and the http_port in the config have to
|
||||||
|
# agree, and the shared Secret above pins 3128.
|
||||||
|
apiVersion: crawl.example.com/v1alpha1
|
||||||
|
kind: Proxy
|
||||||
|
metadata:
|
||||||
|
name: crawl-home-probe-fast-1
|
||||||
|
spec:
|
||||||
|
mode: Managed
|
||||||
|
provider: kvm2-home
|
||||||
|
placement:
|
||||||
|
zone: beelink
|
||||||
|
machineType: small
|
||||||
|
image: debian-13
|
||||||
|
port: 8080
|
||||||
|
cloudInit:
|
||||||
|
inline: |
|
||||||
|
#cloud-config
|
||||||
|
packages:
|
||||||
|
- squid
|
||||||
|
write_files:
|
||||||
|
- path: /etc/squid/conf.d/proxy-operator.conf
|
||||||
|
content: |
|
||||||
|
http_port 8080
|
||||||
|
http_access allow all
|
||||||
|
via off
|
||||||
|
forwarded_for off
|
||||||
|
runcmd:
|
||||||
|
- systemctl restart squid
|
||||||
|
healthCheck:
|
||||||
|
intervalSeconds: 10
|
||||||
|
timeoutSeconds: 3
|
||||||
|
failureThreshold: 2
|
||||||
|
attributes:
|
||||||
|
geo: home
|
||||||
|
purpose: crawl
|
||||||
|
tier: general
|
||||||
Reference in New Issue
Block a user