Architecture gets a provider-health section, component rows and four
decisions; the KVM provider doc gets a health-check section with the
reason table and the rotated-Secret gap; the testing guide lists the new
suites. CLAUDE.md's Architecture TODO and stale "only go.mod exists"
gotcha now point at the real docs, plus the RBAC-marker gotcha found
while wiring the monitor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--provider-check-interval (1m; 0 = off, else at least 10s) and
--provider-check-timeout (15s) are validated before the providers config
loads. Checkers are collected from the unwrapped providers in config
order, so checks never count as provider requests. Three scrape-time
gauges per {provider,target} read the monitor's snapshot: up, check
duration and check timestamp. The e2e suite waits for the kubernetes
ProviderHealth to turn Ready and asserts its gauge in the scrape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A leader-only runnable checks every configured provider concurrently
under one timeout per round (first round at start, then every interval),
keeps the latest results for the metrics collector, logs targets going
down, changing reason or coming back, and publishes one ProviderHealth
object per provider: created with a managed-by label when missing, then
a status merge patch. Per-target lastTransitionTime survives rounds in
which ready does not flip. Objects of providers that are no longer
configured are deleted once at start; with checks turned off, all are.
Reads go through the API reader, so no cluster-wide informer is needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each configured host is checked concurrently over the pooled SSH+libvirt
connection: libvirt version and hostname prove the daemon answers, then
the configured pool must exist and be running, with its free space
reported. Dial failures are now tagged with their stage (SSH material,
TCP, SSH handshake, libvirt connect) by a dialError that keeps the
message and unwraps to the cause, so classification is unchanged and the
check can say why a host is unreachable. A pool verdict is decided after
the call returns so it never drops a working connection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Checker is an optional interface next to Provider: Check returns one
verdict per target and never fails as a whole. The kubernetes provider
lists at most one managed Pod cluster-wide (the verb and scope GC relies
on); the GCP provider fetches a single page of at most one managed
instance through a new Probe seam method, since the SDK treats
MaxResults as a page size and the existing AggregatedList drains every
page.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One operator-owned object per configured provider will carry the
periodic check result: a Ready condition, per-target verdicts with their
own transition times, and a ready/total count for kubectl get. There is
no spec; only a viewer role is shipped since the operator is the sole
writer. Written by hand (kubebuilder is not installed), registered in
the existing AddKnownTypes call rather than a second init().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Periodic, leader-only checks of every configured provider (KVM per host
over the pooled SSH+libvirt connection including the storage pool,
kubernetes apiserver/RBAC, GCP credentials/API), surfaced through a new
cluster-scoped ProviderHealth CRD, Prometheus gauges and transition logs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default manager config gains a kvm-home provider reaching beelink
over Tailscale, the provider sample names the same host, and the
manager kustomization uses the locally built kvm-dev image.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collapse duplicated logic the review found in PR #5 without changing
behaviour: one releaseFinalizer helper for the four finalizer exits in
reconcileDelete, classify reusing isClassified and provider.Class, the
image URL check and format/firmware values shared from the provider
package, imageSource and the metadata parser embedding their config
structs instead of copying fields, and the SSH loader as a small struct
that keeps only the SSH fields and is seeded by New instead of loading
the key twice. Tracing builds its V(2)-only lists only at V(2).
Tests: drop a duplicate New test, fold near-identical tests into tables
(connection drops, metadata parsing, cached images), add mustSSHAuth,
and use wantClass where classes were checked by hand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merging the bootstrap into each Proxy's cloud-config was the wrong
mechanism twice over. Detection only matched a literal #cloud-config
first line, so a jinja template or a MIME archive still fell back to
vendor-data, where cloud-init's default mergers drop it — the exact
failure the merge existed to prevent. And the YAML round-trip re-typed
unquoted scalars: version: 1.10 reached the guest as 1.1, comments were
dropped and keys reordered.
Neither is worth patching, because vendor-data delivered as a shell
script bypasses the merger entirely: cloud-init hands vendor-data to its
own handler chain, writes scripts to a separate directory and runs them
from cc_scripts_vendor, after the user's packages and before the user's
runcmd. The merge could not have been rescued anyway — merge_how in
user-data governs only the first source, and merge_how in vendor-data is
discarded before the document is stored.
The Proxy's cloud-init now reaches the seed byte for byte in any format.
installGuestAgent (default true) turns the script off; a configured
vendorData travels beside it as a MIME multipart, declared text/plain so
cloud-init re-sniffs it and this code never classifies user content.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The kvm-ssh Secret is mounted optional, but kvm.New read the key and
known_hosts eagerly and registry.Build aborts the whole provider set on a
constructor error, so a missing or misnamed Secret exited the manager
before any controller started and took the kubernetes and gcp providers
down with it.
lazyAuth loads the material on the first connection and caches it once it
works, so a Secret created after the manager starts heals the affected
proxies on the next reconcile with no restart. A missing file classifies
transient for that reason; a key that does not parse is permanent.
New still checks both files and logs a WARNING: line when they are
unreadable, keeping the startup diagnostic the eager load was there for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A host name is pasted into the providerID hosts/<name>/domains/<vm>, which
parseProviderID splits on "/", so a name containing a slash produced an ID
that never parsed again: Get, Delete and GC all failed permanently and the
VM leaked with the Proxy's finalizer stuck.
The image map key has the same defect and lands in a file name rather than
an ID: base-<key>.<format> in the host's storage pool. Both now have to
match a plain identifier pattern. Machine-type keys are left alone; they
are only ever compared, never embedded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
reconcileDelete asked Get first and dropped the finalizer only on
not-found, so a permanent Get retried forever: a KVM host removed from
the config, or a providerID the provider cannot resolve, left the Proxy
in Terminating with no way out. It now tries Delete anyway, which is
idempotent and can still finish where Get cannot converge, and releases
the object when that succeeds.
crawl.example.com/skip-cleanup abandons the instance for the cases that
stay stuck, such as revoked credentials. It is checked before the
provider lookup, so it also frees a Proxy whose provider is gone from
the config.
In the KVM provider, Delete over an unconfigured host is now a no-op
success, mirroring removeVolumes over a missing pool. Get keeps failing
permanently there on purpose: the live path reads not-found as "rebuild
it", so a typo in hosts must not look like a missing VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In the kind cluster two KVM Proxies with real cloud-configs booted fine
but never left Provisioning: the guest agent was not installed, so the
provider could not learn the IP. cloud-init merges vendor-data under
user-data with its default mergers, which keep the user-data's value
for any key both define and never append lists, so a cloud-config with
its own `packages:` silently dropped the vendor-data's packages and
runcmd. The integration test passed only because its user-data was
empty.
withGuestAgent now appends qemu-guest-agent to `packages` and the
enable command to `runcmd` of the Proxy's own cloud-config (skipping
entries already present) and re-renders it. Vendor-data is no longer
defaulted; `vendorData` is plain extra data. User-data that is not a
cloud-config document (script, #include, MIME) still gets the bootstrap
as vendor-data. A cloud-config that is not a YAML mapping, or whose
packages/runcmd is not a list, fails the Proxy as ErrPermanent.
Validated in-cluster: both Proxies Ready and healthy in two minutes, 204
through squid from the LAN. Docs, sample comment, CHANGELOG and the
plan execution log (Run 7) updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The manager runs with a read-only root filesystem and the seed ISO
writer stages files in a temp directory, so the first in-cluster Create
failed with "creating iso writer: read-only file system". Mount a 64 MiB
emptyDir on /tmp.
That error is classified ErrPermanent by writeSeed, yet hostPool.call
treated every non-libvirt error as a transport failure and dropped the
cached SSH connection, redialling on each retry. Errors that already
carry a provider sentinel are local verdicts, not evidence about the
link; leave the connection alone for those.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/kvm-provider.md walks through key generation, ssh-copy-id,
ssh-keyscan under the address the operator dials, and the Secret from
the private key plus ~/.ssh/kvm_known_hosts. make kvm-ssh-secret runs
the same as an apply with KVM_SSH_KEY / KVM_KNOWN_HOSTS overrides and
refuses a .pub, the classic mistake. README mirrors the short form.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The known-gaps entry, decision log, sample comment and builder comment
blamed the host; the cause is trixie's GRUB config ("terminal_output
gfxterm serial" with a serial console and no video device), reported
and reproduced upstream and still open. UEFI remains the default for the
reason recorded; bookworm images are noted as unaffected.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 5 on the real host bisected the guest hang: the Debian 13 image
hangs after GRUB under SeaBIOS with or without our seed and with either
CPU model, and boots under OVMF. The image's legacy boot chain is the
problem, not the provider.
images.<key>.firmware is "uefi" (default) or "bios". The domain XML
emits <os firmware="efi"> with secure boot off, so libvirt selects the
OVMF binary and manages the NVRAM file, which the existing nvram undefine
flag removes. Direct-URL images default to UEFI too. OVMF joins the
hypervisor prerequisites; sample, config table, known gaps, decision log
and the execution log record the finding.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first live run showed the trace working but noisy: testr printed
ts/level/msg and the per-run constants on every line, the error text
appeared twice, host appeared twice on image-cache lines, and the Go
method name was less useful than the libvirt function it maps to.
The tracer now logs rpc=virDomainLookupByName (go-libvirt's names are the
public API names without the vir prefix), error plus libvirtError (code
name) and libvirtCode, and image.go stops re-adding host. The integration
test replaces testr with a compact logr sink: time, logger name, the rpc
as headline, then fields, with provider/host/name omitted since the first
line of the run states them. Docs and execution log updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A tracingAPI decorator wraps the 17-method libvirt seam and logs one
"libvirt call" line per RPC after it returns: method, the identifiers it
was given, curated results, duration, and on failure the error with its
libvirt code, name and message. At V(2) the same line carries the domain
and volume XML, metadata and addresses; upload bodies are never logged.
The pool applies it in call() when V(1) is enabled and logs connect,
dial-failure and drop events, taking the caller's logger explicitly like
every other helper in the package.
The integration test installs a testr logger at KVM_TEST_LOG_LEVEL
(default 1) so the trace interleaves with t.Log output, which also ends
the "log.SetLogger(...) was never called" warning. Docs gained a Logging
section and the test env var; the KVM provider execution log records run
4 (guest hang at GRUB) that prompted this.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run 3 on the real host got past Create and failed in Get: libvirt 11.6
selects the <instance> element by our URI in DomainGetMetadata but hands
it back with no namespace declaration, and the parser required the URI
on the element. The lookup already guarantees the namespace, so match on
the local name only. The fake now strips the declaration the way libvirt
does, and a new test pins the bare form. Execution log updated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first run against a real hypervisor (beelink, Ubuntu, libvirt 11.6)
failed at DomainCreate with "Could not open base image: Permission denied"
even with correct DAC ownership. libvirt's virt-aa-helper only grants QEMU
access to disk sources that carry a path; the <disk type="volume"> form
chosen in Step 2 gives it pool and volume names instead, so the overlay and
its backing image get no AppArmor rule on any stock Debian/Ubuntu host.
buildDomainXML now takes DiskPath/SeedPath and emits type="file" disks.
ensureOverlay and writeSeed return the volume handles and Create resolves
both paths with StorageVolGetPath before defining the domain. The domain
XML test rejects any volume/pool reference so this cannot drift back; the
Create test asserts the sources equal the paths libvirt reported.
Execution log: Step 2 note marked reverted, Step 7 entry opened with the
two failed runs and the diagnosis.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Registers "kvm" in the provider builtins, mounts an optional kvm-ssh
Secret at /etc/proxy-operator/kvm, adds a kvm-home providers-config
sample and a proxy_kvm.yaml Proxy sample, documents the provider in
docs/kvm-provider.md (prerequisites, config reference, image cache and
its do-not-replace-in-use rule, known gaps), and updates the README
and architecture provider sections and decision log.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Behind -tags=integration and self-skipping unless KVM_TEST_* names a
host: one sequential lifecycle (image fetch, Create, idempotent
re-Create, Running with IP, ListByTag, Delete, pool state afterwards),
driven through the provider directly so it needs no Step 5 wiring.
Adds `make test-integration-kvm` and documents the run and the
deviation from the testcontainers convention in docs/testing.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Create looks the domain up first and only starts it when it exists;
otherwise it caches the image, reuses or creates the overlay, rebuilds
the seed ISO unconditionally (a cancelled upload can leave a truncated
seed that no later attempt would repair), defines and starts the domain.
Get maps libvirt state plus the first routable IPv4 from ARP or the
guest agent onto the provider states. Delete destroys, undefines and
removes the per-VM volumes by name, tolerating every already-gone case.
ListByTag keeps managed domains across all hosts, skipping domains it
cannot read and failing the sweep only for an unreachable host.
Also corrects the Step 2 seed determinism test: the iso9660 writer
stamps the build time into the volume descriptor, so builds differ by a
few bytes; the test now compares label, files and size.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
resolveImage maps placement.image to a configured key or URL; imageCache
streams the download through StorageVolUpload with a sha256 tee, writes
a marker volume when the upload is verified, and re-fetches when the
marker is missing or the configured checksum changed. Errors carry
taxonomy sentinels that classify now honours. Adds the in-memory fake
libvirt API used from here on.
Step 3b of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Per-host connection pool over SSH with the libvirt unix socket forwarded
through the session, lazy dial and redial after disconnect, context
cancellation for go-libvirt's context-less RPCs, handshake deadline and
SSH keepalives. Error classification onto the provider taxonomy. Tested
with a fake dialer and an in-process SSH server that forwards unix sockets.
Step 3 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Introduces internal/provider/kvm with the side-effect-free pieces of the
provider: providerID format/parse, the namespaced <metadata> element that
carries the GC contract, q35 domain XML over pool volumes, volume XML for
overlays and raw seeds, the NoCloud seed ISO built with kdomanski/iso9660,
and config defaults. All unit-tested.
Step 2 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds the kvm block to the providers config (SSH access, images map,
machine-type profiles, hosts list) with validation and table tests.
validate() now rejects mismatched type-specific blocks generically via
setBlocks() instead of pairwise checks.
Step 1 of docs/plans/2026-09-04-1404-kvm-provider.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Design for a kvm provider type: SSH-forwarded libvirt socket, multi-host
config with placement.zone selecting the host, bridged networking with
ARP/guest-agent IP discovery, operator-fetched distro cloud images cached
in the host's storage pool, and a NoCloud seed ISO carrying the CRD's
cloud-init user-data.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
New Providers section: named-instance config model, a table of the
implemented kubernetes and gcp providers, and a four-step guide for
adding a new backend (interface contract, GC tagging, error taxonomy,
wiring in config.go and main.go).
Co-Authored-By: Claude <noreply@anthropic.com>
run-demo.sh opens a 2x2 tmux grid: egress-IP table looping in a
netshoot pod, kubectl get px watch, and both create scripts running
with COUNT (default 4) proxies. show-egress-ips-table.sh is the
pane-sized one-line-per-proxy variant used by the driver.
Co-Authored-By: Claude <noreply@anthropic.com>
docs/demo/show-egress-ips.sh probes each healthy proxy from the
discovery API against an IP-echo site; create-kubernetes-proxies.sh and
create-gcp-proxies.sh bulk-create demo Proxies, the gcp one spreading
them across randomly picked EU zones.
Co-Authored-By: Claude <noreply@anthropic.com>
Full client-facing reference: auth, all routes with schemas and curl
examples, selection/cooldown semantics, caveats. README and
architecture.md link to it.
Co-Authored-By: Claude <noreply@anthropic.com>
Distilled from the house pattern across sibling projects: tag push +
workflow_dispatch triggers, REGISTRY_TOKEN login, raw docker build/push
to gitea.home.hrajfrisbee.cz. Adds a lightweight test gate, an immutable
sha-<12> tag, :latest only on real tag pushes, and a concurrency group.
Co-Authored-By: Claude <noreply@anthropic.com>
The option.WithLogger logger also reaches cloud.google.com/go/auth,
which logged its token exchange at Debug — JWT assertion and bearer
token included. wireLogger now allowlists only the compute client's
api request/response records at Debug (fail-closed for future SDK
additions); Warn/Error pass through. String fields over 1KiB (e.g.
Shielded-VM UEFI dbx blobs) are elided recursively by default; the new
--gcp-wire-log-full-payloads flag restores verbatim payloads.
Co-Authored-By: Claude <noreply@anthropic.com>
Inject an option.WithLogger slog logger bridged to the zap sink via
logr.ToSlogHandler with a V(1) shift, so the SDK's Debug-level
"api request"/"api response" records (URL, headers, full payloads)
appear only at --zap-log-level=5. Startup warning when active, since
raw insert payloads include cloud-init user-data. Note WithLogger
overrides GOOGLE_SDK_GO_LOGGING_LEVEL for this client.
Co-Authored-By: Claude <noreply@anthropic.com>
New internal/version package: ldflags-stamped Commit with a
debug.ReadBuildInfo VCS fallback for host builds. Startup log line
carries commit + Go version; --version prints the hash and exits.
Makefile computes GIT_COMMIT (12 chars, -dirty on any local change) and
passes it to docker-build/buildx; Dockerfile injects it via -ldflags and
an org.opencontainers.image.revision label. make build now uses ./cmd —
file-argument builds skip Go's automatic VCS stamp.
Co-Authored-By: Claude <noreply@anthropic.com>
One V(1) line per GCP API call (insert/get/delete/aggregatedList) with
outcome and operation name, V(2) request/per-instance detail, and raw
googleapi status+reasons logged before classify collapses them. Curated
fields only — cloud-init user-data never reaches logs (test-enforced).
Co-Authored-By: Claude <noreply@anthropic.com>
config/manager/kustomization.yaml: commit the images: stanza that
`kustomize edit set image` (run by make deploy, including inside make
test-e2e) writes into this tracked file. It showed up as unexplained
drift twice; committing it once ends that -- the edit is idempotent, so
future deploy/e2e runs produce no diff. The example.com image name is
the e2e suite's placeholder default and gets overridden by IMG= on any
real deploy.
Execution log: the Status checklist's Step 3 line still said "Mock
provider" from before the pivot; a fresh session resuming from the
checklist alone would have been misled.
Co-Authored-By: Claude <noreply@anthropic.com>
Follows the approved lean-down plan
(docs/plans/2026-08-08-1335-lean-scaffold-cleanup.md). Removes from the
application's deployed footprint:
- config/network-policy/ and its commented enable line -- the user does
not need network policies at the moment.
- The webhook-only halves of config/default/kustomization.yaml: the
commented ../webhook and ../certmanager resource lines, the
manager_webhook_patch.yaml reference, the serving-cert ->
Validating/Mutating WebhookConfiguration cainjection replacement
blocks, and the crdkustomizecainjection* scaffold markers -- anchors
only for `kubebuilder create webhook`, which is a permanent non-goal.
- The two commented [WEBHOOK] blocks in config/crd/kustomization.yaml
plus the now-empty patches: key; kept the one-line
crdkustomizeresource marker since `kubebuilder create api` could
legitimately run again.
- config/crd/kustomizeconfig.yaml, whose only consumer was the removed
configurations: block.
Explicitly kept per user direction: all of config/prometheus/, the
paired metrics-TLS-via-cert-manager plumbing (cert_metrics_manager_patch
+ the metrics-certs/ServiceMonitor replacement halves), all RBAC
manifests including the admin/editor/viewer helper roles, and all
developer tooling.
Also records in the execution log why the webhook machinery existed at
all: kubebuilder init emits it unconditionally, verified against the
v4.15.0 binary that no init flag can suppress it -- scaffold-then-prune
is the only supported path, and the pruning pass should have happened
at Step 0.
Verified: kustomize build clean on config/default and config/crd,
go build/vet clean with and without -tags=e2e, make test green with
coverage identical to pre-cleanup.
Co-Authored-By: Claude <noreply@anthropic.com>
Approved plan for stripping the remaining webhook-only scaffold remnants
and config/network-policy/ from the application footprint, with explicit
keep decisions for prometheus/monitoring manifests, the paired
metrics-TLS plumbing, all RBAC manifests, and all developer tooling.
Also records why the webhook machinery existed at all (kubebuilder init
emits it unconditionally; verified no init flag can suppress it) and the
Step 0 process gap that let it survive until now.
Co-Authored-By: Claude <noreply@anthropic.com>
kubebuilder's generic scaffold defensively wires up webhook TLS-cert
machinery and an unconditional cert-manager install in the e2e suite, in
case a project grows admission webhooks later. This one never will --
the spec's non-goals explicitly rule out admission webhooks and
cert-manager wiring -- so none of it does anything. Verified before
removing: no config/webhook/, no +kubebuilder:webhook markers anywhere,
and config/*/kustomization.yaml's [CERTMANAGER] blocks are all inert
(never uncommented).
cmd/main.go: drops the webhook import, the three webhook-cert-* flags,
and the WebhookServer wiring on ctrl.Options -- the manager now runs
with no webhook server, correctly, since nothing registers one. Left
the metrics-cert flags alone; those are unrelated to webhooks.
test/e2e/e2e_suite_test.go: drops the unconditional cert-manager
install/uninstall around the suite.
test/utils/utils.go: drops the now-dead InstallCertManager/
UninstallCertManager/IsCertManagerCRDsInstalled and their warnError
helper, plus UncommentCode -- unrelated to cert-manager, but found to
have zero callers even before this cleanup.
Left the inert commented-out [WEBHOOK]/[CERTMANAGER] kustomize blocks
and kubebuilder's scaffold marker comments alone: pure comments, no
runtime behavior, unlike the cert-manager install this actually removed.
Verified clean with both build tags (go build/vet, and -tags=e2e for
test/e2e). make test unchanged and green.
Co-Authored-By: Claude <noreply@anthropic.com>
Create/Get/Delete/ListByTag against real corev1.Pod objects in the same
cluster the operator runs in, running an ubuntu/squid container -- picked
by actually checking Docker Hub metadata (Canonical-published, rebuilt
the same day this was decided, 50M+ pulls) rather than guessing an image
reference. It's a public image, so kind nodes pull it directly with no
build/load step.
providerID is "<namespace>/<podName>", parsed via
cache.SplitMetaNamespaceKey -- the same self-contained-providerID
reasoning the plan already calls for on the GCP provider's zone-qualified
IDs. Pod state maps to InstanceState with Succeeded/Failed/Unknown all
collapsing to Terminated, since the reconciler already treats Stopped and
Terminated identically; Running-without-PodIP maps to Provisioning so an
empty IP is never published.
The client is built internally via ctrl.GetConfig() (in-cluster or local
kubeconfig, whichever applies), not threaded through the registry
Constructor signature -- this is what lets `make run` against a local
kind cluster and running in-cluster share the exact same code path with
no provider-specific wiring in cmd/main.go. New() is deliberately
untested (0% coverage): it's the one function that must never run under
`go test`, since it would happily connect to whatever cluster the
developer's kubeconfig points at. Tests construct Provider via an
unexported newWithClient(client, cfg) instead.
ListByTag lists Pods across every namespace (orphan GC needs to find
every tagged Pod regardless of where it landed), which means this
provider's RBAC has to be a ClusterRole rather than namespace-scoped --
flagged now, wired in Step 10.
provider.Config gains KubernetesConfig (replacing MockConfig) and drops
the FailWith*/fault-injection surface entirely, since that need is now
served by a small in-test stub Provider for reconciler tests (Step 4),
not a config-driven mechanism on a real provider package.
Tests use sigs.k8s.io/controller-runtime/pkg/client/fake -- real Pod
objects, the real client.Client interface -- at 77.6% coverage.
make test green across the whole repo.
Co-Authored-By: Claude <noreply@anthropic.com>
The mock provider (state simulated via an injectable clock, a hand-rolled
shared/refcounted CONNECT-proxy listener per port to work around macOS's
loopback restrictions) worked, but the user felt it was too far removed
from the real system to build confidence in, and doesn't need the
automated test suite to stay fast enough to justify that complexity — a
kind-based verification pass "once in a while" is an acceptable trade for
tests that actually look like the final product.
Replacing it with a provider that creates real Pods in the same cluster,
running an actual Squid container. internal/provider/registry was already
designed to have zero dependency on any concrete provider package, so
removing this one required no changes anywhere else in the tree — go
build is clean with nothing implementing provider.Provider yet.
docs/plans/2026-08-07-1747-proxy-operator.md's Step 3 (and every other
reference to the mock provider throughout the plan) is updated in this
same commit to describe the replacement. Narrative on why and the
replacement's design lands in docs/plans-executions once it's built.
Co-Authored-By: Claude <noreply@anthropic.com>
An in-memory provider.Provider whose state (Provisioning -> Running ->
Terminated -> purged) is a pure function of an injectable clock, not
background timers, so it's deterministic under tests and correct under
real time with no goroutine lifecycle to leak.
Once an instance is observed Running, it lazily acquires a real HTTP
CONNECT proxy listener so the health engine's through-the-proxy probe
(later steps) genuinely tunnels a request end to end, instead of the
healthcheck being simulated or bypassed for local development.
Redesigned the listener sharing model from what the plan assumed: the
plan's "one loopback IP per instance" doesn't work on macOS (only
127.0.0.1 binds without a privileged ifconfig alias, unlike Linux where
the whole 127.0.0.0/8 routes to loopback by default), and there's no
channel for a provider to report a port back to the reconciler anyway
(EffectivePort() is spec-only). Instances now share one real listener
per port, reference-counted at the package level rather than per
Provider instance, since a bound TCP port is a genuinely process-global
OS resource -- two separately configured mock-typed provider entries
must not both try to bind the same default port.
Fault injection wired both ways: MockConfig.FailNextCreates/FailWith for
demos, InjectCreateFailures(n, class) for tests. Create is idempotent by
name.
Caught and fixed a real test flake (not a logic bug): the freePort test
helper asked the OS for a free port via bind-then-close, a TOCTOU race
under t.Parallel() that let two tests collide on the same "free" port.
Replaced it with a monotonic counter, since these tests only need
uniqueness within the test run.
internal/provider/mock at 91.1% coverage, including an end-to-end test
that opens real sockets: Create -> Get past provisionDelay -> a real
http.Client tunnelling a CONNECT through the mock to a real TLS origin.
make test green across the whole repo.
Co-Authored-By: Claude <noreply@anthropic.com>
The Provider interface (Create/Get/Delete/ListByTag), Instance, and
CreateRequest that every cloud backend implements — kept independent of
api/v1alpha1 so this package has no CRD-type coupling.
Error taxonomy (ErrNotFound/ErrQuotaExceeded/ErrTransient/ErrPermanent)
wrapped via a multi-error Unwrap() []error, so errors.Is and errors.As
both work off the same value: the reconciler branches on classification,
logs keep the underlying SDK error. Unclassified errors default to
ErrTransient — retrying is always safer than latching Failed.
Deterministic instance naming (SHA-256 -> base32 -> 16 chars, 22 total
with the "proxy-" prefix) satisfying GCP's RFC1035 name rules with
headroom, and idempotency-tested across 10k UIDs with zero collisions.
--providers-config YAML parsing (config.go) with fail-fast validation:
unknown type, duplicate name, missing gcp.project, mismatched
type/config-block, and strict-mode rejection of unknown keys.
internal/provider/registry/registry.go takes its type->constructor map
as a parameter rather than hardcoding it, so the package has zero import
on internal/provider/mock or internal/provider/gcp (neither exists yet —
mock is Step 3, gcp is Step 8) and compiles today. Explicit wiring moves
to the composition root in cmd/main.go (Step 10).
Deferred internal/provider/metrics.go (the WithMetrics decorator) to
Step 9, where the Prometheus vectors it needs actually get built —
nothing in this step depends on it.
internal/provider at 96.2% coverage, internal/provider/registry at 100%.
make test green.
Co-Authored-By: Claude <noreply@anthropic.com>
Full ProxySpec/ProxyStatus/Proxy types per the plan: PlacementSpec,
CloudInitSpec, EndpointSpec, HealthCheckSpec, SecretKeySelector, all
defaults, and 7 CEL XValidation rules enforcing mode/provider
immutability, provider/endpoint required-iff-Managed/External, and
cloud-init exactly-one-of inline/secretRef.
Applies the four corrections identified during planning that would
otherwise be silent bugs: MaxLeases as *int32 (so an explicit 0 survives
Go round-trips instead of re-defaulting to 5), HealthCheck's
default={} marker (so nested defaults apply even when the field is
omitted entirely), MinLength=1 on Provider/CloudInit.Inline (so the CEL
has() checks stay simple), and listType=map on Conditions.
Adds pure helpers (EffectivePort, EffectiveHost, HealthCheckOrDefault,
MaxLeasesOrDefault) with table-driven tests, for use by the health
engine, discovery API, and spec-hash computation in later steps.
Patches the scaffolded placeholder controller test's resource literal to
a schema-valid spec so it survives the new CRD validation — the test
itself is rewritten wholesale in Step 4 alongside the real reconciler.
Regenerated deepcopy and the CRD; make test green (envtest confirmed all
7 CEL rules enforced by a real apiserver).
Co-Authored-By: Claude <noreply@anthropic.com>
Completion status previously lived only in my in-session TodoWrite list
(ephemeral, doesn't survive the session) or had to be inferred from which
steps had an entry in the execution log. Neither is a durable, explicit
answer to "what's left."
Adds a markdown checklist at the top of each plans-executions file, one
line per plan step, checked in the same commit as that step's entry. A
new session resuming this plan can read the file alone and know exactly
where to pick up.
Co-Authored-By: Claude <noreply@anthropic.com>
Codifies the pattern the rewritten Step 0 entry demonstrated: one line of
context before each command snippet explaining why it was run that way,
explicit call-outs when something deviated from the plan rather than
presenting the working command as the first thing tried, and a closing
"worth noting" paragraph. Points at the Step 0 entry itself as the
reference example so future steps match its depth without re-deriving it.
Co-Authored-By: Claude <noreply@anthropic.com>
The Step 0 entry described what happened in prose but omitted the actual
kubebuilder/go install invocations, which is exactly the detail a future
reader would want to copy and reproduce. Codifies "include the real
command line, not a paraphrase" in CLAUDE.md and rewrites the Step 0
entry with the exact commands run, including the go install path
correction (kubebuilder v4.15.0 is the module root now, not
.../cmd/kubebuilder) and the controller-gen invocations kubebuilder ran
on its own.
Co-Authored-By: Claude <noreply@anthropic.com>
The plan for this build was written to docs/plans/ only after Step 0's
implementation had already started, instead of as its own first action
right after ExitPlanMode. CLAUDE.md said *where* to save plans but not
*when* relative to other work, so that ordering wasn't actually enforced.
Makes it explicit: copying the plan into docs/plans/ and committing it is
its own checkpoint that blocks starting Step 0.
Also adds a docs/plans-executions/ convention: one running, chronological
summary file per plan, appended to after each completed step, for a
human-readable narrative of how the plan actually went (judgment calls,
spec gaps, surprises) without duplicating the plan or the diff.
Backfills the Step 0 entry for the proxy-operator plan retroactively.
Co-Authored-By: Claude <noreply@anthropic.com>
Bootstraps the empty repo into a kubebuilder go/v4 project: group
crawl.example.com, version v1alpha1, kind Proxy (namespaced). Keeps the
existing module path and preserves the repo's Go/testing/changelog
conventions from CLAUDE.md untouched.
Drops the scaffolded GitHub Actions workflows since the remote is Gitea,
not GitHub. Everything else is default kubebuilder output, unmodified,
so later diffs stay reviewable against a known baseline.
Full implementation plan: docs/plans/2026-08-07-1747-proxy-operator.md
Co-Authored-By: Claude <noreply@anthropic.com>
kacerrandClaude Opus 5