# Build prompt: `proxy-operator` — Kubernetes operator for managing crawling-proxy VMs

You are building a production-quality **prototype** of a Kubernetes operator in Go. Read this entire spec before writing any code. Everything below is a requirement unless explicitly marked "non-goal" or "nice-to-have". Where this spec is silent, prefer the boring, idiomatic kubebuilder/controller-runtime convention over cleverness.

## 1. Context and purpose

A crawling department uses a fleet of small HTTP proxy VMs spread across cloud providers (GCP today, up to ~5 providers soon) to work around rate limiting. The fleet is small (tens of VMs). This operator makes those VMs first-class Kubernetes objects so they can be managed via GitOps:

- Provision/deprovision proxy VMs from a spec (image + cloud-init) via pluggable cloud providers.
- Track their state: IP, provisioning phase, health.
- Actively healthcheck each proxy **through the proxy** (a real HTTP request via the proxy, not a TCP dial).
- Track proxies provisioned outside the operator ("external" proxies) as equal citizens for discovery/health.
- Expose an HTTP discovery API for crawler clients: **list** healthy proxies filtered by labels, and **lease** a proxy (TTL-based assignment with server-side usage tracking).

Design principle: proxies are **immutable cattle**. Any meaningful spec change means replace (delete + recreate the VM), never in-place mutation. This is deliberate — do not build in-place update logic.

## 2. Tech stack (pin these — do not silently downgrade)

- Go **1.26** (`go 1.26` in go.mod)
- **kubebuilder v4** scaffold (go/v4 plugin), latest release (v4.15+)
- **controller-runtime v0.24.x**, k8s.io/* **v0.36.x** (Kubernetes 1.36 API level)
- GCP: `cloud.google.com/go/compute/apiv1` (the modern Cloud Client Library, **not** the legacy `google.golang.org/api/compute/v1` unless a needed call is missing there)
- CRD validation via **CEL validation rules** and kubebuilder markers. **No admission webhooks** in the prototype.
- Tests: standard `testing` + `envtest` for the controller. No test framework dependencies beyond what kubebuilder scaffolds (ginkgo is acceptable since the scaffold generates it, but table-driven std tests are preferred for units).

If any pinned version is unavailable in your environment, use the closest available and record the substitution prominently in the README.

## 3. Repository layout

Standard kubebuilder go/v4 layout. Module path: `github.com/CHANGEME/proxy-operator` (make it trivially renameable — no hardcoded module strings outside go.mod and imports).

```
api/v1alpha1/           # Proxy types
internal/controller/    # Proxy reconciler
internal/provider/      # provider interface + registry
internal/provider/mock/ # in-memory provider
internal/provider/gcp/  # GCP provider
internal/health/        # healthcheck engine
internal/discovery/     # HTTP API (list + lease)
internal/lease/         # lease store (in-memory, interface-first)
config/                 # CRDs, RBAC, manager kustomize (scaffold-generated, kept working)
config/samples/         # sample Proxy CRs: mock, gcp, external
docs/architecture.md    # short: components, data flow, one ASCII diagram
```

## 4. The `Proxy` CRD

Group `crawl.example.com`, version `v1alpha1`, kind `Proxy`, cluster-scoped: **no** — make it **namespaced** (fleet lives in one namespace; namespacing keeps RBAC simple).

### Spec

```go
type ProxySpec struct {
    // Provisioning mode: "Managed" (operator creates the VM) or "External"
    // (VM exists elsewhere; operator only tracks/healthchecks it).
    // Immutable after creation (enforce with CEL).
    Mode ProvisioningMode `json:"mode"`

    // Provider name matching a configured provider ("mock", "gcp-eu", ...).
    // Required iff mode==Managed. Immutable (CEL).
    Provider string `json:"provider,omitempty"`

    // Provider-opaque placement/size settings. Keep this a small typed struct,
    // NOT map[string]string: Region, Zone, MachineType, Image. Providers may
    // ignore fields that don't apply to them.
    Placement *PlacementSpec `json:"placement,omitempty"`

    // Cloud-init user-data. Either inline or from a Secret key. Exactly one
    // (CEL). Changing it on a Managed proxy triggers replacement.
    CloudInit *CloudInitSpec `json:"cloudInit,omitempty"`

    // Endpoint config for External mode: host, port. Required iff External (CEL).
    Endpoint *EndpointSpec `json:"endpoint,omitempty"`

    // Proxy port for Managed mode (the port the proxy listens on once the VM
    // is up). Default 3128.
    Port int32 `json:"port,omitempty"`

    // Selection attributes exposed to the discovery API (geo, asn, purpose...).
    // Deliberately separate from k8s object labels, which stay an operator
    // implementation concern.
    Attributes map[string]string `json:"attributes,omitempty"`

    HealthCheck *HealthCheckSpec `json:"healthCheck,omitempty"` // probeURL, interval, timeout, failureThreshold, successThreshold — all defaulted
    // Max concurrent leases handed out for this proxy. Default 5. 0 = unleasable (list-only visibility).
    MaxLeases int32 `json:"maxLeases,omitempty"`
}
```

### Status

```go
type ProxyStatus struct {
    Phase       ProxyPhase         `json:"phase,omitempty"` // Pending, Provisioning, Ready, Unhealthy, Deleting, Failed
    ProviderID  string             `json:"providerID,omitempty"` // opaque cloud resource ID
    IP          string             `json:"ip,omitempty"`
    Conditions  []metav1.Condition `json:"conditions,omitempty"` // Provisioned, Healthy — standard metav1 conditions with reasons
    LastHealthCheckTime *metav1.Time `json:"lastHealthCheckTime,omitempty"`
    LatencyMillis       int64        `json:"latencyMillis,omitempty"`
    ObservedGeneration  int64        `json:"observedGeneration,omitempty"`
}
```

Printer columns: Mode, Provider, Phase, IP, Healthy, Age.

**Status-write discipline:** healthchecks run frequently; do not write status on every probe. Write only on transition (healthy↔unhealthy, latency bucket change of >50%, phase change). This matters — unbounded status churn is a known operator anti-pattern.

## 5. Provider interface

Keep it brutally minimal. This is the contract five future providers must implement, so resist enrichment:

```go
type Provider interface {
    // Create starts VM creation. MUST be idempotent: name derives
    // deterministically from the CR ("proxy-" + short hash of CR UID), so a
    // repeat call after a crash finds the existing VM instead of duplicating.
    // May return before the VM is running.
    Create(ctx context.Context, req CreateRequest) (providerID string, err error)

    // Get returns current state. Returning (nil, ErrNotFound) is normal.
    Get(ctx context.Context, providerID string) (*Instance, error) // Instance: ID, IP, State (Provisioning|Running|Stopped|Terminated)

    // Delete is idempotent; deleting a non-existent instance is not an error.
    Delete(ctx context.Context, providerID string) error

    // ListByTag returns all instances this operator ever tagged, for orphan GC.
    ListByTag(ctx context.Context) ([]Instance, error)
}
```

Error contract: providers wrap errors into a small taxonomy the reconciler can branch on — `ErrNotFound`, `ErrQuotaExceeded` (retry slow: requeue ≥5 min), `ErrTransient` (retry with backoff), `ErrPermanent` (set Failed phase, stop retrying, surface in condition reason). Use `errors.Is`-compatible sentinel/wrapper types.

Every cloud resource a provider creates MUST carry a tag/label `proxy-operator-uid=<CR UID>` plus `proxy-operator-managed=true`. This is the GC contract.

### Provider configuration

Static YAML config file mounted into the manager pod, path from `--providers-config`. Named provider instances (so "gcp-eu" and "gcp-us" can be two configs of type `gcp`):

```yaml
providers:
  - name: mock
    type: mock
  - name: gcp-eu
    type: gcp
    gcp:
      project: my-project
      # auth: Application Default Credentials (workload identity in-cluster,
      #       gcloud ADC locally). No key-file plumbing in the prototype.
```

Parse at startup, fail fast on unknown `type`. A registry maps type → constructor.

### Mock provider

In-memory, thread-safe. Simulates async provisioning: instance is `Provisioning` for a configurable duration (default 5s) then `Running` with a fake IP from a private range. Supports fault injection via config (fail next N creates, inject quota error) — needed for controller tests.

### GCP provider

`cloud.google.com/go/compute/apiv1`. Insert instance with: machine type, zone, image, network tag, labels (the GC tags), cloud-init via `user-data` metadata key, ephemeral external IP. Wait for the insert Operation **without blocking reconcile**: Create returns after the operation is submitted; reconciler discovers readiness via Get polling (requeue). Delete likewise fire-and-forget + poll. Keep the implementation to the minimal calls: instances.Insert, instances.Get, instances.Delete, instances.AggregatedList (filtered by label) — nothing else.

## 6. Reconciler semantics

- **Finalizer** `crawl.example.com/proxy-cleanup` on Managed proxies; on delete, call provider Delete, requeue until Get returns NotFound, then remove finalizer. External proxies get no finalizer.
- **State machine, not step list.** Every reconcile derives desired action from (spec, status, provider Get). No multi-step sequences relying on in-memory state — the controller must resume correctly from any crash point.
- **Replacement on change:** compute a spec-hash (placement + cloudInit + image + port) and store it in an annotation. Hash mismatch on a Ready proxy → delete VM, clear providerID, re-provision. Document this loudly in the README (changing a proxy = its IP changes).
- **External mode:** skip provisioning entirely; phase goes Pending → Ready(when first healthcheck passes)/Unhealthy. IP comes from `spec.endpoint`.
- **Orphan GC:** a manager `Runnable` (not part of reconcile) sweeps every 10 min: for each provider, `ListByTag`, kill instances whose `proxy-operator-uid` matches no live CR (check deletionTimestamp semantics carefully; skip instances younger than 10 min to avoid racing in-flight creates). Log every kill at Warn with provider ID and UID.
- Standard controller hygiene: exponential backoff via requeue-with-error, rate-limited workqueue defaults, `MaxConcurrentReconciles: 3`.

## 7. Health engine

A manager `Runnable` running one probe loop (single goroutine with a timer wheel or per-proxy tickers — your call, but bounded goroutines). For each proxy with an IP:

- Build an `http.Client` with `Transport.Proxy` pointing at `http://<ip>:<port>`, per-probe timeout from spec.
- `GET` the probe URL (default `https://www.gstatic.com/generate_204`, overridable per-proxy). Success = expected status code (default 204/200) within timeout. This exercises a real CONNECT through the proxy — the whole point; a proxy that TCP-accepts but can't egress must go Unhealthy.
- Threshold logic (failureThreshold consecutive fails → Unhealthy; successThreshold → Healthy) lives in the engine; it pushes transitions to the reconciler via a channel or does a direct conditional status patch — pick one and document why.
- Record latency (EWMA or last-value — last-value fine for prototype).

Health engine reads proxies from the manager's cached client (no direct API reads in the hot loop).

## 8. Discovery + lease HTTP API

An HTTP server as a manager `Runnable`, listening on `:8090` (flag-configurable). JSON. Auth: single static bearer token from env `DISCOVERY_TOKEN`; empty = auth disabled (prototype). Reads go through the informer cache.

- `GET /v1/proxies?attr.geo=eu&attr.purpose=crawl&healthy=true` — filter on `spec.attributes` equality (each `attr.<key>=<value>` query param) and health. Returns id (namespace/name), ip, port, attributes, phase, healthy, latencyMillis, activeLeases, maxLeases.
- `POST /v1/leases` body `{"selector": {"geo":"eu"}, "ttlSeconds": 300}` — choose a healthy proxy matching selector with free lease capacity, **least-loaded first** (fewest active leases, tie-break lowest latency). Returns `{"leaseID": "...", "proxy": {...}, "expiresAt": "..."}`. 409 with a clear body if nothing matches.
- `DELETE /v1/leases/{id}` — early release. Idempotent.
- `POST /v1/leases/{id}/report` body `{"result": "rate_limited"|"banned"|"ok", "target": "example.com"}` — records a cooldown: proxy excluded from lease selection **for that target** for a configurable window (default 15 min, flag). Selector-matching leases may pass `"target"`; leases without a target hit the global pool as before. Cooldown state is advisory and in-memory.

**Lease store:** in-memory behind a `LeaseStore` interface (Acquire/Release/Report/ActiveCount/ExpireLoop). TTL expiry via background loop. Document the accepted prototype limitation: operator restart drops leases and cooldowns — clients must tolerate a lease vanishing (their requests still work; they just re-lease). The interface exists so a CRD- or Redis-backed store can replace it without touching handlers.

## 9. Observability

- Structured logging via the scaffold's zap setup. Log provisioning transitions at Info, provider errors at Error with provider name + providerID.
- Prometheus metrics on the standard controller-runtime metrics endpoint: `proxy_operator_proxies{phase=...}` gauge, `proxy_operator_healthcheck_duration_seconds` histogram (label: proxy), `proxy_operator_healthcheck_failures_total` counter, `proxy_operator_leases_active` gauge, `proxy_operator_lease_requests_total{outcome=granted|no_match}` counter, `proxy_operator_provider_requests_total{provider,op,result}` counter.

## 10. Tests (part of the deliverable, not optional)

- **envtest** controller suite: Managed proxy with mock provider reaches Ready; spec change triggers replacement (providerID changes); delete runs finalizer and removes the mock instance; External proxy reaches Ready once fake healthcheck passes; quota error from mock sets condition + slow requeue.
- **Unit:** provider name-derivation idempotency; error taxonomy mapping; lease store (acquire/expire/limit/least-loaded/cooldown filtering); discovery handlers with an httptest server against a fake cache; health engine threshold transitions using an httptest proxy stub.
- GCP provider: unit-test request construction only (no live cloud calls; no heavy GCP mocks — factor the API surface behind a thin interface so tests inject a fake).
- `make test` green; `go vet` clean.

## 11. README + demo

README with: 60-second architecture summary, quickstart on **kind** using the mock provider end-to-end (`kind create cluster` → `make install` → `make run` with a sample providers-config → apply sample CR → watch it go Ready → curl the list and lease endpoints — full copy-pasteable commands), the GCP setup notes (ADC, required IAM roles: `roles/compute.instanceAdmin.v1` scoped guidance), the immutable-replacement caveat, and the lease-loss-on-restart caveat.

## 12. Non-goals for the prototype (do NOT build)

- Admission webhooks, cert-manager wiring
- Multi-cluster anything
- Persistent lease storage
- Autoscaling of the fleet, proxy software installation logic beyond passing cloud-init through
- A second CRD (no ProxyProvider/ProxyPool kinds — provider config stays in the file)
- Helm chart (kustomize from the scaffold is enough)

## 13. Working style requirements

- Commit-quality code: no TODO-stubs in core paths, no panics on expected errors, contexts propagated everywhere, no `time.Sleep` in reconcile logic.
- Where you make a judgment call this spec doesn't cover, note it in `docs/architecture.md` under "Decisions".
- Before finishing: run the full test suite, run the kind quickstart yourself if the environment allows, and fix what breaks. The bar is: a competent SRE clones the repo, follows the README, and has a leased mock proxy in under 10 minutes.