egress-proxies-operator/docs/prompts/__initial-prompt.md

Build prompt: proxy-operator — Kubernetes operator for managing crawling-proxy VMs

You are building a production-quality prototype of a Kubernetes operator in Go. Read this entire spec before writing any code. Everything below is a requirement unless explicitly marked "non-goal" or "nice-to-have". Where this spec is silent, prefer the boring, idiomatic kubebuilder/controller-runtime convention over cleverness.

1. Context and purpose

A crawling department uses a fleet of small HTTP proxy VMs spread across cloud providers (GCP today, up to ~5 providers soon) to work around rate limiting. The fleet is small (tens of VMs). This operator makes those VMs first-class Kubernetes objects so they can be managed via GitOps:

• Provision/deprovision proxy VMs from a spec (image + cloud-init) via pluggable cloud providers.
• Track their state: IP, provisioning phase, health.
• Actively healthcheck each proxy through the proxy (a real HTTP request via the proxy, not a TCP dial).
• Track proxies provisioned outside the operator ("external" proxies) as equal citizens for discovery/health.
• Expose an HTTP discovery API for crawler clients: list healthy proxies filtered by labels, and lease a proxy (TTL-based assignment with server-side usage tracking).

Design principle: proxies are immutable cattle. Any meaningful spec change means replace (delete + recreate the VM), never in-place mutation. This is deliberate — do not build in-place update logic.

2. Tech stack (pin these — do not silently downgrade)

• Go 1.26 (go 1.26 in go.mod)
• kubebuilder v4 scaffold (go/v4 plugin), latest release (v4.15+)
• controller-runtime v0.24.x, k8s.io/* v0.36.x (Kubernetes 1.36 API level)
• GCP: cloud.google.com/go/compute/apiv1 (the modern Cloud Client Library, not the legacy google.golang.org/api/compute/v1 unless a needed call is missing there)
• CRD validation via CEL validation rules and kubebuilder markers. No admission webhooks in the prototype.
• Tests: standard testing + envtest for the controller. No test framework dependencies beyond what kubebuilder scaffolds (ginkgo is acceptable since the scaffold generates it, but table-driven std tests are preferred for units).

If any pinned version is unavailable in your environment, use the closest available and record the substitution prominently in the README.

3. Repository layout

Standard kubebuilder go/v4 layout. Module path: github.com/CHANGEME/proxy-operator (make it trivially renameable — no hardcoded module strings outside go.mod and imports).

api/v1alpha1/           # Proxy types
internal/controller/    # Proxy reconciler
internal/provider/      # provider interface + registry
internal/provider/mock/ # in-memory provider
internal/provider/gcp/  # GCP provider
internal/health/        # healthcheck engine
internal/discovery/     # HTTP API (list + lease)
internal/lease/         # lease store (in-memory, interface-first)
config/                 # CRDs, RBAC, manager kustomize (scaffold-generated, kept working)
config/samples/         # sample Proxy CRs: mock, gcp, external
docs/architecture.md    # short: components, data flow, one ASCII diagram

4. The Proxy CRD

Group crawl.example.com, version v1alpha1, kind Proxy, cluster-scoped: no — make it namespaced (fleet lives in one namespace; namespacing keeps RBAC simple).

Spec

type ProxySpec struct {
    // Provisioning mode: "Managed" (operator creates the VM) or "External"
    // (VM exists elsewhere; operator only tracks/healthchecks it).
    // Immutable after creation (enforce with CEL).
    Mode ProvisioningMode `json:"mode"`

    // Provider name matching a configured provider ("mock", "gcp-eu", ...).
    // Required iff mode==Managed. Immutable (CEL).
    Provider string `json:"provider,omitempty"`

    // Provider-opaque placement/size settings. Keep this a small typed struct,
    // NOT map[string]string: Region, Zone, MachineType, Image. Providers may
    // ignore fields that don't apply to them.
    Placement *PlacementSpec `json:"placement,omitempty"`

    // Cloud-init user-data. Either inline or from a Secret key. Exactly one
    // (CEL). Changing it on a Managed proxy triggers replacement.
    CloudInit *CloudInitSpec `json:"cloudInit,omitempty"`

    // Endpoint config for External mode: host, port. Required iff External (CEL).
    Endpoint *EndpointSpec `json:"endpoint,omitempty"`

    // Proxy port for Managed mode (the port the proxy listens on once the VM
    // is up). Default 3128.
    Port int32 `json:"port,omitempty"`

    // Selection attributes exposed to the discovery API (geo, asn, purpose...).
    // Deliberately separate from k8s object labels, which stay an operator
    // implementation concern.
    Attributes map[string]string `json:"attributes,omitempty"`

    HealthCheck *HealthCheckSpec `json:"healthCheck,omitempty"` // probeURL, interval, timeout, failureThreshold, successThreshold — all defaulted
    // Max concurrent leases handed out for this proxy. Default 5. 0 = unleasable (list-only visibility).
    MaxLeases int32 `json:"maxLeases,omitempty"`
}

Status

type ProxyStatus struct {
    Phase       ProxyPhase         `json:"phase,omitempty"` // Pending, Provisioning, Ready, Unhealthy, Deleting, Failed
    ProviderID  string             `json:"providerID,omitempty"` // opaque cloud resource ID
    IP          string             `json:"ip,omitempty"`
    Conditions  []metav1.Condition `json:"conditions,omitempty"` // Provisioned, Healthy — standard metav1 conditions with reasons
    LastHealthCheckTime *metav1.Time `json:"lastHealthCheckTime,omitempty"`
    LatencyMillis       int64        `json:"latencyMillis,omitempty"`
    ObservedGeneration  int64        `json:"observedGeneration,omitempty"`
}

Printer columns: Mode, Provider, Phase, IP, Healthy, Age.

Status-write discipline: healthchecks run frequently; do not write status on every probe. Write only on transition (healthy↔unhealthy, latency bucket change of >50%, phase change). This matters — unbounded status churn is a known operator anti-pattern.

5. Provider interface

Keep it brutally minimal. This is the contract five future providers must implement, so resist enrichment:

type Provider interface {
    // Create starts VM creation. MUST be idempotent: name derives
    // deterministically from the CR ("proxy-" + short hash of CR UID), so a
    // repeat call after a crash finds the existing VM instead of duplicating.
    // May return before the VM is running.
    Create(ctx context.Context, req CreateRequest) (providerID string, err error)

    // Get returns current state. Returning (nil, ErrNotFound) is normal.
    Get(ctx context.Context, providerID string) (*Instance, error) // Instance: ID, IP, State (Provisioning|Running|Stopped|Terminated)

    // Delete is idempotent; deleting a non-existent instance is not an error.
    Delete(ctx context.Context, providerID string) error

    // ListByTag returns all instances this operator ever tagged, for orphan GC.
    ListByTag(ctx context.Context) ([]Instance, error)
}

Error contract: providers wrap errors into a small taxonomy the reconciler can branch on — ErrNotFound, ErrQuotaExceeded (retry slow: requeue ≥5 min), ErrTransient (retry with backoff), ErrPermanent (set Failed phase, stop retrying, surface in condition reason). Use errors.Is-compatible sentinel/wrapper types.

Every cloud resource a provider creates MUST carry a tag/label proxy-operator-uid=<CR UID> plus proxy-operator-managed=true. This is the GC contract.

Provider configuration

Static YAML config file mounted into the manager pod, path from --providers-config. Named provider instances (so "gcp-eu" and "gcp-us" can be two configs of type gcp):

providers:
  - name: mock
    type: mock
  - name: gcp-eu
    type: gcp
    gcp:
      project: my-project
      # auth: Application Default Credentials (workload identity in-cluster,
      #       gcloud ADC locally). No key-file plumbing in the prototype.

Parse at startup, fail fast on unknown type. A registry maps type → constructor.

Mock provider

In-memory, thread-safe. Simulates async provisioning: instance is Provisioning for a configurable duration (default 5s) then Running with a fake IP from a private range. Supports fault injection via config (fail next N creates, inject quota error) — needed for controller tests.

GCP provider

cloud.google.com/go/compute/apiv1. Insert instance with: machine type, zone, image, network tag, labels (the GC tags), cloud-init via user-data metadata key, ephemeral external IP. Wait for the insert Operation without blocking reconcile: Create returns after the operation is submitted; reconciler discovers readiness via Get polling (requeue). Delete likewise fire-and-forget + poll. Keep the implementation to the minimal calls: instances.Insert, instances.Get, instances.Delete, instances.AggregatedList (filtered by label) — nothing else.

6. Reconciler semantics

• Finalizer crawl.example.com/proxy-cleanup on Managed proxies; on delete, call provider Delete, requeue until Get returns NotFound, then remove finalizer. External proxies get no finalizer.
• State machine, not step list. Every reconcile derives desired action from (spec, status, provider Get). No multi-step sequences relying on in-memory state — the controller must resume correctly from any crash point.
• Replacement on change: compute a spec-hash (placement + cloudInit + image + port) and store it in an annotation. Hash mismatch on a Ready proxy → delete VM, clear providerID, re-provision. Document this loudly in the README (changing a proxy = its IP changes).
• External mode: skip provisioning entirely; phase goes Pending → Ready(when first healthcheck passes)/Unhealthy. IP comes from spec.endpoint.
• Orphan GC: a manager Runnable (not part of reconcile) sweeps every 10 min: for each provider, ListByTag, kill instances whose proxy-operator-uid matches no live CR (check deletionTimestamp semantics carefully; skip instances younger than 10 min to avoid racing in-flight creates). Log every kill at Warn with provider ID and UID.
• Standard controller hygiene: exponential backoff via requeue-with-error, rate-limited workqueue defaults, MaxConcurrentReconciles: 3.

7. Health engine

A manager Runnable running one probe loop (single goroutine with a timer wheel or per-proxy tickers — your call, but bounded goroutines). For each proxy with an IP:

• Build an http.Client with Transport.Proxy pointing at http://<ip>:<port>, per-probe timeout from spec.
• GET the probe URL (default https://www.gstatic.com/generate_204, overridable per-proxy). Success = expected status code (default 204/200) within timeout. This exercises a real CONNECT through the proxy — the whole point; a proxy that TCP-accepts but can't egress must go Unhealthy.
• Threshold logic (failureThreshold consecutive fails → Unhealthy; successThreshold → Healthy) lives in the engine; it pushes transitions to the reconciler via a channel or does a direct conditional status patch — pick one and document why.
• Record latency (EWMA or last-value — last-value fine for prototype).

Health engine reads proxies from the manager's cached client (no direct API reads in the hot loop).

8. Discovery + lease HTTP API

An HTTP server as a manager Runnable, listening on :8090 (flag-configurable). JSON. Auth: single static bearer token from env DISCOVERY_TOKEN; empty = auth disabled (prototype). Reads go through the informer cache.

• GET /v1/proxies?attr.geo=eu&attr.purpose=crawl&healthy=true — filter on spec.attributes equality (each attr.<key>=<value> query param) and health. Returns id (namespace/name), ip, port, attributes, phase, healthy, latencyMillis, activeLeases, maxLeases.
• POST /v1/leases body {"selector": {"geo":"eu"}, "ttlSeconds": 300} — choose a healthy proxy matching selector with free lease capacity, least-loaded first (fewest active leases, tie-break lowest latency). Returns {"leaseID": "...", "proxy": {...}, "expiresAt": "..."}. 409 with a clear body if nothing matches.
• DELETE /v1/leases/{id} — early release. Idempotent.
• POST /v1/leases/{id}/report body {"result": "rate_limited"|"banned"|"ok", "target": "example.com"} — records a cooldown: proxy excluded from lease selection for that target for a configurable window (default 15 min, flag). Selector-matching leases may pass "target"; leases without a target hit the global pool as before. Cooldown state is advisory and in-memory.

Lease store: in-memory behind a LeaseStore interface (Acquire/Release/Report/ActiveCount/ExpireLoop). TTL expiry via background loop. Document the accepted prototype limitation: operator restart drops leases and cooldowns — clients must tolerate a lease vanishing (their requests still work; they just re-lease). The interface exists so a CRD- or Redis-backed store can replace it without touching handlers.

9. Observability

• Structured logging via the scaffold's zap setup. Log provisioning transitions at Info, provider errors at Error with provider name + providerID.
• Prometheus metrics on the standard controller-runtime metrics endpoint: proxy_operator_proxies{phase=...} gauge, proxy_operator_healthcheck_duration_seconds histogram (label: proxy), proxy_operator_healthcheck_failures_total counter, proxy_operator_leases_active gauge, proxy_operator_lease_requests_total{outcome=granted|no_match} counter, proxy_operator_provider_requests_total{provider,op,result} counter.

10. Tests (part of the deliverable, not optional)

• envtest controller suite: Managed proxy with mock provider reaches Ready; spec change triggers replacement (providerID changes); delete runs finalizer and removes the mock instance; External proxy reaches Ready once fake healthcheck passes; quota error from mock sets condition + slow requeue.
• Unit: provider name-derivation idempotency; error taxonomy mapping; lease store (acquire/expire/limit/least-loaded/cooldown filtering); discovery handlers with an httptest server against a fake cache; health engine threshold transitions using an httptest proxy stub.
• GCP provider: unit-test request construction only (no live cloud calls; no heavy GCP mocks — factor the API surface behind a thin interface so tests inject a fake).
• make test green; go vet clean.

11. README + demo

README with: 60-second architecture summary, quickstart on kind using the mock provider end-to-end (kind create cluster → make install → make run with a sample providers-config → apply sample CR → watch it go Ready → curl the list and lease endpoints — full copy-pasteable commands), the GCP setup notes (ADC, required IAM roles: roles/compute.instanceAdmin.v1 scoped guidance), the immutable-replacement caveat, and the lease-loss-on-restart caveat.

12. Non-goals for the prototype (do NOT build)

• Admission webhooks, cert-manager wiring
• Multi-cluster anything
• Persistent lease storage
• Autoscaling of the fleet, proxy software installation logic beyond passing cloud-init through
• A second CRD (no ProxyProvider/ProxyPool kinds — provider config stays in the file)
• Helm chart (kustomize from the scaffold is enough)

13. Working style requirements

• Commit-quality code: no TODO-stubs in core paths, no panics on expected errors, contexts propagated everywhere, no time.Sleep in reconcile logic.
• Where you make a judgment call this spec doesn't cover, note it in docs/architecture.md under "Decisions".
• Before finishing: run the full test suite, run the kind quickstart yourself if the environment allows, and fix what breaks. The bar is: a competent SRE clones the repo, follows the README, and has a leased mock proxy in under 10 minutes.