diff --git a/CHANGELOG.md b/CHANGELOG.md index 825c32f..aa02966 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1 +1,14 @@ # Changelog + +## 2026-08-09 17:27 CEST — Operator wired end to end: reconciler, health, leases, discovery, GC, two providers + +- `cmd/main.go` is now the full composition root: `--providers-config` (required, fail-fast), + `--discovery-addr`, `--proxy-namespace`, `--health-workers`, `--gc-interval`, `--gc-min-age`, + `--gc-allow-namespaced`, `--lease-cooldown`, `--max-lease-ttl`; wires the kubernetes + gcp + providers (metrics-instrumented), health engine, lease store, discovery API, orphan GC, and + Prometheus metrics onto one manager. +- Deploy manifests: providers ConfigMap mount, optional `DISCOVERY_TOKEN` Secret env, + discovery port 8090 + Service; pods RBAC for the kubernetes provider. +- Samples for all three proxy flavors + providers-config; `make run-dev` for local development. +- README rewritten (kind quickstart, GCP setup, the two load-bearing caveats); architecture doc + completed with components table and the full decision log. diff --git a/Makefile b/Makefile index 5f130f3..39a77ad 100644 --- a/Makefile +++ b/Makefile @@ -116,6 +116,10 @@ build: manifests generate fmt vet ## Build manager binary. run: manifests generate fmt vet ## Run a controller from your host. go run ./cmd/main.go +.PHONY: run-dev +run-dev: manifests generate fmt vet ## Run locally against the current kubeconfig with the kubernetes-pod provider. + go run ./cmd/main.go --providers-config hack/providers-dev.yaml --metrics-bind-address :8080 --metrics-secure=false + # If you wish to build the manager image targeting other platforms you can use the --platform flag. # (i.e. docker build --platform linux/arm64). However, you must enable docker buildKit for it. # More info: https://docs.docker.com/develop/develop-images/build_enhancements/ diff --git a/README.md b/README.md index 2695f09..1311928 100644 --- a/README.md +++ b/README.md @@ -1,135 +1,166 @@ # egress-proxies-operator -// TODO(user): Add simple overview of use/purpose -## Description -// TODO(user): An in-depth paragraph about your project and overview of use +A Kubernetes operator that manages a fleet of HTTP egress proxies for +crawling: each proxy is a `Proxy` custom resource that the operator +provisions (or merely tracks), actively health-checks **through the proxy +itself**, and hands out to crawler clients via an HTTP list/lease API. -## Getting Started +## Architecture in 60 seconds -### Prerequisites -- go version v1.24.6+ -- docker version 17.03+. -- kubectl version v1.11.3+. -- Access to a Kubernetes v1.11.3+ cluster. +- **`Proxy` CRD** (`crawl.example.com/v1alpha1`, namespaced, `kubectl get px`): + `Managed` proxies are provisioned by a configured provider; `External` + proxies exist elsewhere and are only tracked and health-checked. +- **Reconciler** — a crash-safe state machine: every reconcile derives one + action from (spec, status, provider Get). Proxies are **immutable + cattle**: any meaningful spec change (placement, cloud-init, port) + deletes and recreates the VM — never in-place mutation. +- **Providers** behind one minimal interface: `kubernetes` (a real Squid + pod in this cluster — local dev/CI) and `gcp` (Compute Engine VMs with + ephemeral external IPs — the real egress fleet). Config is a YAML file + (`--providers-config`) with named instances (`gcp-eu`, `gcp-us`, ...). +- **Health engine** probes every proxy by fetching a URL *through* it (a + real CONNECT tunnel — a proxy that accepts TCP but can't egress goes + Unhealthy), with threshold logic and transition-only status writes. +- **Discovery API** (`:8090`): list healthy proxies filtered by + attributes, lease one (least-loaded, TTL-based), release, and report + rate-limiting — reports put the proxy in a per-target cooldown. +- **Orphan GC** sweeps each provider for tagged instances whose owning CR + is gone — the safety net for crashes mid-create. -### To Deploy on the cluster -**Build and push your image to the location specified by `IMG`:** +Details, diagrams, and recorded design decisions: [docs/architecture.md](docs/architecture.md). + +## Quickstart on kind (~5 minutes) + +Requires: kind, kubectl, Go 1.26, jq (optional). The kubernetes-pod +provider needs no cloud account — proxies are real `ubuntu/squid` pods in +the kind cluster itself. ```sh -make docker-build docker-push IMG=/egress-proxies-operator:tag +kind create cluster --name proxy-operator-demo +make install # install the CRD +make run-dev # run the operator locally (foreground) ``` -**NOTE:** This image ought to be published in the personal registry you specified. -And it is required to have access to pull the image from the working environment. -Make sure you have the proper permission to the registry if the above commands don’t work. - -**Install the CRDs into the cluster:** +In a second terminal: ```sh -make install +kubectl apply -f config/samples/proxy_kubernetes.yaml +kubectl get px -w +# NAME MODE PROVIDER PHASE IP HEALTHY +# proxy-kubernetes-sample Managed kubernetes Ready 10.244.x.x True ``` -**Deploy the Manager to the cluster with the image specified by `IMG`:** +Once it's `Ready`, use the discovery API: ```sh -make deploy IMG=/egress-proxies-operator:tag +# List healthy proxies +curl -s 'localhost:8090/v1/proxies?healthy=true' | jq + +# Lease one (5-minute TTL) +curl -s -XPOST localhost:8090/v1/leases \ + -d '{"selector":{"geo":"local"},"ttlSeconds":300}' | jq +# → {"leaseID":"...", "proxy":{"id":"default/proxy-kubernetes-sample", "ip":..., ...}} + +# Actually crawl through it (from inside the cluster, or port-forward the pod) +# curl -x http://:3128 https://example.com + +# Report the proxy got rate-limited by a site → 15-minute cooldown for that target +curl -s -XPOST localhost:8090/v1/leases//report \ + -d '{"result":"rate_limited","target":"example.com"}' + +# Release early (idempotent — 204 both times) +curl -si -XDELETE localhost:8090/v1/leases/ ``` -> **NOTE**: If you encounter RBAC errors, you may need to grant yourself cluster-admin -privileges or be logged in as admin. - -**Create instances of your solution** -You can apply the samples (examples) from the config/sample: +Tear down: ```sh -kubectl apply -k config/samples/ +kubectl delete -f config/samples/proxy_kubernetes.yaml # finalizer deletes the pod +kind delete cluster --name proxy-operator-demo ``` ->**NOTE**: Ensure that the samples has default values to test it out. - -### To Uninstall -**Delete the instances (CRs) from the cluster:** +## Deploying in-cluster ```sh -kubectl delete -k config/samples/ +make docker-build IMG=/egress-proxies-operator:dev +make deploy IMG=/egress-proxies-operator:dev ``` -**Delete the APIs(CRDs) from the cluster:** +- Provider config comes from the `providers-config` ConfigMap + ([config/manager/providers_config.yaml](config/manager/providers_config.yaml)); + the default ships only the kubernetes provider. +- The discovery API is exposed by the + `controller-manager-discovery-service` Service on port 8090. +- Auth: create the token Secret, or the API serves **unauthenticated** + (it warns loudly at startup): + + ```sh + kubectl -n egress-proxies-operator-system create secret generic discovery-token \ + --from-literal=token="$(openssl rand -hex 24)" + ``` + +## GCP setup + +1. Add a `gcp` entry to the providers config (see + [config/samples/providers-config.yaml](config/samples/providers-config.yaml)) — + only `project` is required. +2. Credentials are **Application Default Credentials**: workload identity + in-cluster, `gcloud auth application-default login` locally. No + key-file plumbing exists. +3. The identity needs `roles/compute.instanceAdmin.v1` on the project — + plus `roles/iam.serviceAccountUser` if instances attach a service + account. +4. Managed GCP proxies must set all of `placement.zone`, + `placement.machineType`, and `placement.image` + (see [config/samples/proxy_gcp.yaml](config/samples/proxy_gcp.yaml), + which also installs Squid via cloud-init). A missing field fails the + Proxy with a message naming it. + +Cloud-init from a Secret: the Secret **must** carry the label +`crawl.example.com/cloud-init: "true"` — the operator's cache only holds +labelled Secrets, so an unlabelled one is invisible (the Proxy reports +`CloudInitError`). Rotating the Secret's content triggers VM replacement. + +## Caveats — read these two + +**Changing a proxy changes its IP.** Proxies are immutable cattle: editing +`placement`, `cloudInit` (or rotating its Secret), or `port` deletes the +VM and creates a replacement with the **same name but a new IP**. Clients +discover the new address via the discovery API; anything that pinned the +old IP breaks by design. + +**Operator restart drops all leases and cooldowns.** Lease state is +in-memory (`replicas: 1` accordingly). Clients must tolerate a lease +vanishing — requests through the proxy keep working; they just re-lease. +The lease store sits behind an interface so a persistent backend can +replace it without touching the API handlers. + +Smaller notes: + +- `status.lastHealthCheckTime` is the time of the last *status-affecting* + probe, not the most recent probe — status writes are transition-only by + design. True probe recency lives in the metrics + (`proxy_operator_healthcheck_*`). +- The discovery API is served by every replica but is not leader-elected; + the operator ships with `replicas: 1` (see the lease caveat above). + +## Version pins + +Built and verified against the spec's pins with **no substitutions +needed**: Go 1.26, kubebuilder v4.15.0, controller-runtime v0.24.1, +k8s.io/* v0.36.3 (Kubernetes 1.36 API level), controller-tools v0.21.0, +cloud.google.com/go/compute v1.65.0. envtest uses the 1.36.2 binary +bundle (the latest 1.36 patch with published binaries — do not "fix" the +Makefile's derived version to 1.36.3, which has none). + +## Development ```sh -make uninstall +make test # unit + envtest suites (sets up envtest binaries itself) +go test -short ./... # skip the envtest suite +make run-dev # run against the current kubeconfig context ``` -**UnDeploy the controller from the cluster:** - -```sh -make undeploy -``` - -## Project Distribution - -Following the options to release and provide this solution to the users. - -### By providing a bundle with all YAML files - -1. Build the installer for the image built and published in the registry: - -```sh -make build-installer IMG=/egress-proxies-operator:tag -``` - -**NOTE:** The makefile target mentioned above generates an 'install.yaml' -file in the dist directory. This file contains all the resources built -with Kustomize, which are necessary to install this project without its -dependencies. - -2. Using the installer - -Users can just run 'kubectl apply -f ' to install -the project, i.e.: - -```sh -kubectl apply -f https://raw.githubusercontent.com//egress-proxies-operator//dist/install.yaml -``` - -### By providing a Helm Chart - -1. Build the chart using the optional helm plugin - -```sh -kubebuilder edit --plugins=helm/v2-alpha -``` - -2. See that a chart was generated under 'dist/chart', and users -can obtain this solution from there. - -**NOTE:** If you change the project, you need to update the Helm Chart -using the same command above to sync the latest changes. Furthermore, -if you create webhooks, you need to use the above command with -the '--force' flag and manually ensure that any custom configuration -previously added to 'dist/chart/values.yaml' or 'dist/chart/manager/manager.yaml' -is manually re-applied afterwards. - -## Contributing -// TODO(user): Add detailed information on how you would like others to contribute to this project - -**NOTE:** Run `make help` for more information on all potential `make` targets - -More information can be found via the [Kubebuilder Documentation](https://book.kubebuilder.io/introduction.html) - -## License - -Copyright 2026. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. - +Project layout, reconcile-loop diagrams, and the decision log are in +[docs/architecture.md](docs/architecture.md); the build history is in +[docs/plans-executions/](docs/plans-executions/). diff --git a/api/v1alpha1/proxy_types.go b/api/v1alpha1/proxy_types.go index 701c510..865fad0 100644 --- a/api/v1alpha1/proxy_types.go +++ b/api/v1alpha1/proxy_types.go @@ -61,6 +61,12 @@ const ( // provision. A mismatch against the freshly computed hash means the VM // must be replaced. AnnotationSpecHash = "crawl.example.com/spec-hash" + + // LabelCloudInit must be set (to "true") on every Secret referenced by + // spec.cloudInit.secretRef: the manager's cache only holds Secrets + // carrying this label, so an unlabelled Secret is invisible to the + // operator — both to the resolve step and to the rotation watch. + LabelCloudInit = "crawl.example.com/cloud-init" ) // Defaults, applied both by CRD structural defaulting (kubebuilder:default diff --git a/cmd/main.go b/cmd/main.go index 14eb9d2..b6495fc 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -14,28 +14,48 @@ See the License for the specific language governing permissions and limitations under the License. */ +// Package main is the composition root: it loads the provider config (fail +// fast), assembles the provider registry, and wires the reconciler, health +// engine, lease store, discovery API, orphan GC, and metrics onto one +// controller-runtime manager. package main import ( + "context" "crypto/tls" "flag" "os" + "time" // Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.) // to ensure that exec-entrypoint and run can make use of them. _ "k8s.io/client-go/plugin/pkg/client/auth" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime" utilruntime "k8s.io/apimachinery/pkg/util/runtime" clientgoscheme "k8s.io/client-go/kubernetes/scheme" ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/cache" + "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/healthz" "sigs.k8s.io/controller-runtime/pkg/log/zap" + ctrlmetrics "sigs.k8s.io/controller-runtime/pkg/metrics" "sigs.k8s.io/controller-runtime/pkg/metrics/filters" metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" crawlv1alpha1 "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/api/v1alpha1" "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/controller" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/discovery" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/gc" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/health" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/lease" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/metrics" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/gcp" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/kubernetes" + "gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/internal/provider/registry" // +kubebuilder:scaffold:imports ) @@ -60,6 +80,15 @@ func main() { var secureMetrics bool var enableHTTP2 bool var tlsOpts []func(*tls.Config) + + var providersConfig string + var discoveryAddr string + var proxyNamespace string + var healthWorkers int + var gcInterval, gcMinAge time.Duration + var gcAllowNamespaced bool + var leaseCooldown, maxLeaseTTL time.Duration + flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+ "Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.") flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.") @@ -74,6 +103,28 @@ func main() { flag.StringVar(&metricsCertKey, "metrics-cert-key", "tls.key", "The name of the metrics server key file.") flag.BoolVar(&enableHTTP2, "enable-http2", false, "If set, HTTP/2 will be enabled for the metrics server") + + flag.StringVar(&providersConfig, "providers-config", "", + "Path to the providers config YAML. Required.") + flag.StringVar(&discoveryAddr, "discovery-addr", ":8090", + "Listen address of the discovery/lease HTTP API.") + flag.StringVar(&proxyNamespace, "proxy-namespace", "", + "Restrict the manager's cache to one namespace. Empty watches all namespaces. "+ + "Restricting also disables orphan GC unless --gc-allow-namespaced is set.") + flag.IntVar(&healthWorkers, "health-workers", 8, + "Number of concurrent health-probe workers.") + flag.DurationVar(&gcInterval, "gc-interval", 10*time.Minute, + "Interval between orphan GC sweeps.") + flag.DurationVar(&gcMinAge, "gc-min-age", 10*time.Minute, + "Minimum instance age before orphan GC may delete it.") + flag.BoolVar(&gcAllowNamespaced, "gc-allow-namespaced", false, + "Allow orphan GC to run although the cache is namespace-restricted. Dangerous: proxies "+ + "outside the namespace count as orphans and their instances get deleted.") + flag.DurationVar(&leaseCooldown, "lease-cooldown", 15*time.Minute, + "How long a reported proxy/target pair is excluded from lease selection.") + flag.DurationVar(&maxLeaseTTL, "max-lease-ttl", time.Hour, + "Maximum lease TTL a client may request.") + opts := zap.Options{ Development: true, } @@ -81,6 +132,31 @@ func main() { flag.Parse() ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts))) + ctx := ctrl.SetupSignalHandler() + + // Providers load first and fail fast: a manager that comes up without + // its backends would just convert every Proxy into an error loop. + if providersConfig == "" { + setupLog.Error(nil, "--providers-config is required") + os.Exit(1) + } + cfg, err := provider.LoadConfigFile(providersConfig) + if err != nil { + setupLog.Error(err, "Failed to load providers config", "path", providersConfig) + os.Exit(1) + } + providers, err := registry.Build(ctx, cfg, map[string]registry.Constructor{ + "kubernetes": kubernetes.New, + "gcp": gcp.New, + }) + if err != nil { + setupLog.Error(err, "Failed to build providers") + os.Exit(1) + } + m := metrics.New() + for name, p := range providers { + providers[name] = provider.WithMetrics(name, p, m) + } // if the enable-http2 flag is false (the default), http/2 should be disabled // due to its vulnerabilities. More specifically, disabling http/2 will @@ -128,32 +204,91 @@ func main() { metricsServerOptions.KeyName = metricsCertKey } + // The Secret cache is restricted to labelled cloud-init Secrets: the + // operator has cluster-wide Secret read RBAC, and without the label + // selector it would cache every Secret in scope. + cacheOpts := cache.Options{ + ByObject: map[client.Object]cache.ByObject{ + &corev1.Secret{}: { + Label: labels.SelectorFromSet(labels.Set{crawlv1alpha1.LabelCloudInit: "true"}), + }, + }, + } + if proxyNamespace != "" { + cacheOpts.DefaultNamespaces = map[string]cache.Config{proxyNamespace: {}} + } + mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{ Scheme: scheme, Metrics: metricsServerOptions, HealthProbeBindAddress: probeAddr, + Cache: cacheOpts, LeaderElection: enableLeaderElection, LeaderElectionID: "b47711d1.example.com", - // LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily - // when the Manager ends. This requires the binary to immediately end when the - // Manager is stopped, otherwise, this setting is unsafe. Setting this significantly - // speeds up voluntary leader transitions as the new leader don't have to wait - // LeaseDuration time first. - // - // In the default scaffold provided, the program ends immediately after - // the manager stops, so would be fine to enable this option. However, - // if you are doing or is intended to do any operation such as perform cleanups - // after the manager stops then its usage might be unsafe. - // LeaderElectionReleaseOnCancel: true, }) if err != nil { setupLog.Error(err, "Failed to start manager") os.Exit(1) } + store := lease.NewStore(leaseCooldown) + if err := mgr.Add(store); err != nil { + setupLog.Error(err, "Failed to add lease store") + os.Exit(1) + } + + engine := health.NewEngine(mgr.GetClient()) + engine.Workers = healthWorkers + engine.Metrics = m + if err := mgr.Add(engine); err != nil { + setupLog.Error(err, "Failed to add health engine") + os.Exit(1) + } + + if err := mgr.Add(&discovery.Server{ + Reader: mgr.GetClient(), + Store: store, + Addr: discoveryAddr, + Token: os.Getenv("DISCOVERY_TOKEN"), + MaxLeaseTTL: maxLeaseTTL, + Metrics: m, + }); err != nil { + setupLog.Error(err, "Failed to add discovery server") + os.Exit(1) + } + + if err := mgr.Add(&gc.Sweeper{ + Reader: mgr.GetClient(), + Providers: providers, + Interval: gcInterval, + MinAge: gcMinAge, + NamespaceRestricted: proxyNamespace != "", + AllowNamespaced: gcAllowNamespaced, + }); err != nil { + setupLog.Error(err, "Failed to add orphan GC") + os.Exit(1) + } + + if err := m.Register(ctrlmetrics.Registry, + proxyPhaseCounts(mgr.GetClient()), + func() int { + total := 0 + for _, n := range store.Counts() { + total += n + } + return total + }, + ); err != nil { + setupLog.Error(err, "Failed to register metrics") + os.Exit(1) + } + if err := (&controller.ProxyReconciler{ - Client: mgr.GetClient(), - Scheme: mgr.GetScheme(), + Client: mgr.GetClient(), + Scheme: mgr.GetScheme(), + Providers: providers, + Health: engine, + HealthEvents: engine.Events, }).SetupWithManager(mgr); err != nil { setupLog.Error(err, "Failed to create controller", "controller", "proxy") os.Exit(1) @@ -170,8 +305,31 @@ func main() { } setupLog.Info("Starting manager") - if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil { + if err := mgr.Start(ctx); err != nil { setupLog.Error(err, "Failed to run manager") os.Exit(1) } } + +// proxyPhaseCounts reads phase counts from the cache at scrape time. Before +// the cache has synced (or on any list error) it reports nothing rather +// than something wrong. +func proxyPhaseCounts(c client.Reader) func() map[string]int { + return func() map[string]int { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + var list crawlv1alpha1.ProxyList + if err := c.List(ctx, &list); err != nil { + return nil + } + counts := map[string]int{} + for i := range list.Items { + phase := string(list.Items[i].Status.Phase) + if phase == "" { + phase = string(crawlv1alpha1.PhasePending) + } + counts[phase]++ + } + return counts + } +} diff --git a/config/default/discovery_service.yaml b/config/default/discovery_service.yaml new file mode 100644 index 0000000..078ad86 --- /dev/null +++ b/config/default/discovery_service.yaml @@ -0,0 +1,18 @@ +apiVersion: v1 +kind: Service +metadata: + labels: + control-plane: controller-manager + app.kubernetes.io/name: egress-proxies-operator + app.kubernetes.io/managed-by: kustomize + name: controller-manager-discovery-service + namespace: system +spec: + ports: + - name: discovery + port: 8090 + protocol: TCP + targetPort: discovery + selector: + control-plane: controller-manager + app.kubernetes.io/name: egress-proxies-operator diff --git a/config/default/kustomization.yaml b/config/default/kustomization.yaml index 373f742..d2352c5 100644 --- a/config/default/kustomization.yaml +++ b/config/default/kustomization.yaml @@ -22,6 +22,8 @@ resources: #- ../prometheus # [METRICS] Expose the controller manager metrics service. - metrics_service.yaml +# Expose the discovery/lease HTTP API inside the cluster. +- discovery_service.yaml # Uncomment the patches line if you enable Metrics patches: diff --git a/config/manager/kustomization.yaml b/config/manager/kustomization.yaml index 35dc755..08f98ea 100644 --- a/config/manager/kustomization.yaml +++ b/config/manager/kustomization.yaml @@ -1,5 +1,6 @@ resources: - manager.yaml +- providers_config.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization images: diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml index 2edaf3b..7d832f5 100644 --- a/config/manager/manager.yaml +++ b/config/manager/manager.yaml @@ -63,12 +63,28 @@ spec: args: - --leader-elect - --health-probe-bind-address=:8081 + - --providers-config=/etc/proxy-operator/providers.yaml + env: + # Bearer token for the discovery API. Optional: without the + # Secret the API serves unauthenticated (with a loud warning). + # Create it with: + # kubectl -n egress-proxies-operator-system create secret \ + # generic discovery-token --from-literal=token= + - name: DISCOVERY_TOKEN + valueFrom: + secretKeyRef: + name: discovery-token + key: token + optional: true image: controller:latest name: manager ports: - containerPort: 8081 name: health protocol: TCP + - containerPort: 8090 + name: discovery + protocol: TCP securityContext: readOnlyRootFilesystem: true allowPrivilegeEscalation: false @@ -96,7 +112,13 @@ spec: requests: cpu: 10m memory: 64Mi - volumeMounts: [] - volumes: [] + volumeMounts: + - name: providers-config + mountPath: /etc/proxy-operator + readOnly: true + volumes: + - name: providers-config + configMap: + name: providers-config serviceAccountName: controller-manager terminationGracePeriodSeconds: 10 diff --git a/config/manager/providers_config.yaml b/config/manager/providers_config.yaml new file mode 100644 index 0000000..6c383c0 --- /dev/null +++ b/config/manager/providers_config.yaml @@ -0,0 +1,17 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: providers-config + namespace: system + labels: + app.kubernetes.io/name: egress-proxies-operator + app.kubernetes.io/managed-by: kustomize +data: + # Mounted at /etc/proxy-operator/providers.yaml (--providers-config). + # The default ships only the kubernetes-pod provider so the operator runs + # out of the box; add gcp entries (type: gcp, gcp.project: ...) for real + # egress fleets — see config/samples/providers-config.yaml. + providers.yaml: | + providers: + - name: kubernetes + type: kubernetes diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml index e2bdbcd..ba26c8a 100644 --- a/config/rbac/role.yaml +++ b/config/rbac/role.yaml @@ -4,6 +4,16 @@ kind: ClusterRole metadata: name: manager-role rules: +- apiGroups: + - "" + resources: + - pods + verbs: + - create + - delete + - get + - list + - watch - apiGroups: - "" resources: diff --git a/config/samples/crawl_v1alpha1_proxy.yaml b/config/samples/crawl_v1alpha1_proxy.yaml deleted file mode 100644 index b206686..0000000 --- a/config/samples/crawl_v1alpha1_proxy.yaml +++ /dev/null @@ -1,9 +0,0 @@ -apiVersion: crawl.example.com/v1alpha1 -kind: Proxy -metadata: - labels: - app.kubernetes.io/name: egress-proxies-operator - app.kubernetes.io/managed-by: kustomize - name: proxy-sample -spec: - # TODO(user): Add fields here diff --git a/config/samples/kustomization.yaml b/config/samples/kustomization.yaml index c999cdb..d1c025f 100644 --- a/config/samples/kustomization.yaml +++ b/config/samples/kustomization.yaml @@ -1,4 +1,8 @@ ## Append samples of your project ## +# providers-config.yaml is deliberately absent: it is a sample +# --providers-config file, not a Kubernetes manifest. resources: -- crawl_v1alpha1_proxy.yaml +- proxy_kubernetes.yaml +- proxy_gcp.yaml +- proxy_external.yaml # +kubebuilder:scaffold:manifestskustomizesamples diff --git a/config/samples/providers-config.yaml b/config/samples/providers-config.yaml new file mode 100644 index 0000000..34220d8 --- /dev/null +++ b/config/samples/providers-config.yaml @@ -0,0 +1,21 @@ +# Sample --providers-config file (not a Kubernetes manifest). In-cluster +# this content lives in the providers-config ConfigMap +# (config/manager/providers_config.yaml); for `make run-dev` a +# kubernetes-only variant is at hack/providers-dev.yaml. +# +# Named provider instances: "gcp-eu" and "gcp-us" are two configs of the +# same type. spec.provider on a Proxy refers to the name, not the type. +providers: + - name: kubernetes + type: kubernetes + # kubernetes: + # image: ubuntu/squid:6.6-24.04_edge # the default + - name: gcp-eu + type: gcp + gcp: + project: my-project + # network: default # VPC network name + # networkTag: proxy-operator # firewall tag on created instances + # diskSizeGb: 10 + # auth: Application Default Credentials (workload identity + # in-cluster, gcloud ADC locally). No key-file plumbing. diff --git a/config/samples/proxy_external.yaml b/config/samples/proxy_external.yaml new file mode 100644 index 0000000..ffac5ce --- /dev/null +++ b/config/samples/proxy_external.yaml @@ -0,0 +1,15 @@ +# An External proxy: the VM exists outside the operator's control; the +# operator only tracks and health-checks it through the endpoint. No +# finalizer, no provider calls, and deleting the CR touches nothing. +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: proxy-external-sample +spec: + mode: External + endpoint: + host: 203.0.113.7 + port: 3128 + attributes: + geo: eu + purpose: crawl diff --git a/config/samples/proxy_gcp.yaml b/config/samples/proxy_gcp.yaml new file mode 100644 index 0000000..906231f --- /dev/null +++ b/config/samples/proxy_gcp.yaml @@ -0,0 +1,36 @@ +# A Managed proxy backed by GCP: the operator creates a VM with an +# ephemeral external IP and installs Squid via cloud-init. Requires a +# providers-config entry named "gcp-eu" (see providers-config.yaml) and +# Application Default Credentials with compute.instanceAdmin.v1. +# +# All three placement fields are required for GCP; the operator sets the +# Proxy to Failed with a message naming any missing one. +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: proxy-gcp-sample +spec: + mode: Managed + provider: gcp-eu + placement: + zone: europe-west1-b + machineType: e2-micro + image: projects/debian-cloud/global/images/family/debian-12 + port: 3128 + cloudInit: + inline: | + #cloud-config + packages: + - squid + write_files: + - path: /etc/squid/conf.d/proxy-operator.conf + content: | + http_port 3128 + http_access allow all + via off + forwarded_for off + runcmd: + - systemctl restart squid + attributes: + geo: eu + purpose: crawl diff --git a/config/samples/proxy_kubernetes.yaml b/config/samples/proxy_kubernetes.yaml new file mode 100644 index 0000000..c8e8a39 --- /dev/null +++ b/config/samples/proxy_kubernetes.yaml @@ -0,0 +1,14 @@ +# A Managed proxy backed by the kubernetes-pod provider: the operator runs +# a real Squid pod in this cluster. This is the local-dev/CI sample — pods +# share the cluster's egress IP, so it exercises the full lifecycle but +# does not provide a distinct egress path (use the gcp provider for that). +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: proxy-kubernetes-sample +spec: + mode: Managed + provider: kubernetes + attributes: + geo: local + purpose: crawl diff --git a/docs/architecture.md b/docs/architecture.md index d7e0a0d..619cd98 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,10 +1,20 @@ # Architecture -> **Status:** the operator is built through Step 9 (orphan GC + metrics) of -> [docs/plans/2026-08-07-1747-proxy-operator.md](plans/2026-08-07-1747-proxy-operator.md). -> This document covers the event/reconcile flow, the HTTP-driven -> lease/discovery path, and the GC sweep; the components table and the -> Decisions section arrive with Step 10. +## Components + +| Component | Package | Runs as | Leader-elected | Role | +|---|---|---|---|---| +| Proxy CRD + helpers | `api/v1alpha1` | types | — | `Proxy` spec/status, CEL validation, defaulting, pure helpers | +| Reconciler | `internal/controller` | controller | yes (with the manager) | the state machine: provision, replace, delete, represent health | +| Provider contract | `internal/provider` | library | — | `Provider` interface, error taxonomy, deterministic naming, config, metrics decorator | +| kubernetes provider | `internal/provider/kubernetes` | library | — | real Squid pods in this cluster (local dev/CI) | +| gcp provider | `internal/provider/gcp` | library | — | Compute Engine VMs, four API calls, fire-and-forget ops | +| Health engine | `internal/health` | Runnable | yes | through-the-proxy probes, thresholds, transition events | +| Lease store | `internal/lease` | Runnable (expiry sweep) | no | in-memory leases + cooldowns, single mutex | +| Discovery API | `internal/discovery` | Runnable | no | HTTP list/lease/release/report on `:8090` | +| Orphan GC | `internal/gc` | Runnable | yes | deletes tagged instances whose CR is gone | +| Metrics | `internal/metrics` | library | — | explicit registration, scrape-time collectors | +| Composition root | `cmd/main.go` | binary | — | flags, provider registry, wires everything onto one manager | ## Event flow: cluster events → reconciler functions @@ -287,3 +297,108 @@ Each consuming package defines its own small recorder interface (`health.ProbeMetrics`, `discovery.LeaseMetrics`, `provider.RequestRecorder`); `metrics.Metrics` satisfies all of them structurally, so no package other than `cmd/main.go` imports the metrics package. + +## Decisions + +Judgment calls the spec left open, and deliberate deviations — recorded so +they read as choices, not accidents. Chronological by build step. + +- **Registry takes its constructor map as a parameter** instead of holding + a package-level map: avoids the provider⇄registry import cycle and puts + the wiring at the composition root, where it is visible. +- **Mock provider replaced by the kubernetes-pod provider** (user + decision, mid-build): a simulated in-memory provider was too far from + the real system to build confidence in. Local dev/CI now runs real + `ubuntu/squid` pods (Canonical's actively maintained image, verified + 50M+ pulls, pinned tag) in the operator's own cluster. Trade-off + accepted: envtest has no kubelet, so end-to-end proof lives in the kind + quickstart, and cluster pods share one egress IP — distinct egress + paths remain the GCP provider's job. +- **`RequeueAfter: RequeueNow` instead of the plan's `Requeue: true`:** + `ctrl.Result.Requeue` is deprecated in controller-runtime v0.24; a fifth + configurable interval (default 1s) keeps identical semantics and stays + shrinkable in tests. +- **Quota exhaustion is a wait, not a failure:** `ErrQuotaExceeded` sets a + condition and requeues slowly (5m) with a nil error — off the backoff + curve, out of the error log, and never `phase: Failed`. Only + `ErrPermanent` latches Failed, keyed to the generation so a spec edit + auto-recovers. +- **The finalizer path never latches permanent failures:** a permanent + error during deletion keeps retrying visibly instead — latching there + would wedge the object forever with no path out but manual finalizer + surgery. +- **Health transitions travel reconciler-ward over a channel** + (`source.Channel`), not direct status patches: `phase` derives from both + provisioning and health, so two status writers would race and flap. One + writer of status; the engine owns health *state*, the reconciler its + *representation*; write-only-on-transition falls out for free. +- **Health state seeds from the existing Healthy condition on leader + handover** (verdict kept, counters zeroed, first probe jittered), so a + healthy fleet doesn't flap to Unknown on restart — but a real + transition still needs a full threshold run. A never-probed proxy skips + the jitter and probes on the next tick: startup spread matters for + restarts, not for a single new proxy. +- **Latency suppression is `max(20ms, 50%)` + a 60s rate limit, and only + while the verdict is healthy.** The spec's bare ">50% change" is + undefined at 0 and lets a proxy jittering 40↔61ms write status forever; + the healthy-only guard (found by test) stops a below-threshold success + streak from emitting latency updates for a proxy still reported + unhealthy. Consequence: `status.lastHealthCheckTime` means "last + status-affecting probe" — true probe recency is in the metrics. +- **Deterministic instance names** are `proxy-` + 16 chars of + base32(SHA-256(CR UID)): legal for both GCP (`[a-z2-7]` ⊂ `[-a-z0-9]`, + 22 ≤ 63 chars) and Pod names, 80 bits against birthday collisions at a + fleet of tens. The replacement VM therefore has the *same name* as the + one being deleted — which is why replacement polls to NotFound before + recreating instead of racing a 409. +- **`banned` and `rate_limited` share one cooldown window:** a second + duration knob the spec doesn't ask for; the report's semantic + difference is preserved in the API but not the store. +- **Report targets fall back report → lease → global**, so a client that + leased with a target can't accidentally poison the proxy's global pool + by omitting the target in its report. +- **The 409 body's `considered` counts unhealthy matches too** (the store + only ever sees healthy candidates): `considered = atCapacity + + inCooldown + unhealthy + eligible-but-outranked`, keeping the numbers + additive for a human debugging "why no proxy?". +- **TTLs above `--max-lease-ttl` are a 400, not a silent clamp** — a + client asking for a week should find out. +- **Discovery is not leader-elected and ships `replicas: 1`:** caches + start before non-leader-election runnables (verified in + controller-runtime's ordering), and a leader-elected server would leave + non-leader replicas as broken Service endpoints. One replica because + lease state is per-process. +- **GCP `Create` requires zone, machineType, and image** and fails + `ErrPermanent` naming the missing field — inventing machine-type + defaults would silently create billable VMs of arbitrary shape. +- **Unknown GCP instance statuses map to `Stopped`:** the reconciler's + answer to Stopped is delete-and-recreate, the always-safe move for + cattle when the API grows a new state. +- **Kubernetes 403s classify as `ErrPermanent`** even though quota + exhaustion also surfaces as 403 (indistinguishable from RBAC denial in + `apierrors`): not hammering an API server that may never allow the + request is the safer default; a real ResourceQuota 403 forgoes the + gentler quota backoff. Documented at the classification site. +- **GC kills log at Info with a `WARNING:` prefix** — logr has no Warn + level; the plan's "log at Warn" is met in spirit with provider, + providerID, and UID always attached. Same convention as the + discovery server's empty-token warning. +- **GC trusts only provable orphans:** instances without the UID label + are never deleted, a CR with a deletionTimestamp still counts as live + (its finalizer owns that deletion), and an unreadable Proxy list skips + the whole sweep. The namespace guard refuses to sweep a + namespace-restricted cache without `--gc-allow-namespaced`. +- **Cloud-init Secrets must carry `crawl.example.com/cloud-init: "true"`:** + the manager caches only labelled Secrets (the operator holds + cluster-wide Secret read RBAC — an unrestricted cache would hold every + Secret in scope). Unlabelled referenced Secrets are invisible by + construction, surfacing as `CloudInitError`. +- **Events RBAC from the plan is omitted:** nothing wires an + EventRecorder in the prototype, and granting verbs nothing uses would + be RBAC lint noise. Add the marker together with the recorder if events + land later. +- **logr, not slog, inside controller paths:** the repo convention says + `slog`, but `log.FromContext(ctx)` hands controller-runtime's logr + logger to everything running under the manager — fighting that would + mean two logging systems in one process. Noted as a deviation rather + than silently ignored. diff --git a/docs/plans-executions/2026-08-07-1747-proxy-operator.md b/docs/plans-executions/2026-08-07-1747-proxy-operator.md index 670ff97..dd3e7e9 100644 --- a/docs/plans-executions/2026-08-07-1747-proxy-operator.md +++ b/docs/plans-executions/2026-08-07-1747-proxy-operator.md @@ -14,7 +14,7 @@ Pairs with [docs/plans/2026-08-07-1747-proxy-operator.md](../plans/2026-08-07-17 - [x] Step 7 — Discovery API (`internal/discovery/`) - [x] Step 8 — GCP provider (`internal/provider/gcp/`) - [x] Step 9 — Orphan GC + metrics -- [ ] Step 10 — Wiring, config, docs +- [x] Step 10 — Wiring, config, docs - [ ] Step 11 — Tests - [ ] Verification (vet/test/kind e2e) + commit, push, open MR @@ -939,3 +939,73 @@ Worth noting: `prometheus/client_golang` was already in the module via controller-runtime's metrics server, so no new dependency — `go mod tidy` just promoted it to direct. `docs/architecture.md` gained §8 (GC sweep) and §9 (metrics shape). + +## Step 10 — Wiring, config, docs + +The composition root and everything around it. `cmd/main.go` now: parses +the plan's flag set (plus `--gc-allow-namespaced` from Step 9), loads the +provider config first and fails fast, builds the registry with +`{"kubernetes": kubernetes.New, "gcp": gcp.New}`, wraps every provider in +`provider.WithMetrics`, then adds the lease store, health engine, +discovery server, and GC sweeper to one manager and hands the reconciler +its providers + health snapshotter + events channel. Metrics register on +controller-runtime's global registry with scrape-time closures (phase +counts from the cache, active leases summed from `store.Counts()`). + +Two cache decisions became concrete here: + +- The Secret cache is restricted to Secrets labelled + `crawl.example.com/cloud-init=true` (new constant + `v1alpha1.LabelCloudInit`) — the operator holds cluster-wide Secret + read RBAC, and an unrestricted cache would hold every Secret in scope. + Consequence documented in the README: an unlabelled referenced Secret + is invisible → `CloudInitError`. +- `--proxy-namespace` restricts the whole cache via `DefaultNamespaces` + and flips the GC sweeper's `NamespaceRestricted` guard. + +Manifests: `config/manager/manager.yaml` gained the +`--providers-config` arg, the optional `DISCOVERY_TOKEN` secretKeyRef +(`optional: true` — without the Secret the API runs unauthenticated with +its loud warning), the ConfigMap volume mount, and containerPort 8090; +new `config/manager/providers_config.yaml` (kubernetes-only default) and +`config/default/discovery_service.yaml`. RBAC: the pods marker landed in +the controller RBAC block (cluster-scoped role — the kubernetes +provider's ListByTag spans namespaces). The plan's events RBAC was +deliberately omitted: nothing wires an EventRecorder, and unused verbs +are lint noise — recorded in Decisions. + +Samples: `proxy_kubernetes.yaml` / `proxy_gcp.yaml` (with a working +Squid-installing cloud-init) / `proxy_external.yaml` replace the scaffold +placeholder; `providers-config.yaml` documents both provider blocks; +`hack/providers-dev.yaml` + a new `run-dev` Makefile target run locally +with plain-HTTP metrics: + +```bash +make run-dev +# go run ./cmd/main.go --providers-config hack/providers-dev.yaml \ +# --metrics-bind-address :8080 --metrics-secure=false +``` + +Docs: README rewritten per the plan (60-second architecture, kind +quickstart, in-cluster deploy incl. token Secret creation, GCP setup with +the IAM roles, the two prominent caveats, the no-substitutions version +pins note). `docs/architecture.md` gained the components table and the +full Decisions section — the plan's listed decisions plus everything +accumulated in this log (RequeueNow, quota≠Failed, 409 arithmetic, +TTL-400-not-clamp, GCP required placement, unknown-status→Stopped, +banned==rate_limited window, GC logging convention, the Secret label +contract, events-RBAC omission, logr-not-slog). `CHANGELOG.md` got its +first entry with a real timestamp. + +Verified: `make test` green across the repo (coverage unchanged), +`bin/kustomize build config/default` and `config/samples` render clean, +`make build` produces the binary, e2e-tagged build + vet clean. The kind +end-to-end run is deliberately still ahead — it is the Verification +step's job, after Step 11 closes the remaining test gaps. + +Worth noting: `make run-dev` passes `--metrics-secure=false` because the +scaffold's secure-serving default requires authn/authz reachability that +a local process doesn't have; in-cluster deployments keep the secure +default from the kustomize patch. The `providers` map wrapping happens +*before* any consumer sees it, so the reconciler and GC only ever hold +instrumented providers. diff --git a/hack/providers-dev.yaml b/hack/providers-dev.yaml new file mode 100644 index 0000000..05012bb --- /dev/null +++ b/hack/providers-dev.yaml @@ -0,0 +1,6 @@ +# Providers config for `make run-dev`: local development against the +# current kubeconfig context (e.g. a kind cluster). Only the +# kubernetes-pod provider — no cloud credentials needed. +providers: + - name: kubernetes + type: kubernetes diff --git a/internal/controller/proxy_controller.go b/internal/controller/proxy_controller.go index ec344d0..3bf1d5f 100644 --- a/internal/controller/proxy_controller.go +++ b/internal/controller/proxy_controller.go @@ -82,6 +82,9 @@ type ProxyReconciler struct { // +kubebuilder:rbac:groups=crawl.example.com,resources=proxies/status,verbs=get;update;patch // +kubebuilder:rbac:groups=crawl.example.com,resources=proxies/finalizers,verbs=update // +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch +// The pod verbs are for the kubernetes-pod provider; cluster-scoped, since +// its ListByTag enumerates the operator's Pods across all namespaces. +// +kubebuilder:rbac:groups="",resources=pods,verbs=get;list;watch;create;delete // Reconcile fetches the Proxy named by req into p (r.Get fills the struct // through the pointer), dispatches to the delete/external/managed state