Wire the composition root: flags, providers, runnables, manifests, samples, docs
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
243
README.md
243
README.md
@@ -1,135 +1,166 @@
|
||||
# egress-proxies-operator
|
||||
// TODO(user): Add simple overview of use/purpose
|
||||
|
||||
## Description
|
||||
// TODO(user): An in-depth paragraph about your project and overview of use
|
||||
A Kubernetes operator that manages a fleet of HTTP egress proxies for
|
||||
crawling: each proxy is a `Proxy` custom resource that the operator
|
||||
provisions (or merely tracks), actively health-checks **through the proxy
|
||||
itself**, and hands out to crawler clients via an HTTP list/lease API.
|
||||
|
||||
## Getting Started
|
||||
## Architecture in 60 seconds
|
||||
|
||||
### Prerequisites
|
||||
- go version v1.24.6+
|
||||
- docker version 17.03+.
|
||||
- kubectl version v1.11.3+.
|
||||
- Access to a Kubernetes v1.11.3+ cluster.
|
||||
- **`Proxy` CRD** (`crawl.example.com/v1alpha1`, namespaced, `kubectl get px`):
|
||||
`Managed` proxies are provisioned by a configured provider; `External`
|
||||
proxies exist elsewhere and are only tracked and health-checked.
|
||||
- **Reconciler** — a crash-safe state machine: every reconcile derives one
|
||||
action from (spec, status, provider Get). Proxies are **immutable
|
||||
cattle**: any meaningful spec change (placement, cloud-init, port)
|
||||
deletes and recreates the VM — never in-place mutation.
|
||||
- **Providers** behind one minimal interface: `kubernetes` (a real Squid
|
||||
pod in this cluster — local dev/CI) and `gcp` (Compute Engine VMs with
|
||||
ephemeral external IPs — the real egress fleet). Config is a YAML file
|
||||
(`--providers-config`) with named instances (`gcp-eu`, `gcp-us`, ...).
|
||||
- **Health engine** probes every proxy by fetching a URL *through* it (a
|
||||
real CONNECT tunnel — a proxy that accepts TCP but can't egress goes
|
||||
Unhealthy), with threshold logic and transition-only status writes.
|
||||
- **Discovery API** (`:8090`): list healthy proxies filtered by
|
||||
attributes, lease one (least-loaded, TTL-based), release, and report
|
||||
rate-limiting — reports put the proxy in a per-target cooldown.
|
||||
- **Orphan GC** sweeps each provider for tagged instances whose owning CR
|
||||
is gone — the safety net for crashes mid-create.
|
||||
|
||||
### To Deploy on the cluster
|
||||
**Build and push your image to the location specified by `IMG`:**
|
||||
Details, diagrams, and recorded design decisions: [docs/architecture.md](docs/architecture.md).
|
||||
|
||||
## Quickstart on kind (~5 minutes)
|
||||
|
||||
Requires: kind, kubectl, Go 1.26, jq (optional). The kubernetes-pod
|
||||
provider needs no cloud account — proxies are real `ubuntu/squid` pods in
|
||||
the kind cluster itself.
|
||||
|
||||
```sh
|
||||
make docker-build docker-push IMG=<some-registry>/egress-proxies-operator:tag
|
||||
kind create cluster --name proxy-operator-demo
|
||||
make install # install the CRD
|
||||
make run-dev # run the operator locally (foreground)
|
||||
```
|
||||
|
||||
**NOTE:** This image ought to be published in the personal registry you specified.
|
||||
And it is required to have access to pull the image from the working environment.
|
||||
Make sure you have the proper permission to the registry if the above commands don’t work.
|
||||
|
||||
**Install the CRDs into the cluster:**
|
||||
In a second terminal:
|
||||
|
||||
```sh
|
||||
make install
|
||||
kubectl apply -f config/samples/proxy_kubernetes.yaml
|
||||
kubectl get px -w
|
||||
# NAME MODE PROVIDER PHASE IP HEALTHY
|
||||
# proxy-kubernetes-sample Managed kubernetes Ready 10.244.x.x True
|
||||
```
|
||||
|
||||
**Deploy the Manager to the cluster with the image specified by `IMG`:**
|
||||
Once it's `Ready`, use the discovery API:
|
||||
|
||||
```sh
|
||||
make deploy IMG=<some-registry>/egress-proxies-operator:tag
|
||||
# List healthy proxies
|
||||
curl -s 'localhost:8090/v1/proxies?healthy=true' | jq
|
||||
|
||||
# Lease one (5-minute TTL)
|
||||
curl -s -XPOST localhost:8090/v1/leases \
|
||||
-d '{"selector":{"geo":"local"},"ttlSeconds":300}' | jq
|
||||
# → {"leaseID":"...", "proxy":{"id":"default/proxy-kubernetes-sample", "ip":..., ...}}
|
||||
|
||||
# Actually crawl through it (from inside the cluster, or port-forward the pod)
|
||||
# curl -x http://<proxy-ip>:3128 https://example.com
|
||||
|
||||
# Report the proxy got rate-limited by a site → 15-minute cooldown for that target
|
||||
curl -s -XPOST localhost:8090/v1/leases/<leaseID>/report \
|
||||
-d '{"result":"rate_limited","target":"example.com"}'
|
||||
|
||||
# Release early (idempotent — 204 both times)
|
||||
curl -si -XDELETE localhost:8090/v1/leases/<leaseID>
|
||||
```
|
||||
|
||||
> **NOTE**: If you encounter RBAC errors, you may need to grant yourself cluster-admin
|
||||
privileges or be logged in as admin.
|
||||
|
||||
**Create instances of your solution**
|
||||
You can apply the samples (examples) from the config/sample:
|
||||
Tear down:
|
||||
|
||||
```sh
|
||||
kubectl apply -k config/samples/
|
||||
kubectl delete -f config/samples/proxy_kubernetes.yaml # finalizer deletes the pod
|
||||
kind delete cluster --name proxy-operator-demo
|
||||
```
|
||||
|
||||
>**NOTE**: Ensure that the samples has default values to test it out.
|
||||
|
||||
### To Uninstall
|
||||
**Delete the instances (CRs) from the cluster:**
|
||||
## Deploying in-cluster
|
||||
|
||||
```sh
|
||||
kubectl delete -k config/samples/
|
||||
make docker-build IMG=<registry>/egress-proxies-operator:dev
|
||||
make deploy IMG=<registry>/egress-proxies-operator:dev
|
||||
```
|
||||
|
||||
**Delete the APIs(CRDs) from the cluster:**
|
||||
- Provider config comes from the `providers-config` ConfigMap
|
||||
([config/manager/providers_config.yaml](config/manager/providers_config.yaml));
|
||||
the default ships only the kubernetes provider.
|
||||
- The discovery API is exposed by the
|
||||
`controller-manager-discovery-service` Service on port 8090.
|
||||
- Auth: create the token Secret, or the API serves **unauthenticated**
|
||||
(it warns loudly at startup):
|
||||
|
||||
```sh
|
||||
kubectl -n egress-proxies-operator-system create secret generic discovery-token \
|
||||
--from-literal=token="$(openssl rand -hex 24)"
|
||||
```
|
||||
|
||||
## GCP setup
|
||||
|
||||
1. Add a `gcp` entry to the providers config (see
|
||||
[config/samples/providers-config.yaml](config/samples/providers-config.yaml)) —
|
||||
only `project` is required.
|
||||
2. Credentials are **Application Default Credentials**: workload identity
|
||||
in-cluster, `gcloud auth application-default login` locally. No
|
||||
key-file plumbing exists.
|
||||
3. The identity needs `roles/compute.instanceAdmin.v1` on the project —
|
||||
plus `roles/iam.serviceAccountUser` if instances attach a service
|
||||
account.
|
||||
4. Managed GCP proxies must set all of `placement.zone`,
|
||||
`placement.machineType`, and `placement.image`
|
||||
(see [config/samples/proxy_gcp.yaml](config/samples/proxy_gcp.yaml),
|
||||
which also installs Squid via cloud-init). A missing field fails the
|
||||
Proxy with a message naming it.
|
||||
|
||||
Cloud-init from a Secret: the Secret **must** carry the label
|
||||
`crawl.example.com/cloud-init: "true"` — the operator's cache only holds
|
||||
labelled Secrets, so an unlabelled one is invisible (the Proxy reports
|
||||
`CloudInitError`). Rotating the Secret's content triggers VM replacement.
|
||||
|
||||
## Caveats — read these two
|
||||
|
||||
**Changing a proxy changes its IP.** Proxies are immutable cattle: editing
|
||||
`placement`, `cloudInit` (or rotating its Secret), or `port` deletes the
|
||||
VM and creates a replacement with the **same name but a new IP**. Clients
|
||||
discover the new address via the discovery API; anything that pinned the
|
||||
old IP breaks by design.
|
||||
|
||||
**Operator restart drops all leases and cooldowns.** Lease state is
|
||||
in-memory (`replicas: 1` accordingly). Clients must tolerate a lease
|
||||
vanishing — requests through the proxy keep working; they just re-lease.
|
||||
The lease store sits behind an interface so a persistent backend can
|
||||
replace it without touching the API handlers.
|
||||
|
||||
Smaller notes:
|
||||
|
||||
- `status.lastHealthCheckTime` is the time of the last *status-affecting*
|
||||
probe, not the most recent probe — status writes are transition-only by
|
||||
design. True probe recency lives in the metrics
|
||||
(`proxy_operator_healthcheck_*`).
|
||||
- The discovery API is served by every replica but is not leader-elected;
|
||||
the operator ships with `replicas: 1` (see the lease caveat above).
|
||||
|
||||
## Version pins
|
||||
|
||||
Built and verified against the spec's pins with **no substitutions
|
||||
needed**: Go 1.26, kubebuilder v4.15.0, controller-runtime v0.24.1,
|
||||
k8s.io/* v0.36.3 (Kubernetes 1.36 API level), controller-tools v0.21.0,
|
||||
cloud.google.com/go/compute v1.65.0. envtest uses the 1.36.2 binary
|
||||
bundle (the latest 1.36 patch with published binaries — do not "fix" the
|
||||
Makefile's derived version to 1.36.3, which has none).
|
||||
|
||||
## Development
|
||||
|
||||
```sh
|
||||
make uninstall
|
||||
make test # unit + envtest suites (sets up envtest binaries itself)
|
||||
go test -short ./... # skip the envtest suite
|
||||
make run-dev # run against the current kubeconfig context
|
||||
```
|
||||
|
||||
**UnDeploy the controller from the cluster:**
|
||||
|
||||
```sh
|
||||
make undeploy
|
||||
```
|
||||
|
||||
## Project Distribution
|
||||
|
||||
Following the options to release and provide this solution to the users.
|
||||
|
||||
### By providing a bundle with all YAML files
|
||||
|
||||
1. Build the installer for the image built and published in the registry:
|
||||
|
||||
```sh
|
||||
make build-installer IMG=<some-registry>/egress-proxies-operator:tag
|
||||
```
|
||||
|
||||
**NOTE:** The makefile target mentioned above generates an 'install.yaml'
|
||||
file in the dist directory. This file contains all the resources built
|
||||
with Kustomize, which are necessary to install this project without its
|
||||
dependencies.
|
||||
|
||||
2. Using the installer
|
||||
|
||||
Users can just run 'kubectl apply -f <URL for YAML BUNDLE>' to install
|
||||
the project, i.e.:
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/<org>/egress-proxies-operator/<tag or branch>/dist/install.yaml
|
||||
```
|
||||
|
||||
### By providing a Helm Chart
|
||||
|
||||
1. Build the chart using the optional helm plugin
|
||||
|
||||
```sh
|
||||
kubebuilder edit --plugins=helm/v2-alpha
|
||||
```
|
||||
|
||||
2. See that a chart was generated under 'dist/chart', and users
|
||||
can obtain this solution from there.
|
||||
|
||||
**NOTE:** If you change the project, you need to update the Helm Chart
|
||||
using the same command above to sync the latest changes. Furthermore,
|
||||
if you create webhooks, you need to use the above command with
|
||||
the '--force' flag and manually ensure that any custom configuration
|
||||
previously added to 'dist/chart/values.yaml' or 'dist/chart/manager/manager.yaml'
|
||||
is manually re-applied afterwards.
|
||||
|
||||
## Contributing
|
||||
// TODO(user): Add detailed information on how you would like others to contribute to this project
|
||||
|
||||
**NOTE:** Run `make help` for more information on all potential `make` targets
|
||||
|
||||
More information can be found via the [Kubebuilder Documentation](https://book.kubebuilder.io/introduction.html)
|
||||
|
||||
## License
|
||||
|
||||
Copyright 2026.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
Project layout, reconcile-loop diagrams, and the decision log are in
|
||||
[docs/architecture.md](docs/architecture.md); the build history is in
|
||||
[docs/plans-executions/](docs/plans-executions/).
|
||||
|
||||
Reference in New Issue
Block a user