From 849ec1083ecc111f2f4b708b628895657af10ff7 Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 11 Aug 2026 19:37:42 +0200 Subject: [PATCH 01/10] Add plan: distilled Gitea Actions image-build workflow Co-Authored-By: Claude --- .../2026-08-11-1935-gitea-build-workflow.md | 135 ++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100644 docs/plans/2026-08-11-1935-gitea-build-workflow.md diff --git a/docs/plans/2026-08-11-1935-gitea-build-workflow.md b/docs/plans/2026-08-11-1935-gitea-build-workflow.md new file mode 100644 index 0000000..199c27a --- /dev/null +++ b/docs/plans/2026-08-11-1935-gitea-build-workflow.md @@ -0,0 +1,135 @@ +# Plan: Distilled Gitea Actions image-build workflow +**Created:** 2026-08-11 19:35 + +## Context + +This repo (`egress-proxies-operator`) has a Dockerfile, a Makefile with `docker-build`/`docker-push` targets, and a Gitea remote — but no CI workflow (CLAUDE.md flags this as a TODO). A survey of all projects under `/Users/jan.novak/srv` found 9 image-build workflows, all variations of one lineage: trigger on `workflow_dispatch` + tag push, `docker login` to `gitea.home.hrajfrisbee.cz` with `secrets.REGISTRY_TOKEN`, raw `docker build`/`docker push`, `runs-on: ubuntu-latest`, `permissions: {contents: read, packages: write}`. + +The best individual ideas are scattered: +- **aviso_v2**: quality-gate job before build; computes `sha-` as a second immutable tag via `$GITHUB_OUTPUT`. +- **gateway-helper-operator** (closest sibling — same kubebuilder shape): passes build args (`GIT_COMMIT` etc.), tags `:latest` alongside the version tag. +- **psmf-data-sync test.yaml**: `actions/setup-go@v5` with `go-version-file: go.mod` + module cache (proven to work on the act_runner). + +Goal: distill these into one `build.yaml` for this repo. User decisions: triggers = **tags + manual dispatch only** (house convention, no builds from main); build tool = **raw docker CLI** (the runner bind-mounts docker.sock, so this just works); **lightweight test gate** (`go vet` + `go build` + `go test -short`, no envtest download); **amd64 only**. + +## The workflow + +Create `.gitea/workflows/build.yaml`: + +```yaml +name: Build and Push + +on: + workflow_dispatch: + inputs: + tag: + description: 'Image tag' + required: true + default: 'latest' + push: + tags: + - '*' + +concurrency: + group: build-${{ github.ref }} + cancel-in-progress: true + +jobs: + check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version-file: go.mod + cache: true + - name: Vet + run: go vet ./... + - name: Build + run: go build ./... + - name: Test (short) + run: go test -short ./... + + build: + needs: check + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + steps: + - uses: actions/checkout@v4 + + - name: Compute image tags + id: meta + run: | + if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then + TAG="${{ inputs.tag }}" + else + TAG="${{ github.ref_name }}" + fi + echo "tag=$TAG" >> "$GITHUB_OUTPUT" + echo "sha=sha-$(echo '${{ github.sha }}' | cut -c1-12)" >> "$GITHUB_OUTPUT" + + - name: Login to Gitea registry + run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login -u ${{ github.actor }} --password-stdin gitea.home.hrajfrisbee.cz + + - name: Build and push + run: | + IMAGE=gitea.home.hrajfrisbee.cz/${{ github.repository }} + docker build \ + --build-arg GIT_COMMIT=$(echo '${{ github.sha }}' | cut -c1-12) \ + --label org.opencontainers.image.source=https://gitea.home.hrajfrisbee.cz/${{ github.repository }} \ + --label org.opencontainers.image.created=$(date -u +%Y-%m-%dT%H:%M:%SZ) \ + -t "$IMAGE:${{ steps.meta.outputs.tag }}" \ + -t "$IMAGE:${{ steps.meta.outputs.sha }}" \ + . + docker push "$IMAGE:${{ steps.meta.outputs.tag }}" + docker push "$IMAGE:${{ steps.meta.outputs.sha }}" + + - name: Push latest (tag builds only) + if: github.event_name == 'push' + run: | + IMAGE=gitea.home.hrajfrisbee.cz/${{ github.repository }} + docker tag "$IMAGE:${{ steps.meta.outputs.tag }}" "$IMAGE:latest" + docker push "$IMAGE:latest" +``` + +### What's distilled vs. improved over the existing workflows + +Distilled (house patterns kept as-is): triggers, `REGISTRY_TOKEN` + `github.actor` login, image name `gitea.home.hrajfrisbee.cz/${{ github.repository }}`, `ubuntu-latest`, raw docker CLI, `permissions` block. + +Improvements none of the existing workflows have all of: +1. **`sha-<12>` immutable tag** alongside the human tag (aviso_v2 had this; nobody else) — lets deployments pin exactly what was built. +2. **Test gate** (aviso_v2 had one; the Go projects don't) — lightweight variant per user choice; uses `go-version-file: go.mod` so the Go version never drifts from the module. +3. **`GIT_COMMIT` build arg** matches the Makefile/Dockerfile contract — the binary's `internal/version.Commit` and the `org.opencontainers.image.revision` label get the real commit (12-char, same width as the Makefile's `git rev-parse --short=12`; no `-dirty` needed since CI checkouts are clean). +4. **`:latest` only on real tag pushes**, not manual dispatch — gateway-helper pushed `latest` unconditionally, which lets an ad-hoc dispatch of an old ref clobber `latest`. +5. **`concurrency` group** — cancels a superseded run of the same ref (none of the 20 surveyed workflows have this). +6. **OCI `source`/`created` labels** added at build time (revision label already comes from the Dockerfile). + +## Files + +- **Create** `.gitea/workflows/build.yaml` — content above. +- **Update** `CLAUDE.md` — replace the `TODO: no .gitea/workflows/ CI pipeline exists yet` note in the Git Commits section with a short CI/CD subsection describing the workflow (triggers, secret, tags produced). +- **Update** `CHANGELOG.md` — new top entry (after user confirms it works, per convention; timestamp via `date "+%Y-%m-%d %H:%M %Z"`). +- Copy this plan to `docs/plans/YYYY-MM-DD-HHMM-gitea-build-workflow.md` (timestamp via `date "+%Y-%m-%d-%H%M"`) and commit it first, per CLAUDE.md ordering rule. + +## Branch & MR + +House convention: feature → own branch + MR. Dockerfile and `cmd/` already exist on `main`, so: + +1. `git checkout -b feat/gitea-build-workflow origin/main` (do not touch the current `feat/proxy-operator` branch's uncommitted `.claude/settings.json` change — leave it be). +2. Commit plan file, then the workflow + CLAUDE.md update (with `Co-Authored-By: Claude `). +3. `git push -u origin feat/gitea-build-workflow`, open MR with `tea pr create --base main --head feat/gitea-build-workflow`. Do not merge. + +Note: `main` has no `internal/`/`test/` dirs yet (those are on `feat/proxy-operator`), which is fine — the workflow only fires on tags/dispatch, and by then the operator branch will be merged. `go build ./...` / `go test -short ./...` work on both branch states. + +## Prerequisite (user action) + +`REGISTRY_TOKEN` secret must exist in this repo's Gitea settings (Settings → Actions → Secrets): a personal access token with `write:package` scope — same as every other project uses. Flag this in the MR description. + +## Verification + +The workflow doesn't trigger on branch pushes, so end-to-end verification happens after merge: +1. Local sanity: `docker build --build-arg GIT_COMMIT=test -t scratch-check .` (confirms the build args/labels line is valid) — or at minimum a YAML parse check. +2. After the MR merges: run the workflow manually via Gitea UI (Actions → Build and Push → Run workflow, tag `manual-test`), confirm both `manual-test` and `sha-…` tags appear under Packages, and that `:latest` was NOT updated. +3. Then push a real version tag (e.g. `v0.1.0`) and confirm `v0.1.0`, `sha-…`, and `latest` all appear. From 95c487415b2bd6a6acfe0351d580ba5587cb6bdb Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 11 Aug 2026 19:39:23 +0200 Subject: [PATCH 02/10] Add Gitea Actions image-build workflow Distilled from the house pattern across sibling projects: tag push + workflow_dispatch triggers, REGISTRY_TOKEN login, raw docker build/push to gitea.home.hrajfrisbee.cz. Adds a lightweight test gate, an immutable sha-<12> tag, :latest only on real tag pushes, and a concurrency group. Co-Authored-By: Claude --- .gitea/workflows/build.yaml | 76 +++++++++++++++++++ CLAUDE.md | 14 +++- .../2026-08-11-1935-gitea-build-workflow.md | 42 ++++++++++ 3 files changed, 130 insertions(+), 2 deletions(-) create mode 100644 .gitea/workflows/build.yaml create mode 100644 docs/plans-executions/2026-08-11-1935-gitea-build-workflow.md diff --git a/.gitea/workflows/build.yaml b/.gitea/workflows/build.yaml new file mode 100644 index 0000000..66fccac --- /dev/null +++ b/.gitea/workflows/build.yaml @@ -0,0 +1,76 @@ +name: Build and Push + +on: + workflow_dispatch: + inputs: + tag: + description: 'Image tag' + required: true + default: 'latest' + push: + tags: + - '*' + +concurrency: + group: build-${{ github.ref }} + cancel-in-progress: true + +jobs: + check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version-file: go.mod + cache: true + - name: Vet + run: go vet ./... + - name: Build + run: go build ./... + - name: Test (short) + run: go test -short ./... + + build: + needs: check + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + steps: + - uses: actions/checkout@v4 + + - name: Compute image tags + id: meta + run: | + if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then + TAG="${{ inputs.tag }}" + else + TAG="${{ github.ref_name }}" + fi + echo "tag=$TAG" >> "$GITHUB_OUTPUT" + echo "sha=sha-$(echo '${{ github.sha }}' | cut -c1-12)" >> "$GITHUB_OUTPUT" + + - name: Login to Gitea registry + run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login -u ${{ github.actor }} --password-stdin gitea.home.hrajfrisbee.cz + + - name: Build and push + run: | + IMAGE=gitea.home.hrajfrisbee.cz/${{ github.repository }} + docker build \ + --build-arg GIT_COMMIT=$(echo '${{ github.sha }}' | cut -c1-12) \ + --label org.opencontainers.image.source=https://gitea.home.hrajfrisbee.cz/${{ github.repository }} \ + --label org.opencontainers.image.created=$(date -u +%Y-%m-%dT%H:%M:%SZ) \ + -t "$IMAGE:${{ steps.meta.outputs.tag }}" \ + -t "$IMAGE:${{ steps.meta.outputs.sha }}" \ + . + docker push "$IMAGE:${{ steps.meta.outputs.tag }}" + docker push "$IMAGE:${{ steps.meta.outputs.sha }}" + + # Only real tag pushes move :latest — an ad-hoc dispatch of an old ref must not clobber it. + - name: Push latest (tag builds only) + if: github.event_name == 'push' + run: | + IMAGE=gitea.home.hrajfrisbee.cz/${{ github.repository }} + docker tag "$IMAGE:${{ steps.meta.outputs.tag }}" "$IMAGE:latest" + docker push "$IMAGE:latest" diff --git a/CLAUDE.md b/CLAUDE.md index 4be0be9..57e257e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -199,8 +199,18 @@ Always append a `Co-Authored-By` trailer to indicate AI assistance: Co-Authored-By: Claude -TODO: no `.gitea/workflows/` CI pipeline exists yet — add a CI/CD subsection here once -one is set up. +### CI/CD + +`.gitea/workflows/build.yaml` builds the manager image and pushes it to the Gitea +registry. Triggers: any tag push, or manual `workflow_dispatch` with a `tag` input. +A lightweight `check` job (`go vet` / `go build` / `go test -short`) gates the build. + +- Images: `gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator:` plus an + immutable `sha-<12-char-commit>` tag on every build; `:latest` moves only on real + tag pushes, never on manual dispatch. +- Requires the `REGISTRY_TOKEN` repo secret (Gitea PAT with `write:package`), + same convention as the other projects on this Gitea instance. +- The commit is baked into the binary via the `GIT_COMMIT` build arg (see Dockerfile). ## Gotchas diff --git a/docs/plans-executions/2026-08-11-1935-gitea-build-workflow.md b/docs/plans-executions/2026-08-11-1935-gitea-build-workflow.md new file mode 100644 index 0000000..17b0c89 --- /dev/null +++ b/docs/plans-executions/2026-08-11-1935-gitea-build-workflow.md @@ -0,0 +1,42 @@ +# Execution: Distilled Gitea Actions image-build workflow + +Plan: `docs/plans/2026-08-11-1935-gitea-build-workflow.md` + +- [x] Step 1 — Create `.gitea/workflows/build.yaml` +- [x] Step 2 — Replace CLAUDE.md CI TODO with a CI/CD subsection +- [ ] Step 3 — Push branch + open MR +- [ ] Step 4 — CHANGELOG entry (after the first successful run is confirmed) + +## Steps 1–2 — workflow + CLAUDE.md + +The workflow distills the house pattern from 9 sibling projects (survey in the plan) +plus improvements none of them combine: an immutable `sha-<12>` tag, a lightweight +test gate, `:latest` moving only on real tag pushes, and a `concurrency` group. + +Work happened in a git worktree off `origin/main` so the main checkout (which had +unrelated uncommitted changes) stayed untouched: + +```bash +git worktree add -b feat/gitea-build-workflow \ + "$SCRATCH/wt-build" origin/main +``` + +Deviation from the plan's assumptions: while planning, `feat/proxy-operator` was +still unmerged and the plan noted `main` lacked `internal/`/`test/`. By execution +time `origin/main` had moved (`076bc66..f7000f7` — the proxy-operator MR merged), +so the branch and its CI gate cover the full operator code. + +Verified the workflow parses and the check-gate commands pass on this exact tree +(ruby stands in for a YAML linter because the system python3 has no `yaml` module): + +```bash +ruby -ryaml -e "YAML.load_file('.gitea/workflows/build.yaml'); puts 'YAML OK'" +go vet ./... && go build ./... && go test -short ./... # all packages ok +``` + +Worth noting: the workflow itself cannot run end-to-end until (a) the MR merges +(it only triggers on tags / manual dispatch, not branch pushes) and (b) the +`REGISTRY_TOKEN` secret is created in this repo's Gitea settings (PAT with +`write:package`). First real verification = manual dispatch with tag +`manual-test`, expecting `manual-test` + `sha-…` in Packages and `:latest` +untouched. From 57e3ea22cf6df1efa6fc0088f2276aff9753c1f3 Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 11 Aug 2026 19:41:57 +0200 Subject: [PATCH 03/10] Record MR creation in plan execution summary Co-Authored-By: Claude --- .../2026-08-11-1935-gitea-build-workflow.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/plans-executions/2026-08-11-1935-gitea-build-workflow.md b/docs/plans-executions/2026-08-11-1935-gitea-build-workflow.md index 17b0c89..b283f33 100644 --- a/docs/plans-executions/2026-08-11-1935-gitea-build-workflow.md +++ b/docs/plans-executions/2026-08-11-1935-gitea-build-workflow.md @@ -4,7 +4,7 @@ Plan: `docs/plans/2026-08-11-1935-gitea-build-workflow.md` - [x] Step 1 — Create `.gitea/workflows/build.yaml` - [x] Step 2 — Replace CLAUDE.md CI TODO with a CI/CD subsection -- [ ] Step 3 — Push branch + open MR +- [x] Step 3 — Push branch + open MR - [ ] Step 4 — CHANGELOG entry (after the first successful run is confirmed) ## Steps 1–2 — workflow + CLAUDE.md @@ -34,6 +34,17 @@ ruby -ryaml -e "YAML.load_file('.gitea/workflows/build.yaml'); puts 'YAML OK'" go vet ./... && go build ./... && go test -short ./... # all packages ok ``` +## Step 3 — push + MR + +Branch pushed and MR opened with `tea` (the worktree was then removed and the main +checkout switched onto the branch so the files are visible locally): + +```bash +tea pr create --title "Add Gitea Actions image-build workflow" \ + --description "..." --base main --head feat/gitea-build-workflow +# → https://gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator/pulls/2 +``` + Worth noting: the workflow itself cannot run end-to-end until (a) the MR merges (it only triggers on tags / manual dispatch, not branch pushes) and (b) the `REGISTRY_TOKEN` secret is created in this repo's Gitea settings (PAT with From 9230b1213cefdb69673504950d302fc1d031f80a Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 11 Aug 2026 19:46:55 +0200 Subject: [PATCH 04/10] Document Gitea CI and required secrets in README Co-Authored-By: Claude --- README.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/README.md b/README.md index 4ac1071..b00e9fd 100644 --- a/README.md +++ b/README.md @@ -167,6 +167,40 @@ cloud.google.com/go/compute v1.65.0. envtest uses the 1.36.2 binary bundle (the latest 1.36 patch with published binaries — do not "fix" the Makefile's derived version to 1.36.3, which has none). +## Gitea CI + +[.gitea/workflows/build.yaml](.gitea/workflows/build.yaml) builds the +manager image and pushes it to this Gitea instance's container registry. +It runs on **any tag push** or manually via **Run workflow** (with a `tag` +input) — never on branch pushes. A lightweight `check` job (`go vet`, +`go build`, `go test -short`) gates the build. + +Every build pushes two tags to +`gitea.home.hrajfrisbee.cz/kacerr/egress-proxies-operator`: + +- the human tag (the git tag, or the dispatch input), and +- an immutable `sha-<12-char-commit>` tag — pin deployments to this one. + +`:latest` is additionally updated on real tag pushes only, so a manual +dispatch of an old ref can never clobber it. The commit is baked into the +binary (`internal/version.Commit`) via the `GIT_COMMIT` build arg. + +### Mandatory Gitea secrets + +Set under **Settings → Actions → Secrets** in this repo: + +| Secret | Required by | What it is | +| ---------------- | ----------------------------- | ---------------------------------------- | +| `REGISTRY_TOKEN` | `build.yaml` (registry login) | Gitea PAT with the `write:package` scope | + +The token is paired with `${{ github.actor }}` as the username, so it +must belong to the user triggering the workflow — same convention as the +other projects on this instance. + +Without `REGISTRY_TOKEN` the `check` job still passes but the build job +fails at the `docker login` step. No other secrets are needed — the +workflow does not deploy anywhere. + ## Development ```sh From e7fdae0859348c9e566235abb4f1d227b343bbee Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 11 Aug 2026 21:53:02 +0200 Subject: [PATCH 05/10] Add plan: discovery API documentation Co-Authored-By: Claude --- .../2026-08-11-2152-discovery-api-docs.md | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 docs/plans/2026-08-11-2152-discovery-api-docs.md diff --git a/docs/plans/2026-08-11-2152-discovery-api-docs.md b/docs/plans/2026-08-11-2152-discovery-api-docs.md new file mode 100644 index 0000000..d19197f --- /dev/null +++ b/docs/plans/2026-08-11-2152-discovery-api-docs.md @@ -0,0 +1,58 @@ +# Plan: Discovery API documentation (docs/api.md) +**Created:** 2026-08-11 21:52 + + +## Context + +The operator serves an HTTP discovery/lease API on `:8090` ([internal/discovery/](internal/discovery/)) that crawler clients use to list proxies, acquire TTL leases, release them, and report rate-limiting. There is no dedicated API reference today: README has four quickstart curls (no auth header, no schemas), and `docs/architecture.md` §7 has an ASCII route map. The user wants full documentation with curl examples for every feature. + +**Decisions made with user:** doc lives in a new `docs/api.md`; commit straight to `main` (no MR). + +## Deliverable + +### 1. New file `docs/api.md` — full API reference + +Content (all facts verified against code during planning): + +- **Overview & base URL** — what the API is; in-cluster FQDN `http://egress-proxies-operator-controller-manager-discovery-service.egress-proxies-operator-system.svc.cluster.local:8090` (Service: `config/default/discovery_service.yaml`); local access via `kubectl port-forward svc/egress-proxies-operator-controller-manager-discovery-service 8090:8090`. +- **Authentication** — static bearer token from `DISCOVERY_TOKEN` env var (populated from the optional `discovery-token` Secret, key `token`; `config/manager/manager.yaml`). Empty token ⇒ auth disabled with startup warning. Curl: `-H "Authorization: Bearer $TOKEN"` on every example. `/healthz` always exempt. +- **Conventions** — JSON everywhere; error envelope `{"error":"","message":""}`; request bodies capped at 64 KiB; proxies with a deletion timestamp are excluded from all responses. +- **Configuration table** — `--discovery-addr` (default `:8090`), `--max-lease-ttl` (default 1h), `--lease-cooldown` (default 15m), `DISCOVERY_TOKEN`. Note the shipped Deployment passes none of these flags, so defaults apply. +- **Endpoints**, each with request/response schema, status codes, and a copy-pasteable curl example: + - `GET /healthz` — liveness, unauthenticated. + - `GET /v1/proxies` — filters `healthy=true|false` (else 400 `invalid_query`) and repeatable `attr.=` (verbatim equality on `spec.attributes`, all pairs must match). Response `{"proxies":[proxyView...],"count":N}` sorted by id. Full `proxyView` field table: `id` (ns/name), `ip`, `port`, `attributes`, `phase` (Pending/Provisioning/Ready/Unhealthy/Deleting/Failed), `healthy` (condition `Healthy` == True), `latencyMillis`, `activeLeases`, `maxLeases` (default 5; explicit 0 = unleasable). + - `POST /v1/leases` — body `{selector, ttlSeconds, target}` all optional; TTL defaults 5m, capped at max-lease-ttl (else 400 `invalid_ttl`). 201 `{leaseID, proxy, expiresAt, ttlSeconds}`; 409 `no_match` with `considered/atCapacity/inCooldown/unhealthy` counts (documented meanings). + - `DELETE /v1/leases/{id}` — early release; always 204, idempotent. + - `POST /v1/leases/{id}/report` — body `{result: ok|rate_limited|banned, target}`; 204, 404 `unknown_lease`, 400 `invalid_result`. `ok` is a pure ack; `rate_limited` and `banned` behave identically today (both start one cooldown window). +- **Selection & cooldown semantics** (short section — this is the non-obvious part clients need): + - Selection order: fewest active leases → lowest latency → lexicographic id; deterministic; only healthy, non-deleting proxies with free capacity are candidates. + - Cooldown: 15m default (`--lease-cooldown`), keyed `{proxy, target}`. Report with a target blocks only leases requesting that target; report without a target (and lease without one) creates a **global** cooldown blocking all acquisitions of that proxy — call this footgun out explicitly. + - Expired leases stay reportable for one cooldown window past TTL. +- **End-to-end workflow example** — numbered curl walkthrough: acquire → use `proxy.ip:port` as HTTP proxy (`curl -x`) → report `rate_limited` on 429 → release. Using a `jq`-extracted `leaseID`. +- **Caveats** — lease/cooldown state is in-memory and per-process: single replica only, operator restart drops all leases and cooldowns. + +### 2. Pointers to the new doc (small edits) + +- `README.md`: one-line link near the quickstart curl section ("full reference: docs/api.md"). +- `docs/architecture.md` §7: one-line link to `docs/api.md` as the detailed reference. + +### 3. Housekeeping per CLAUDE.md + +- First action post-approval: copy this plan to `docs/plans/-discovery-api-docs.md` (timestamp from `date "+%Y-%m-%d-%H%M"`), commit it alone. +- Then write the docs, commit to `main` with `Co-Authored-By: Claude ` trailer, push. +- Append execution summary + status checklist to `docs/plans-executions/-discovery-api-docs.md` in the docs commit. +- Add `CHANGELOG.md` entry (timestamp via `date "+%Y-%m-%d %H:%M %Z"`) once the user confirms. + +## Key source files (facts source of truth) + +- [internal/discovery/handlers.go](internal/discovery/handlers.go), [internal/discovery/server.go](internal/discovery/server.go) — routes, schemas, status codes, auth, limits. +- [internal/lease/store.go](internal/lease/store.go) — selection order, cooldown/retention, stats. +- [api/v1alpha1/proxy_types.go](api/v1alpha1/proxy_types.go), [api/v1alpha1/helpers.go](api/v1alpha1/helpers.go) — defaults (port 3128, maxLeases 5), phases, conditions. +- [cmd/main.go](cmd/main.go) — flags/env defaults. +- [config/default/discovery_service.yaml](config/default/discovery_service.yaml), [config/manager/manager.yaml](config/manager/manager.yaml) — service DNS, token secret. + +## Verification + +- Cross-check every documented status code / field name against `internal/discovery/server_test.go` expectations. +- Sanity-run `go build ./... && go test -short ./internal/discovery/ ./internal/lease/` (no code changes expected — confirms docs match current behavior, not a stale tree). +- Optionally lint the curl JSON bodies by piping each through `jq .` locally. From 09845e4eafc189aab5aeaea2e83f41ccf51f5c04 Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 11 Aug 2026 21:55:16 +0200 Subject: [PATCH 06/10] Document the discovery API in docs/api.md Full client-facing reference: auth, all routes with schemas and curl examples, selection/cooldown semantics, caveats. README and architecture.md link to it. Co-Authored-By: Claude --- README.md | 3 +- docs/api.md | 317 ++++++++++++++++++ docs/architecture.md | 2 + .../2026-08-11-2152-discovery-api-docs.md | 38 +++ 4 files changed, 359 insertions(+), 1 deletion(-) create mode 100644 docs/api.md create mode 100644 docs/plans-executions/2026-08-11-2152-discovery-api-docs.md diff --git a/README.md b/README.md index b00e9fd..fe8314b 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,8 @@ kubectl get px -w # proxy-kubernetes-sample Managed kubernetes Ready 10.244.x.x True ``` -Once it's `Ready`, port-forward the discovery API and use it: +Once it's `Ready`, port-forward the discovery API and use it (full +reference with schemas and error codes: [docs/api.md](docs/api.md)): ```sh kubectl -n egress-proxies-operator-system port-forward \ diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..714fd48 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,317 @@ +# Discovery API reference + +The operator serves an HTTP API (the *discovery API*) that crawler clients +use to find and lease egress proxies: list healthy proxies filtered by +attributes, acquire a TTL-based lease on one, release it early, and report +how a target site treated the proxy. It is implemented in +[`internal/discovery`](../internal/discovery/) with lease state in +[`internal/lease`](../internal/lease/); the only Kubernetes interaction is +reading `Proxy` resources from the manager's cache. + +## Base URL + +The API listens on `:8090` (`--discovery-addr`) inside the manager pod and +is exposed by a Service +([config/default/discovery_service.yaml](../config/default/discovery_service.yaml)). + +In-cluster: + +```text +http://egress-proxies-operator-controller-manager-discovery-service.egress-proxies-operator-system.svc.cluster.local:8090 +``` + +From a workstation, port-forward: + +```sh +kubectl -n egress-proxies-operator-system port-forward \ + svc/egress-proxies-operator-controller-manager-discovery-service 8090:8090 & +``` + +All examples below assume `localhost:8090` via that port-forward. + +## Authentication + +A single static bearer token, read from the `DISCOVERY_TOKEN` environment +variable at startup. The shipped Deployment populates it from the +`discovery-token` Secret (key `token`), which is **optional** — if the +Secret is absent or the token is empty, the API serves **unauthenticated** +(the manager logs a loud warning at startup). Create the Secret: + +```sh +kubectl -n egress-proxies-operator-system create secret generic discovery-token \ + --from-literal=token="$(openssl rand -hex 24)" +``` + +Send the token on every request: + +```sh +export TOKEN= +curl -s -H "Authorization: Bearer $TOKEN" localhost:8090/v1/proxies | jq +``` + +A missing or wrong token gets `401 {"error":"unauthorized",...}`. +`GET /healthz` is always exempt. + +The curl examples below omit the `-H "Authorization: Bearer $TOKEN"` flag +for brevity — add it to every call when auth is enabled. + +## Conventions + +- Requests and responses are JSON. Errors share one envelope: + + ```json + {"error": "", "message": ""} + ``` + +- Request bodies are capped at **64 KiB** (larger bodies fail the JSON + decode with `400 invalid_body`). +- Proxies with a deletion timestamp (being finalized) are excluded from + every response and never offered for lease. + +## Configuration + +| Setting | Default | Meaning | +|---|---|---| +| `--discovery-addr` | `:8090` | Listen address of the API | +| `--max-lease-ttl` | `1h` | Maximum `ttlSeconds` a client may request | +| `--lease-cooldown` | `15m` | Cooldown window applied on `rate_limited`/`banned` reports | +| `DISCOVERY_TOKEN` (env) | empty | Bearer token; empty disables auth | + +The shipped Deployment passes none of these flags, so the defaults apply. + +## Endpoints + +### `GET /healthz` + +Liveness check. Unauthenticated, always `200` with body `ok`. + +```sh +curl -s localhost:8090/healthz +``` + +### `GET /v1/proxies` — list proxies + +Query parameters (all optional): + +| Parameter | Values | Effect | +|---|---|---| +| `healthy` | `true` \| `false` | Keep only proxies whose `Healthy` condition matches. Any other value → `400 invalid_query`. | +| `attr.` | any string | Exact match on `spec.attributes[]`. Repeatable; **all** given pairs must match. | + +List everything: + +```sh +curl -s localhost:8090/v1/proxies | jq +``` + +List healthy proxies in a given geo: + +```sh +curl -s 'localhost:8090/v1/proxies?healthy=true&attr.geo=eu' | jq +``` + +Response — `200`, proxies sorted by `id`, an empty match is `200` with +`"count": 0` (never `404`): + +```json +{ + "proxies": [ + { + "id": "default/proxy-kubernetes-sample", + "ip": "10.244.1.7", + "port": 3128, + "attributes": {"geo": "local"}, + "phase": "Ready", + "healthy": true, + "latencyMillis": 42, + "activeLeases": 1, + "maxLeases": 5 + } + ], + "count": 1 +} +``` + +Proxy object fields (the same shape appears inside lease responses): + +| Field | Meaning | +|---|---| +| `id` | `namespace/name` of the `Proxy` resource; used as the stable key everywhere | +| `ip` | Effective host — `spec.endpoint.host` for `External` proxies, `status.ip` for `Managed` (empty until the backing VM/pod is up) | +| `port` | Effective port (default `3128`) | +| `attributes` | `spec.attributes` — free-form selection labels (`geo`, `asn`, `purpose`, …); omitted when empty | +| `phase` | `Pending` \| `Provisioning` \| `Ready` \| `Unhealthy` \| `Deleting` \| `Failed` | +| `healthy` | `true` iff the `Healthy` condition is `True` (the through-the-proxy health probe passes) | +| `latencyMillis` | Latency of the last status-affecting health probe | +| `activeLeases` | Currently active leases on this proxy | +| `maxLeases` | Lease capacity (default `5`; an explicit `0` means unleasable) | + +### `POST /v1/leases` — acquire a lease + +Picks a healthy proxy with free capacity matching the selector and grants +an exclusive-slot, TTL-based lease on it. + +Request body (every field optional; `{}` is valid): + +```json +{ + "selector": {"geo": "eu"}, + "ttlSeconds": 300, + "target": "example.com" +} +``` + +| Field | Default | Meaning | +|---|---|---| +| `selector` | none | Attribute equality filter, same semantics as `attr.` above | +| `ttlSeconds` | `300` (5 min) | Lease lifetime; must be ≤ `--max-lease-ttl` (default 1 h), else `400 invalid_ttl` | +| `target` | none | The site you intend to crawl; enables per-target cooldowns (see below) | + +```sh +curl -s -XPOST localhost:8090/v1/leases \ + -d '{"selector":{"geo":"eu"},"ttlSeconds":300,"target":"example.com"}' | jq +``` + +Success — `201`: + +```json +{ + "leaseID": "P3X6HHQTPCM5UTGVGE3B5UPS3A", + "proxy": { + "id": "default/proxy-eu-1", + "ip": "34.88.10.20", + "port": 3128, + "attributes": {"geo": "eu"}, + "phase": "Ready", + "healthy": true, + "latencyMillis": 42, + "activeLeases": 1, + "maxLeases": 5 + }, + "expiresAt": "2026-08-11T22:05:00Z", + "ttlSeconds": 300 +} +``` + +Use `proxy.ip` and `proxy.port` as an HTTP proxy for the lease's lifetime: + +```sh +curl -x http://34.88.10.20:3128 https://example.com +``` + +No match — `409` with diagnostic counts explaining why nothing qualified: + +```json +{ + "error": "no_match", + "message": "no healthy proxy with free capacity matched the selector", + "considered": 3, + "atCapacity": 1, + "inCooldown": 1, + "unhealthy": 1 +} +``` + +| Count | Meaning | +|---|---| +| `considered` | Proxies that matched the selector (before health/capacity checks) | +| `atCapacity` | Skipped because `activeLeases >= maxLeases` | +| `inCooldown` | Skipped because of an active cooldown for this target (or a global one) | +| `unhealthy` | Skipped because the `Healthy` condition is not `True` | + +Leases expire on their own — releasing is only needed to free the slot +early. There is no renew/extend endpoint; acquire a new lease instead. + +### `DELETE /v1/leases/{id}` — release early + +Frees the lease's capacity slot immediately. Idempotent: always `204`, +including for unknown or already-expired lease IDs. + +```sh +curl -si -XDELETE localhost:8090/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A +``` + +### `POST /v1/leases/{id}/report` — report an outcome + +Tell the operator how the target site treated the proxy. This is the +feedback signal that drives cooldowns. + +Request body: + +```json +{"result": "rate_limited", "target": "example.com"} +``` + +| Field | Values | Meaning | +|---|---|---| +| `result` | `ok` \| `rate_limited` \| `banned` | Anything else → `400 invalid_result` | +| `target` | optional | Which site produced the result; falls back to the lease's `target`, then to global | + +```sh +curl -si -XPOST localhost:8090/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A/report \ + -d '{"result":"rate_limited","target":"example.com"}' +``` + +Responses: `204` on success, `404 unknown_lease` if the lease ID was never +issued or has aged out. + +Semantics: + +- `ok` is a pure acknowledgement — nothing is recorded. +- `rate_limited` and `banned` currently behave **identically**: both put + the proxy in one cooldown window (default 15 min, `--lease-cooldown`) + for the resolved target. +- An expired lease remains reportable for one cooldown window past its + TTL, so a late "we got rate-limited" still lands. + +## Proxy selection and cooldowns + +How `POST /v1/leases` picks among eligible proxies (healthy, matching the +selector, not being deleted, not at capacity, not in cooldown), in order: + +1. fewest `activeLeases` (least-loaded), +2. lowest `latencyMillis`, +3. lexicographic `id` (deterministic tie-break). + +Cooldowns are keyed by **(proxy, target)**: + +- A report **with a target** blocks that proxy only for lease requests + naming the **same target**. Other targets — and requests with no + target — still get the proxy. +- A report **without a target**, on a lease that also had no target, + creates a **global** cooldown: the proxy is blocked for *all* lease + requests until the window passes. Always pass `target` on leases and + reports unless you really mean "this proxy is bad for everyone". + +## End-to-end example + +```sh +# 1. Acquire a lease for crawling example.com through an EU proxy +LEASE=$(curl -s -XPOST localhost:8090/v1/leases \ + -H "Authorization: Bearer $TOKEN" \ + -d '{"selector":{"geo":"eu"},"ttlSeconds":600,"target":"example.com"}') +LEASE_ID=$(echo "$LEASE" | jq -r .leaseID) +PROXY=$(echo "$LEASE" | jq -r '"\(.proxy.ip):\(.proxy.port)"') + +# 2. Crawl through the leased proxy +curl -x "http://$PROXY" https://example.com/some/page + +# 3. Got a 429? Report it — example.com-bound leases will avoid this +# proxy for the next 15 minutes +curl -s -XPOST "localhost:8090/v1/leases/$LEASE_ID/report" \ + -H "Authorization: Bearer $TOKEN" \ + -d '{"result":"rate_limited","target":"example.com"}' + +# 4. Done early? Release the slot (otherwise the TTL frees it) +curl -s -XDELETE "localhost:8090/v1/leases/$LEASE_ID" \ + -H "Authorization: Bearer $TOKEN" +``` + +## Caveats + +- **Lease and cooldown state is in-memory and per-process.** An operator + restart drops all active leases and cooldowns. Clients must tolerate a + granted lease disappearing (a subsequent report returns `404`). +- **Run a single replica.** The API is served by every manager replica but + is not leader-elected, and lease state is not shared between replicas; + the shipped Deployment pins `replicas: 1`. diff --git a/docs/architecture.md b/docs/architecture.md index 619cd98..7946caf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -213,6 +213,8 @@ Kubernetes interaction is reading Proxies from the manager's cache. The server is a non-leader-elected Runnable (all replicas would serve, but the deployment ships `replicas: 1` because lease state is per-process — an operator restart drops all leases and cooldowns, a documented caveat). +Client-facing reference with request/response schemas and curl examples: +[api.md](api.md). ```text crawler client diff --git a/docs/plans-executions/2026-08-11-2152-discovery-api-docs.md b/docs/plans-executions/2026-08-11-2152-discovery-api-docs.md new file mode 100644 index 0000000..b878d0a --- /dev/null +++ b/docs/plans-executions/2026-08-11-2152-discovery-api-docs.md @@ -0,0 +1,38 @@ +# Execution: Discovery API documentation + +Plan: [2026-08-11-2152-discovery-api-docs.md](../plans/2026-08-11-2152-discovery-api-docs.md) + +- [x] Step 1 — Write `docs/api.md` full API reference +- [x] Step 2 — Add pointers in README and architecture.md + +## Step 1 — docs/api.md + +Wrote the full reference: base URL (in-cluster FQDN + port-forward), bearer +auth, error envelope, configuration table, all five routes with schemas, +status codes and curl examples, the selection/cooldown semantics section, +an end-to-end curl walkthrough, and the in-memory/single-replica caveats. +All facts were taken from the code, not from memory of prior docs. + +Worth noting: the doc explicitly calls out two things no earlier doc +stated for clients — that a report **without** a target on a targetless +lease creates a *global* cooldown (blocking the proxy for everyone), and +that `rate_limited` and `banned` currently behave identically. Both came +straight from `internal/lease/store.go` and are easy to trip over. + +## Step 2 — Pointers + verification + +Added one-line links to the new doc in README's quickstart (above the curl +block) and in `docs/architecture.md` §7. Verified the documented behavior +against the tree rather than trusting the write-up: + +```sh +go build ./... && go test -short ./internal/discovery/ ./internal/lease/ +``` + +Both pass; a grep of `server_test.go` confirmed every documented status +code and error code (`invalid_ttl`, `invalid_query`, `invalid_result`, +`no_match`, `unknown_lease`, 201/204/401/404/409) is asserted by tests. + +Worth noting: CHANGELOG entry deliberately deferred until the user +confirms the docs read well, per the CHANGELOG convention's +"once the user confirms it works" clause. From f3ff6a0ca22b5b9c31e225f39c19aff7152707ba Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 11 Aug 2026 22:01:50 +0200 Subject: [PATCH 07/10] Use $BASE_URL variable in docs/api.md curl examples Co-Authored-By: Claude --- docs/api.md | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/docs/api.md b/docs/api.md index 714fd48..18c6e87 100644 --- a/docs/api.md +++ b/docs/api.md @@ -27,7 +27,13 @@ kubectl -n egress-proxies-operator-system port-forward \ svc/egress-proxies-operator-controller-manager-discovery-service 8090:8090 & ``` -All examples below assume `localhost:8090` via that port-forward. +Set `BASE_URL` to wherever you reach the API; all examples below use it: + +```sh +export BASE_URL=localhost:8090 # via the port-forward above +# or, from inside the cluster: +# export BASE_URL=http://egress-proxies-operator-controller-manager-discovery-service.egress-proxies-operator-system.svc.cluster.local:8090 +``` ## Authentication @@ -46,7 +52,7 @@ Send the token on every request: ```sh export TOKEN= -curl -s -H "Authorization: Bearer $TOKEN" localhost:8090/v1/proxies | jq +curl -s -H "Authorization: Bearer $TOKEN" "$BASE_URL/v1/proxies" | jq ``` A missing or wrong token gets `401 {"error":"unauthorized",...}`. @@ -86,7 +92,7 @@ The shipped Deployment passes none of these flags, so the defaults apply. Liveness check. Unauthenticated, always `200` with body `ok`. ```sh -curl -s localhost:8090/healthz +curl -s "$BASE_URL/healthz" ``` ### `GET /v1/proxies` — list proxies @@ -101,13 +107,13 @@ Query parameters (all optional): List everything: ```sh -curl -s localhost:8090/v1/proxies | jq +curl -s "$BASE_URL/v1/proxies" | jq ``` List healthy proxies in a given geo: ```sh -curl -s 'localhost:8090/v1/proxies?healthy=true&attr.geo=eu' | jq +curl -s "$BASE_URL/v1/proxies?healthy=true&attr.geo=eu" | jq ``` Response — `200`, proxies sorted by `id`, an empty match is `200` with @@ -168,7 +174,7 @@ Request body (every field optional; `{}` is valid): | `target` | none | The site you intend to crawl; enables per-target cooldowns (see below) | ```sh -curl -s -XPOST localhost:8090/v1/leases \ +curl -s -XPOST "$BASE_URL/v1/leases" \ -d '{"selector":{"geo":"eu"},"ttlSeconds":300,"target":"example.com"}' | jq ``` @@ -228,7 +234,7 @@ Frees the lease's capacity slot immediately. Idempotent: always `204`, including for unknown or already-expired lease IDs. ```sh -curl -si -XDELETE localhost:8090/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A +curl -si -XDELETE "$BASE_URL/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A" ``` ### `POST /v1/leases/{id}/report` — report an outcome @@ -248,7 +254,7 @@ Request body: | `target` | optional | Which site produced the result; falls back to the lease's `target`, then to global | ```sh -curl -si -XPOST localhost:8090/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A/report \ +curl -si -XPOST "$BASE_URL/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A/report" \ -d '{"result":"rate_limited","target":"example.com"}' ``` @@ -287,7 +293,7 @@ Cooldowns are keyed by **(proxy, target)**: ```sh # 1. Acquire a lease for crawling example.com through an EU proxy -LEASE=$(curl -s -XPOST localhost:8090/v1/leases \ +LEASE=$(curl -s -XPOST "$BASE_URL/v1/leases" \ -H "Authorization: Bearer $TOKEN" \ -d '{"selector":{"geo":"eu"},"ttlSeconds":600,"target":"example.com"}') LEASE_ID=$(echo "$LEASE" | jq -r .leaseID) @@ -298,12 +304,12 @@ curl -x "http://$PROXY" https://example.com/some/page # 3. Got a 429? Report it — example.com-bound leases will avoid this # proxy for the next 15 minutes -curl -s -XPOST "localhost:8090/v1/leases/$LEASE_ID/report" \ +curl -s -XPOST "$BASE_URL/v1/leases/$LEASE_ID/report" \ -H "Authorization: Bearer $TOKEN" \ -d '{"result":"rate_limited","target":"example.com"}' # 4. Done early? Release the slot (otherwise the TTL frees it) -curl -s -XDELETE "localhost:8090/v1/leases/$LEASE_ID" \ +curl -s -XDELETE "$BASE_URL/v1/leases/$LEASE_ID" \ -H "Authorization: Bearer $TOKEN" ``` From e1abac3e8f33ee517d8104dcdcba1ad25618b8a3 Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 11 Aug 2026 22:21:28 +0200 Subject: [PATCH 08/10] Add plan: demo script for egress IP check via proxies Co-Authored-By: Claude --- docs/plans/2026-08-11-2220-demo-scripts.md | 33 ++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 docs/plans/2026-08-11-2220-demo-scripts.md diff --git a/docs/plans/2026-08-11-2220-demo-scripts.md b/docs/plans/2026-08-11-2220-demo-scripts.md new file mode 100644 index 0000000..8b0d165 --- /dev/null +++ b/docs/plans/2026-08-11-2220-demo-scripts.md @@ -0,0 +1,33 @@ +# Plan: Demo script — egress IP check through each healthy proxy +**Created:** 2026-08-11 22:20 + + +## Context + +First of a planned series of demo scripts under `docs/demo/`. This one showcases the discovery API end to end without leases: list proxies from `$BASE_URL/v1/proxies`, and for each **healthy** one, call an IP-echo site through it (`curl -x`) to show the egress IP that proxy provides — clearly labeling which proxy each request goes through. The user explicitly asked to start by switching to a new branch. The script will be iterated on: **write it but do not commit it** — the user wants to add things to it before anything is committed. + +## Steps + +### Step 0 — Branch + +Create `feat/demo-scripts` off `main`. Commit only the plan copy (`docs/plans/-demo-scripts.md`, timestamp via `date "+%Y-%m-%d-%H%M"`) per CLAUDE.md — nothing else gets committed this round. + +### Step 1 — `docs/demo/show-egress-ips.sh` (new file, executable, left uncommitted) + +Style: match [.devcontainer/post-install.sh](.devcontainer/post-install.sh) — `#!/bin/bash`, `set -euo pipefail`, `ERROR:`/`WARNING:` messages, `${VAR}` braces. + +Behavior: + +1. **Args/env:** `BASE_URL` is `$1` (required; missing → usage text + exit 1, e.g. `usage: show-egress-ips.sh (e.g. localhost:8090)`). Optional env: `TOKEN` (bearer token, same name docs/api.md uses; when set, send `Authorization: Bearer $TOKEN`), `IP_ECHO_URL` (default `https://api.ipify.org?format=json` — returns `{"ip":"..."}`). +2. **Dependency check:** `command -v curl`, `command -v jq` → `ERROR` + exit 1 if missing. +3. **Fetch** `"$BASE_URL/v1/proxies"` once (no server-side `healthy` filter — fetch all so unhealthy ones can be shown as skipped, which makes the demo more informative). Fail with a clear error if curl or JSON parsing fails. +4. **Iterate** proxies with `jq -c '.proxies[]'`; for each, extract `id`, `ip`, `port`, `healthy`: + - unhealthy → print `--- skipping (unhealthy) ---` + - healthy → print a clear banner naming the proxy before the request, e.g. `=== via — http://: ===`, then `curl -sS --max-time 10 -x "http://${ip}:${port}" "$IP_ECHO_URL"`; print the JSON response. A failed probe prints `WARNING: request through failed` and continues (guard so `set -e` doesn't kill the loop). +5. Finish with a one-line summary: N proxies, M probed, K skipped/failed. + +Reference for API shapes: [docs/api.md](docs/api.md) (`proxies[].id/ip/port/healthy`; `curl -x http://ip:port` usage is already documented there and in README). + +### Deliberately deferred (user will iterate on the script first) + +- No commit of the script, no push beyond the plan commit, no MR, no execution summary, no CHANGELOG — all wait until the user says the script (or script set) is ready. From 0d68111bc24e6293b4de8d00295ca9135050e333 Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 11 Aug 2026 22:40:24 +0200 Subject: [PATCH 09/10] Add demo scripts: egress IP check and bulk proxy creation docs/demo/show-egress-ips.sh probes each healthy proxy from the discovery API against an IP-echo site; create-kubernetes-proxies.sh and create-gcp-proxies.sh bulk-create demo Proxies, the gcp one spreading them across randomly picked EU zones. Co-Authored-By: Claude --- docs/demo/create-gcp-proxies.sh | 94 +++++++++++++++++++ docs/demo/create-kubernetes-proxies.sh | 48 ++++++++++ docs/demo/show-egress-ips.sh | 74 +++++++++++++++ .../2026-08-11-2220-demo-scripts.md | 38 ++++++++ 4 files changed, 254 insertions(+) create mode 100755 docs/demo/create-gcp-proxies.sh create mode 100755 docs/demo/create-kubernetes-proxies.sh create mode 100755 docs/demo/show-egress-ips.sh create mode 100644 docs/plans-executions/2026-08-11-2220-demo-scripts.md diff --git a/docs/demo/create-gcp-proxies.sh b/docs/demo/create-gcp-proxies.sh new file mode 100755 index 0000000..3a3665d --- /dev/null +++ b/docs/demo/create-gcp-proxies.sh @@ -0,0 +1,94 @@ +#!/bin/bash +# Demo: create N Managed proxies backed by the gcp provider, named +# proxy-gcp-demo-1 .. proxy-gcp-demo-N, each in a randomly picked EU zone +# so the fleet gets egress IPs from different locations. +# +# Usage: +# ./create-gcp-proxies.sh +# +# Optional environment: +# NAMESPACE namespace to create the proxies in (default: current context) +# GCP_PROVIDER provider NAME from providers.yaml (default: gcp-eu) +# ZONES space-separated zone list to pick from (default: EU zones below) +set -euo pipefail + +if [ $# -ne 1 ] || ! [[ "$1" =~ ^[1-9][0-9]*$ ]]; then + echo "usage: $(basename "$0") (positive integer)" >&2 + exit 1 +fi +count="$1" +provider="${GCP_PROVIDER:-gcp-eu}" + +# GCP zones in the EU where e2-micro is generally available. Override with +# ZONES="zone1 zone2 ..." if your project has quota only in some of them. +default_zones=( + europe-west1-b europe-west1-c europe-west1-d # Belgium + europe-west2-a europe-west2-b europe-west2-c # London + europe-west3-a europe-west3-b europe-west3-c # Frankfurt + europe-west4-a europe-west4-b europe-west4-c # Netherlands + europe-west6-a europe-west6-b europe-west6-c # Zurich + europe-west8-a europe-west8-b europe-west8-c # Milan + europe-west9-a europe-west9-b europe-west9-c # Paris + europe-central2-a europe-central2-b europe-central2-c # Warsaw + europe-north1-a europe-north1-b europe-north1-c # Finland + europe-southwest1-a europe-southwest1-b europe-southwest1-c # Madrid +) +if [ -n "${ZONES:-}" ]; then + read -r -a zones <<< "${ZONES}" +else + zones=("${default_zones[@]}") +fi + +if ! command -v kubectl &> /dev/null; then + echo "ERROR: kubectl is required but not installed" >&2 + exit 1 +fi + +ns_args=() +if [ -n "${NAMESPACE:-}" ]; then + ns_args=(-n "${NAMESPACE}") +fi + +for i in $(seq 1 "${count}"); do + zone="${zones[RANDOM % ${#zones[@]}]}" + echo "Creating proxy-gcp-demo-${i} in ${zone} ..." + kubectl apply "${ns_args[@]}" -f - << EOF +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: proxy-gcp-demo-${i} +spec: + mode: Managed + provider: ${provider} # must match a provider NAME in providers.yaml + placement: + zone: ${zone} + machineType: e2-micro + # debian-cloud images have no cloud-init, so spec.cloudInit (passed as + # user-data metadata) would be silently ignored there. Ubuntu images do. + image: projects/ubuntu-os-cloud/global/images/family/ubuntu-2404-lts-amd64 + port: 3128 + cloudInit: + inline: | + #cloud-config + package_update: true + packages: + - squid + write_files: + - path: /etc/squid/conf.d/proxy-operator.conf + content: | + http_access allow all + via off + forwarded_for off + runcmd: + - systemctl restart squid + attributes: + geo: eu + zone: ${zone} + purpose: crawl +EOF +done + +echo "" +echo "Created ${count} proxies. VMs take a few minutes to provision and pass" +echo "the health check. Watch them come up with:" +echo " kubectl get px ${ns_args[*]:-} -w" diff --git a/docs/demo/create-kubernetes-proxies.sh b/docs/demo/create-kubernetes-proxies.sh new file mode 100755 index 0000000..226ccf4 --- /dev/null +++ b/docs/demo/create-kubernetes-proxies.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# Demo: create N Managed proxies backed by the kubernetes-pod provider, +# named proxy-kubernetes-demo-1 .. proxy-kubernetes-demo-N. Pods share the +# cluster's egress IP — this exercises the full lifecycle, not distinct +# egress paths (use create-gcp-proxies.sh for that). +# +# Usage: +# ./create-kubernetes-proxies.sh +# +# Optional environment: +# NAMESPACE namespace to create the proxies in (default: current context) +set -euo pipefail + +if [ $# -ne 1 ] || ! [[ "$1" =~ ^[1-9][0-9]*$ ]]; then + echo "usage: $(basename "$0") (positive integer)" >&2 + exit 1 +fi +count="$1" + +if ! command -v kubectl &> /dev/null; then + echo "ERROR: kubectl is required but not installed" >&2 + exit 1 +fi + +ns_args=() +if [ -n "${NAMESPACE:-}" ]; then + ns_args=(-n "${NAMESPACE}") +fi + +for i in $(seq 1 "${count}"); do + echo "Creating proxy-kubernetes-demo-${i} ..." + kubectl apply "${ns_args[@]}" -f - << EOF +apiVersion: crawl.example.com/v1alpha1 +kind: Proxy +metadata: + name: proxy-kubernetes-demo-${i} +spec: + mode: Managed + provider: kubernetes + attributes: + geo: local + purpose: crawl +EOF +done + +echo "" +echo "Created ${count} proxies. Watch them come up with:" +echo " kubectl get px ${ns_args[*]:-} -w" diff --git a/docs/demo/show-egress-ips.sh b/docs/demo/show-egress-ips.sh new file mode 100755 index 0000000..f0940ed --- /dev/null +++ b/docs/demo/show-egress-ips.sh @@ -0,0 +1,74 @@ +#!/bin/bash +# Demo: list proxies from the discovery API and show the egress IP each +# healthy one provides, by calling an IP-echo site through it. +# +# Usage: +# ./show-egress-ips.sh e.g. ./show-egress-ips.sh localhost:8090 +# +# Optional environment: +# TOKEN bearer token for the discovery API (see docs/api.md) +# IP_ECHO_URL site that returns the caller's IP as JSON +# (default: https://api.ipify.org?format=json) +set -euo pipefail + +if [ $# -ne 1 ]; then + echo "usage: $(basename "$0") (e.g. localhost:8090)" >&2 + exit 1 +fi +BASE_URL="$1" +IP_ECHO_URL="${IP_ECHO_URL:-https://api.ipify.org?format=json}" + +for tool in curl jq; do + if ! command -v "${tool}" &> /dev/null; then + echo "ERROR: ${tool} is required but not installed" >&2 + exit 1 + fi +done + +auth_args=() +if [ -n "${TOKEN:-}" ]; then + auth_args=(-H "Authorization: Bearer ${TOKEN}") +fi + +echo "Fetching proxies from ${BASE_URL}/v1/proxies ..." +if ! proxies_json=$(curl -sS --fail "${auth_args[@]}" "${BASE_URL}/v1/proxies"); then + echo "ERROR: could not fetch proxy list from ${BASE_URL}" >&2 + exit 1 +fi +if ! echo "${proxies_json}" | jq -e . > /dev/null; then + echo "ERROR: response from ${BASE_URL}/v1/proxies is not valid JSON" >&2 + exit 1 +fi + +total=$(echo "${proxies_json}" | jq -r '.count') +echo "Found ${total} proxies" +echo "" + +probed=0 +skipped=0 +failed=0 +while IFS= read -r proxy; do + id=$(echo "${proxy}" | jq -r '.id') + ip=$(echo "${proxy}" | jq -r '.ip') + port=$(echo "${proxy}" | jq -r '.port') + healthy=$(echo "${proxy}" | jq -r '.healthy') + + if [ "${healthy}" != "true" ]; then + echo "--- skipping ${id} (unhealthy) ---" + echo "" + skipped=$((skipped + 1)) + continue + fi + + echo "=== via ${id} — http://${ip}:${port} ===" + if response=$(curl -sS --max-time 10 -x "http://${ip}:${port}" "${IP_ECHO_URL}"); then + echo "${response}" | jq . 2> /dev/null || echo "${response}" + probed=$((probed + 1)) + else + echo "WARNING: request through ${id} failed" >&2 + failed=$((failed + 1)) + fi + echo "" +done < <(echo "${proxies_json}" | jq -c '.proxies[]') + +echo "Done: ${total} proxies — ${probed} probed, ${skipped} skipped (unhealthy), ${failed} failed" diff --git a/docs/plans-executions/2026-08-11-2220-demo-scripts.md b/docs/plans-executions/2026-08-11-2220-demo-scripts.md new file mode 100644 index 0000000..56dc928 --- /dev/null +++ b/docs/plans-executions/2026-08-11-2220-demo-scripts.md @@ -0,0 +1,38 @@ +# Execution: Demo scripts + +Plan: [2026-08-11-2220-demo-scripts.md](../plans/2026-08-11-2220-demo-scripts.md) + +- [x] Step 0 — Branch `feat/demo-scripts` + plan commit +- [x] Step 1 — `docs/demo/show-egress-ips.sh` +- [x] Extra (added iteratively, not in the original plan) — proxy-creation scripts + +## Step 0 + Step 1 + +Branched off `main`, committed the plan alone, then wrote +`docs/demo/show-egress-ips.sh`: takes `BASE_URL` as its argument, lists +`/v1/proxies` (bearer auth via optional `TOKEN` env), probes each healthy +proxy with `curl -x http://ip:port` against an IP-echo site +(`IP_ECHO_URL`, default ipify JSON), banners which proxy each request goes +through, skips unhealthy ones, and ends with a probed/skipped/failed +summary. Per user request the script was left uncommitted for iteration +and no verification beyond `bash -n` was run. + +## Extra — create-kubernetes-proxies.sh, create-gcp-proxies.sh + +Added on the same branch before the first commit: + +- `create-kubernetes-proxies.sh ` — creates + `proxy-kubernetes-demo-1..N` with the kubernetes provider, spec taken + from `config/samples/proxy_kubernetes.yaml`, applied via + `kubectl apply -f -` heredocs. Optional `NAMESPACE` env. +- `create-gcp-proxies.sh ` — creates `proxy-gcp-demo-1..N` from the + user-supplied gcp-eu manifest (e2-micro, Ubuntu 24.04, Squid + cloud-init), each with a zone picked randomly from a hardcoded list of + 27 EU zones so the fleet gets egress IPs from different locations. + Env overrides: `ZONES`, `GCP_PROVIDER` (default `gcp-eu`), `NAMESPACE`. + +Worth noting: beyond the user's sample manifest, the gcp script also +writes the picked zone into `attributes.zone`, so the discovery API +exposes each proxy's location and leases can select on it. The zone list +is static — if a project lacks quota in some region, `ZONES` narrows the +pool; nothing validates zones against the live project. From f6b67006dc41a00bfb58b16e7a0738514c07018c Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 11 Aug 2026 23:10:23 +0200 Subject: [PATCH 10/10] Add tmux demo driver and table variant of egress IP check run-demo.sh opens a 2x2 tmux grid: egress-IP table looping in a netshoot pod, kubectl get px watch, and both create scripts running with COUNT (default 4) proxies. show-egress-ips-table.sh is the pane-sized one-line-per-proxy variant used by the driver. Co-Authored-By: Claude --- docs/demo/run-demo.sh | 67 +++++++++++++++++ docs/demo/show-egress-ips-table.sh | 75 +++++++++++++++++++ .../2026-08-11-2220-demo-scripts.md | 17 +++++ 3 files changed, 159 insertions(+) create mode 100755 docs/demo/run-demo.sh create mode 100755 docs/demo/show-egress-ips-table.sh diff --git a/docs/demo/run-demo.sh b/docs/demo/run-demo.sh new file mode 100755 index 0000000..4376d00 --- /dev/null +++ b/docs/demo/run-demo.sh @@ -0,0 +1,67 @@ +#!/bin/bash +# Demo driver: opens a tmux session with a 2x2 pane grid: +# +# top-left: show-egress-ips-table.sh in a 10s loop, run inside the +# netshoot pod against the in-cluster discovery Service +# top-right: watch -n3 kubectl get px +# bottom-left: create-kubernetes-proxies.sh +# bottom-right: create-gcp-proxies.sh +# +# Usage: +# ./run-demo.sh +# +# Optional environment: +# COUNT proxies each create script makes (default: 4) +# SESSION tmux session name (default: proxy-demo; an existing +# session with this name is killed and recreated) +# NETSHOOT_POD pod to exec into for the egress-IP loop (default: netshoot) +# DEMO_DIR where the demo scripts live (default: this script's dir) +set -euo pipefail + +COUNT="${COUNT:-4}" +SESSION="${SESSION:-proxy-demo}" +NETSHOOT_POD="${NETSHOOT_POD:-netshoot}" +DEMO_DIR="${DEMO_DIR:-$(cd "$(dirname "$0")" && pwd)}" +BASE_URL="http://egress-proxies-operator-controller-manager-discovery-service.egress-proxies-operator-system.svc.cluster.local:8090" + +for tool in tmux kubectl; do + if ! command -v "${tool}" &> /dev/null; then + echo "ERROR: ${tool} is required but not installed" >&2 + exit 1 + fi +done + +if ! kubectl get pod "${NETSHOOT_POD}" &> /dev/null; then + echo "ERROR: pod ${NETSHOOT_POD} not found — start one with:" >&2 + echo " kubectl run netshoot --image=nicolaka/netshoot -- sleep infinity" >&2 + exit 1 +fi + +echo "Copying show-egress-ips-table.sh into pod ${NETSHOOT_POD} ..." +kubectl cp "${DEMO_DIR}/show-egress-ips-table.sh" "${NETSHOOT_POD}:/tmp/show-egress-ips-table.sh" + +if tmux has-session -t "${SESSION}" 2> /dev/null; then + echo "Killing existing tmux session ${SESSION}" + tmux kill-session -t "${SESSION}" +fi + +# 2x2 grid: after these splits pane indexes are 0 top-left, 1 top-right, +# 2 bottom-left, 3 bottom-right; tiled layout evens them into quarters. +tmux new-session -d -s "${SESSION}" +tmux split-window -h -t "${SESSION}:0" +tmux split-window -v -t "${SESSION}:0.0" +tmux split-window -v -t "${SESSION}:0.1" +tmux select-layout -t "${SESSION}:0" tiled + +loop_cmd="BASE_URL=${BASE_URL}; while true; do bash /tmp/show-egress-ips-table.sh \"\$BASE_URL\"; echo; sleep 10; done" +tmux send-keys -t "${SESSION}:0.0" "kubectl exec -it ${NETSHOOT_POD} -- bash -c '${loop_cmd}'" C-m +tmux send-keys -t "${SESSION}:0.1" "watch -n3 kubectl get px" C-m +tmux send-keys -t "${SESSION}:0.2" "bash ${DEMO_DIR}/create-kubernetes-proxies.sh ${COUNT}" C-m +tmux send-keys -t "${SESSION}:0.3" "bash ${DEMO_DIR}/create-gcp-proxies.sh ${COUNT}" C-m + +tmux select-pane -t "${SESSION}:0.2" +if [ -n "${TMUX:-}" ]; then + tmux switch-client -t "${SESSION}" +else + tmux attach-session -t "${SESSION}" +fi diff --git a/docs/demo/show-egress-ips-table.sh b/docs/demo/show-egress-ips-table.sh new file mode 100755 index 0000000..465a146 --- /dev/null +++ b/docs/demo/show-egress-ips-table.sh @@ -0,0 +1,75 @@ +#!/bin/bash +# Demo: condensed-table variant of show-egress-ips.sh, made to fit a small +# tmux pane. One line per proxy: which proxy the request goes through, its +# endpoint, location (zone/geo attribute), and the egress IP the IP-echo +# site saw — or unhealthy/FAILED. +# +# Usage: +# ./show-egress-ips-table.sh e.g. ./show-egress-ips-table.sh localhost:8090 +# +# Optional environment: +# TOKEN bearer token for the discovery API (see docs/api.md) +# IP_ECHO_URL site that returns the caller's IP as JSON with an "ip" field +# (default: https://api.ipify.org?format=json) +set -euo pipefail + +if [ $# -ne 1 ]; then + echo "usage: $(basename "$0") (e.g. localhost:8090)" >&2 + exit 1 +fi +BASE_URL="$1" +IP_ECHO_URL="${IP_ECHO_URL:-https://api.ipify.org?format=json}" + +for tool in curl jq; do + if ! command -v "${tool}" &> /dev/null; then + echo "ERROR: ${tool} is required but not installed" >&2 + exit 1 + fi +done + +auth_args=() +if [ -n "${TOKEN:-}" ]; then + auth_args=(-H "Authorization: Bearer ${TOKEN}") +fi + +if ! proxies_json=$(curl -sS --fail "${auth_args[@]}" "${BASE_URL}/v1/proxies") \ + || ! echo "${proxies_json}" | jq -e . > /dev/null 2>&1; then + echo "ERROR: could not fetch proxy list from ${BASE_URL}" >&2 + exit 1 +fi + +total=$(echo "${proxies_json}" | jq -r '.count') +fmt="%-31s %-21s %-21s %s\n" + +echo "${total} proxies @ $(date +%H:%M:%S)" +# shellcheck disable=SC2059 +printf "${fmt}" "PROXY" "ENDPOINT" "LOCATION" "EGRESS-IP" + +probed=0 +skipped=0 +failed=0 +while IFS= read -r proxy; do + id=$(echo "${proxy}" | jq -r '.id') + endpoint=$(echo "${proxy}" | jq -r '"\(.ip):\(.port)"') + location=$(echo "${proxy}" | jq -r '.attributes.zone // .attributes.geo // "-"') + healthy=$(echo "${proxy}" | jq -r '.healthy') + + if [ "${healthy}" != "true" ]; then + # shellcheck disable=SC2059 + printf "${fmt}" "${id}" "${endpoint}" "${location}" "(unhealthy)" + skipped=$((skipped + 1)) + continue + fi + + if response=$(curl -sS --max-time 10 -x "http://${endpoint}" "${IP_ECHO_URL}" 2> /dev/null); then + egress=$(echo "${response}" | jq -r '.ip // "?"' 2> /dev/null || echo "?") + probed=$((probed + 1)) + else + egress="FAILED" + failed=$((failed + 1)) + fi + # shellcheck disable=SC2059 + printf "${fmt}" "${id}" "${endpoint}" "${location}" "${egress}" +done < <(echo "${proxies_json}" | jq -c '.proxies[]') + +echo "-- ${probed} probed, ${skipped} unhealthy, ${failed} failed --" diff --git a/docs/plans-executions/2026-08-11-2220-demo-scripts.md b/docs/plans-executions/2026-08-11-2220-demo-scripts.md index 56dc928..77614ba 100644 --- a/docs/plans-executions/2026-08-11-2220-demo-scripts.md +++ b/docs/plans-executions/2026-08-11-2220-demo-scripts.md @@ -31,6 +31,23 @@ Added on the same branch before the first commit: 27 EU zones so the fleet gets egress IPs from different locations. Env overrides: `ZONES`, `GCP_PROVIDER` (default `gcp-eu`), `NAMESPACE`. +## Extra — run-demo.sh, show-egress-ips-table.sh + +Second round of iterative additions: + +- `run-demo.sh` — tmux demo driver: 2x2 tiled grid with the egress-IP + table looping every 10s inside a netshoot pod (script `kubectl cp`'d + into the pod, `BASE_URL` set to the in-cluster Service FQDN), + `watch -n3 kubectl get px`, and both create scripts auto-running with + `COUNT` proxies each (default 4). Env knobs: `COUNT`, `SESSION`, + `NETSHOOT_POD`, `DEMO_DIR` (defaults to the script's own dir). +- `show-egress-ips-table.sh` — condensed one-line-per-proxy variant of + `show-egress-ips.sh` sized for a tmux pane: PROXY / ENDPOINT / + LOCATION (zone→geo attribute fallback) / EGRESS-IP columns, with + `(unhealthy)` and `FAILED` inline instead of verbose output. The + verbose script stays for standalone use; the demo driver uses the + table variant. + Worth noting: beyond the user's sample manifest, the gcp script also writes the picked zone into `attributes.zone`, so the discovery API exposes each proxy's location and leases can select on it. The zone list