Compare commits
6 Commits
09845e4eaf
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 5ccd317bec | |||
| 6ff9eebefc | |||
| f6b67006dc | |||
| 0d68111bc2 | |||
| e1abac3e8f | |||
| f3ff6a0ca2 |
@@ -83,7 +83,17 @@
|
|||||||
"Bash(kubectl -n egress-proxies-operator-system get pods -o wide)",
|
"Bash(kubectl -n egress-proxies-operator-system get pods -o wide)",
|
||||||
"Bash(kubectl -n egress-proxies-operator-system get deploy egress-proxies-operator-controller-manager -o jsonpath='{.spec.template.spec.containers[0].args}')",
|
"Bash(kubectl -n egress-proxies-operator-system get deploy egress-proxies-operator-controller-manager -o jsonpath='{.spec.template.spec.containers[0].args}')",
|
||||||
"Bash(kubectl -n egress-proxies-operator-system logs deploy/egress-proxies-operator-controller-manager)",
|
"Bash(kubectl -n egress-proxies-operator-system logs deploy/egress-proxies-operator-controller-manager)",
|
||||||
"Bash(python3 -c \"import json; d=json.load\\(open\\('docs/deploy/sa_key.json'\\)\\); print\\(d.get\\('type'\\), d.get\\('client_email'\\)\\)\")"
|
"Bash(python3 -c \"import json; d=json.load\\(open\\('docs/deploy/sa_key.json'\\)\\); print\\(d.get\\('type'\\), d.get\\('client_email'\\)\\)\")",
|
||||||
|
"Bash(tea pr *)",
|
||||||
|
"Bash(git worktree *)",
|
||||||
|
"Bash(python3 -c \"import yaml; yaml.safe_load\\(open\\('.gitea/workflows/build.yaml'\\)\\); print\\('YAML OK'\\)\")",
|
||||||
|
"Bash(ruby -ryaml -e \"YAML.load_file\\('.gitea/workflows/build.yaml'\\); puts 'YAML OK'\")",
|
||||||
|
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator status --short --branch)",
|
||||||
|
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator log --oneline -1)",
|
||||||
|
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator tag 0.01)",
|
||||||
|
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator push origin 0.01)",
|
||||||
|
"Bash(chmod +x docs/demo/run-demo.sh)",
|
||||||
|
"Bash(bash -n docs/demo/run-demo.sh)"
|
||||||
],
|
],
|
||||||
"additionalDirectories": [
|
"additionalDirectories": [
|
||||||
"/Users/jan.novak/srv/go/egress-proxies-operator/.claude",
|
"/Users/jan.novak/srv/go/egress-proxies-operator/.claude",
|
||||||
|
|||||||
56
README.md
56
README.md
@@ -136,6 +136,62 @@ Cloud-init from a Secret: the Secret **must** carry the label
|
|||||||
labelled Secrets, so an unlabelled one is invisible (the Proxy reports
|
labelled Secrets, so an unlabelled one is invisible (the Proxy reports
|
||||||
`CloudInitError`). Rotating the Secret's content triggers VM replacement.
|
`CloudInitError`). Rotating the Secret's content triggers VM replacement.
|
||||||
|
|
||||||
|
## Providers
|
||||||
|
|
||||||
|
`Managed` proxies are provisioned by a provider — a small compute backend
|
||||||
|
behind one minimal interface. Providers are configured in the
|
||||||
|
`--providers-config` YAML file as **named instances**: `spec.provider` on a
|
||||||
|
Proxy refers to an entry's `name`, not its `type`, so `gcp-eu` and `gcp-us`
|
||||||
|
can be two differently-configured instances of the same `gcp` type (see
|
||||||
|
[config/samples/providers-config.yaml](config/samples/providers-config.yaml)).
|
||||||
|
|
||||||
|
### Implemented providers
|
||||||
|
|
||||||
|
| Type | Creates | Per-instance config | Notes |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `kubernetes` | A real Squid pod (`ubuntu/squid:6.6-24.04_edge` by default) in the same cluster the operator runs in | `image` (optional) | Needs no cloud account — local dev, CI, and the kind quickstart. Pods share the cluster's egress IP, so it exercises the full lifecycle but not distinct egress paths. |
|
||||||
|
| `gcp` | A Compute Engine VM with an ephemeral external IP | `project` (required), `network`, `networkTag`, `diskSizeGb` | The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: `Create` returns as soon as the operation is submitted and the reconciler discovers progress by polling `Get`. Auth is Application Default Credentials — workload identity in-cluster, `gcloud` ADC locally; no key-file plumbing. |
|
||||||
|
|
||||||
|
### Adding a provider
|
||||||
|
|
||||||
|
A new backend (Hetzner, AWS, ...) is four pieces; the contract lives in
|
||||||
|
[internal/provider/provider.go](internal/provider/provider.go):
|
||||||
|
|
||||||
|
1. **Implement the 4-method `Provider` interface** in a new
|
||||||
|
`internal/provider/<type>/` package:
|
||||||
|
- `Create` submits and returns — it never blocks until the VM runs, and
|
||||||
|
must be idempotent keyed on `req.Name` (a deterministic name derived
|
||||||
|
from the Proxy's UID), so a repeat call after a crash finds the
|
||||||
|
existing instance instead of duplicating it.
|
||||||
|
- `Get` returns `provider.ErrNotFound` as a *normal* outcome — the
|
||||||
|
reconciler branches on it for replacement and adoption, so don't
|
||||||
|
treat it as exceptional.
|
||||||
|
- `Delete` is idempotent: deleting an already-gone instance is not an
|
||||||
|
error.
|
||||||
|
- `ListByTag` returns every instance the operator ever tagged, for
|
||||||
|
orphan GC.
|
||||||
|
2. **Tag every created resource** with `LabelManaged=true` and
|
||||||
|
`LabelUID=<Proxy UID>`, and report `CreatedAt` — orphan GC relies on
|
||||||
|
all three to find owned resources and skip in-flight creates.
|
||||||
|
3. **Classify every returned error** with `provider.Wrap` into the
|
||||||
|
four-sentinel taxonomy in
|
||||||
|
[internal/provider/errors.go](internal/provider/errors.go)
|
||||||
|
(`ErrNotFound` / `ErrQuotaExceeded` / `ErrTransient` / `ErrPermanent`)
|
||||||
|
— the reconciler decides retry, slow backoff, or latching `Failed`
|
||||||
|
purely from that classification, never from provider-specific types.
|
||||||
|
4. **Wire it up**: add a type-specific config block in
|
||||||
|
[internal/provider/config.go](internal/provider/config.go), and
|
||||||
|
register the constructor in the builtins map in
|
||||||
|
[cmd/main.go](cmd/main.go) (`"<type>": <pkg>.New`). The registry
|
||||||
|
([internal/provider/registry](internal/provider/registry/registry.go))
|
||||||
|
handles named instances, and the metrics wrapper is applied
|
||||||
|
automatically.
|
||||||
|
|
||||||
|
Test against a fake API seam rather than the real cloud — see the
|
||||||
|
`instancesAPI` seam in
|
||||||
|
[internal/provider/gcp/gcp.go](internal/provider/gcp/gcp.go) for the
|
||||||
|
pattern.
|
||||||
|
|
||||||
## Caveats — read these two
|
## Caveats — read these two
|
||||||
|
|
||||||
**Changing a proxy changes its IP.** Proxies are immutable cattle: editing
|
**Changing a proxy changes its IP.** Proxies are immutable cattle: editing
|
||||||
|
|||||||
28
docs/api.md
28
docs/api.md
@@ -27,7 +27,13 @@ kubectl -n egress-proxies-operator-system port-forward \
|
|||||||
svc/egress-proxies-operator-controller-manager-discovery-service 8090:8090 &
|
svc/egress-proxies-operator-controller-manager-discovery-service 8090:8090 &
|
||||||
```
|
```
|
||||||
|
|
||||||
All examples below assume `localhost:8090` via that port-forward.
|
Set `BASE_URL` to wherever you reach the API; all examples below use it:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export BASE_URL=localhost:8090 # via the port-forward above
|
||||||
|
# or, from inside the cluster:
|
||||||
|
# export BASE_URL=http://egress-proxies-operator-controller-manager-discovery-service.egress-proxies-operator-system.svc.cluster.local:8090
|
||||||
|
```
|
||||||
|
|
||||||
## Authentication
|
## Authentication
|
||||||
|
|
||||||
@@ -46,7 +52,7 @@ Send the token on every request:
|
|||||||
|
|
||||||
```sh
|
```sh
|
||||||
export TOKEN=<the token>
|
export TOKEN=<the token>
|
||||||
curl -s -H "Authorization: Bearer $TOKEN" localhost:8090/v1/proxies | jq
|
curl -s -H "Authorization: Bearer $TOKEN" "$BASE_URL/v1/proxies" | jq
|
||||||
```
|
```
|
||||||
|
|
||||||
A missing or wrong token gets `401 {"error":"unauthorized",...}`.
|
A missing or wrong token gets `401 {"error":"unauthorized",...}`.
|
||||||
@@ -86,7 +92,7 @@ The shipped Deployment passes none of these flags, so the defaults apply.
|
|||||||
Liveness check. Unauthenticated, always `200` with body `ok`.
|
Liveness check. Unauthenticated, always `200` with body `ok`.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
curl -s localhost:8090/healthz
|
curl -s "$BASE_URL/healthz"
|
||||||
```
|
```
|
||||||
|
|
||||||
### `GET /v1/proxies` — list proxies
|
### `GET /v1/proxies` — list proxies
|
||||||
@@ -101,13 +107,13 @@ Query parameters (all optional):
|
|||||||
List everything:
|
List everything:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
curl -s localhost:8090/v1/proxies | jq
|
curl -s "$BASE_URL/v1/proxies" | jq
|
||||||
```
|
```
|
||||||
|
|
||||||
List healthy proxies in a given geo:
|
List healthy proxies in a given geo:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
curl -s 'localhost:8090/v1/proxies?healthy=true&attr.geo=eu' | jq
|
curl -s "$BASE_URL/v1/proxies?healthy=true&attr.geo=eu" | jq
|
||||||
```
|
```
|
||||||
|
|
||||||
Response — `200`, proxies sorted by `id`, an empty match is `200` with
|
Response — `200`, proxies sorted by `id`, an empty match is `200` with
|
||||||
@@ -168,7 +174,7 @@ Request body (every field optional; `{}` is valid):
|
|||||||
| `target` | none | The site you intend to crawl; enables per-target cooldowns (see below) |
|
| `target` | none | The site you intend to crawl; enables per-target cooldowns (see below) |
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
curl -s -XPOST localhost:8090/v1/leases \
|
curl -s -XPOST "$BASE_URL/v1/leases" \
|
||||||
-d '{"selector":{"geo":"eu"},"ttlSeconds":300,"target":"example.com"}' | jq
|
-d '{"selector":{"geo":"eu"},"ttlSeconds":300,"target":"example.com"}' | jq
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -228,7 +234,7 @@ Frees the lease's capacity slot immediately. Idempotent: always `204`,
|
|||||||
including for unknown or already-expired lease IDs.
|
including for unknown or already-expired lease IDs.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
curl -si -XDELETE localhost:8090/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A
|
curl -si -XDELETE "$BASE_URL/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A"
|
||||||
```
|
```
|
||||||
|
|
||||||
### `POST /v1/leases/{id}/report` — report an outcome
|
### `POST /v1/leases/{id}/report` — report an outcome
|
||||||
@@ -248,7 +254,7 @@ Request body:
|
|||||||
| `target` | optional | Which site produced the result; falls back to the lease's `target`, then to global |
|
| `target` | optional | Which site produced the result; falls back to the lease's `target`, then to global |
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
curl -si -XPOST localhost:8090/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A/report \
|
curl -si -XPOST "$BASE_URL/v1/leases/P3X6HHQTPCM5UTGVGE3B5UPS3A/report" \
|
||||||
-d '{"result":"rate_limited","target":"example.com"}'
|
-d '{"result":"rate_limited","target":"example.com"}'
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -287,7 +293,7 @@ Cooldowns are keyed by **(proxy, target)**:
|
|||||||
|
|
||||||
```sh
|
```sh
|
||||||
# 1. Acquire a lease for crawling example.com through an EU proxy
|
# 1. Acquire a lease for crawling example.com through an EU proxy
|
||||||
LEASE=$(curl -s -XPOST localhost:8090/v1/leases \
|
LEASE=$(curl -s -XPOST "$BASE_URL/v1/leases" \
|
||||||
-H "Authorization: Bearer $TOKEN" \
|
-H "Authorization: Bearer $TOKEN" \
|
||||||
-d '{"selector":{"geo":"eu"},"ttlSeconds":600,"target":"example.com"}')
|
-d '{"selector":{"geo":"eu"},"ttlSeconds":600,"target":"example.com"}')
|
||||||
LEASE_ID=$(echo "$LEASE" | jq -r .leaseID)
|
LEASE_ID=$(echo "$LEASE" | jq -r .leaseID)
|
||||||
@@ -298,12 +304,12 @@ curl -x "http://$PROXY" https://example.com/some/page
|
|||||||
|
|
||||||
# 3. Got a 429? Report it — example.com-bound leases will avoid this
|
# 3. Got a 429? Report it — example.com-bound leases will avoid this
|
||||||
# proxy for the next 15 minutes
|
# proxy for the next 15 minutes
|
||||||
curl -s -XPOST "localhost:8090/v1/leases/$LEASE_ID/report" \
|
curl -s -XPOST "$BASE_URL/v1/leases/$LEASE_ID/report" \
|
||||||
-H "Authorization: Bearer $TOKEN" \
|
-H "Authorization: Bearer $TOKEN" \
|
||||||
-d '{"result":"rate_limited","target":"example.com"}'
|
-d '{"result":"rate_limited","target":"example.com"}'
|
||||||
|
|
||||||
# 4. Done early? Release the slot (otherwise the TTL frees it)
|
# 4. Done early? Release the slot (otherwise the TTL frees it)
|
||||||
curl -s -XDELETE "localhost:8090/v1/leases/$LEASE_ID" \
|
curl -s -XDELETE "$BASE_URL/v1/leases/$LEASE_ID" \
|
||||||
-H "Authorization: Bearer $TOKEN"
|
-H "Authorization: Bearer $TOKEN"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
94
docs/demo/create-gcp-proxies.sh
Executable file
94
docs/demo/create-gcp-proxies.sh
Executable file
@@ -0,0 +1,94 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Demo: create N Managed proxies backed by the gcp provider, named
|
||||||
|
# proxy-gcp-demo-1 .. proxy-gcp-demo-N, each in a randomly picked EU zone
|
||||||
|
# so the fleet gets egress IPs from different locations.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./create-gcp-proxies.sh <count>
|
||||||
|
#
|
||||||
|
# Optional environment:
|
||||||
|
# NAMESPACE namespace to create the proxies in (default: current context)
|
||||||
|
# GCP_PROVIDER provider NAME from providers.yaml (default: gcp-eu)
|
||||||
|
# ZONES space-separated zone list to pick from (default: EU zones below)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if [ $# -ne 1 ] || ! [[ "$1" =~ ^[1-9][0-9]*$ ]]; then
|
||||||
|
echo "usage: $(basename "$0") <count> (positive integer)" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
count="$1"
|
||||||
|
provider="${GCP_PROVIDER:-gcp-eu}"
|
||||||
|
|
||||||
|
# GCP zones in the EU where e2-micro is generally available. Override with
|
||||||
|
# ZONES="zone1 zone2 ..." if your project has quota only in some of them.
|
||||||
|
default_zones=(
|
||||||
|
europe-west1-b europe-west1-c europe-west1-d # Belgium
|
||||||
|
europe-west2-a europe-west2-b europe-west2-c # London
|
||||||
|
europe-west3-a europe-west3-b europe-west3-c # Frankfurt
|
||||||
|
europe-west4-a europe-west4-b europe-west4-c # Netherlands
|
||||||
|
europe-west6-a europe-west6-b europe-west6-c # Zurich
|
||||||
|
europe-west8-a europe-west8-b europe-west8-c # Milan
|
||||||
|
europe-west9-a europe-west9-b europe-west9-c # Paris
|
||||||
|
europe-central2-a europe-central2-b europe-central2-c # Warsaw
|
||||||
|
europe-north1-a europe-north1-b europe-north1-c # Finland
|
||||||
|
europe-southwest1-a europe-southwest1-b europe-southwest1-c # Madrid
|
||||||
|
)
|
||||||
|
if [ -n "${ZONES:-}" ]; then
|
||||||
|
read -r -a zones <<< "${ZONES}"
|
||||||
|
else
|
||||||
|
zones=("${default_zones[@]}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! command -v kubectl &> /dev/null; then
|
||||||
|
echo "ERROR: kubectl is required but not installed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
ns_args=()
|
||||||
|
if [ -n "${NAMESPACE:-}" ]; then
|
||||||
|
ns_args=(-n "${NAMESPACE}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
for i in $(seq 1 "${count}"); do
|
||||||
|
zone="${zones[RANDOM % ${#zones[@]}]}"
|
||||||
|
echo "Creating proxy-gcp-demo-${i} in ${zone} ..."
|
||||||
|
kubectl apply "${ns_args[@]}" -f - << EOF
|
||||||
|
apiVersion: crawl.example.com/v1alpha1
|
||||||
|
kind: Proxy
|
||||||
|
metadata:
|
||||||
|
name: proxy-gcp-demo-${i}
|
||||||
|
spec:
|
||||||
|
mode: Managed
|
||||||
|
provider: ${provider} # must match a provider NAME in providers.yaml
|
||||||
|
placement:
|
||||||
|
zone: ${zone}
|
||||||
|
machineType: e2-micro
|
||||||
|
# debian-cloud images have no cloud-init, so spec.cloudInit (passed as
|
||||||
|
# user-data metadata) would be silently ignored there. Ubuntu images do.
|
||||||
|
image: projects/ubuntu-os-cloud/global/images/family/ubuntu-2404-lts-amd64
|
||||||
|
port: 3128
|
||||||
|
cloudInit:
|
||||||
|
inline: |
|
||||||
|
#cloud-config
|
||||||
|
package_update: true
|
||||||
|
packages:
|
||||||
|
- squid
|
||||||
|
write_files:
|
||||||
|
- path: /etc/squid/conf.d/proxy-operator.conf
|
||||||
|
content: |
|
||||||
|
http_access allow all
|
||||||
|
via off
|
||||||
|
forwarded_for off
|
||||||
|
runcmd:
|
||||||
|
- systemctl restart squid
|
||||||
|
attributes:
|
||||||
|
geo: eu
|
||||||
|
zone: ${zone}
|
||||||
|
purpose: crawl
|
||||||
|
EOF
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Created ${count} proxies. VMs take a few minutes to provision and pass"
|
||||||
|
echo "the health check. Watch them come up with:"
|
||||||
|
echo " kubectl get px ${ns_args[*]:-} -w"
|
||||||
48
docs/demo/create-kubernetes-proxies.sh
Executable file
48
docs/demo/create-kubernetes-proxies.sh
Executable file
@@ -0,0 +1,48 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Demo: create N Managed proxies backed by the kubernetes-pod provider,
|
||||||
|
# named proxy-kubernetes-demo-1 .. proxy-kubernetes-demo-N. Pods share the
|
||||||
|
# cluster's egress IP — this exercises the full lifecycle, not distinct
|
||||||
|
# egress paths (use create-gcp-proxies.sh for that).
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./create-kubernetes-proxies.sh <count>
|
||||||
|
#
|
||||||
|
# Optional environment:
|
||||||
|
# NAMESPACE namespace to create the proxies in (default: current context)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if [ $# -ne 1 ] || ! [[ "$1" =~ ^[1-9][0-9]*$ ]]; then
|
||||||
|
echo "usage: $(basename "$0") <count> (positive integer)" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
count="$1"
|
||||||
|
|
||||||
|
if ! command -v kubectl &> /dev/null; then
|
||||||
|
echo "ERROR: kubectl is required but not installed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
ns_args=()
|
||||||
|
if [ -n "${NAMESPACE:-}" ]; then
|
||||||
|
ns_args=(-n "${NAMESPACE}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
for i in $(seq 1 "${count}"); do
|
||||||
|
echo "Creating proxy-kubernetes-demo-${i} ..."
|
||||||
|
kubectl apply "${ns_args[@]}" -f - << EOF
|
||||||
|
apiVersion: crawl.example.com/v1alpha1
|
||||||
|
kind: Proxy
|
||||||
|
metadata:
|
||||||
|
name: proxy-kubernetes-demo-${i}
|
||||||
|
spec:
|
||||||
|
mode: Managed
|
||||||
|
provider: kubernetes
|
||||||
|
attributes:
|
||||||
|
geo: local
|
||||||
|
purpose: crawl
|
||||||
|
EOF
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Created ${count} proxies. Watch them come up with:"
|
||||||
|
echo " kubectl get px ${ns_args[*]:-} -w"
|
||||||
67
docs/demo/run-demo.sh
Executable file
67
docs/demo/run-demo.sh
Executable file
@@ -0,0 +1,67 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Demo driver: opens a tmux session with a 2x2 pane grid:
|
||||||
|
#
|
||||||
|
# top-left: show-egress-ips-table.sh in a 10s loop, run inside the
|
||||||
|
# netshoot pod against the in-cluster discovery Service
|
||||||
|
# top-right: watch -n3 kubectl get px
|
||||||
|
# bottom-left: create-kubernetes-proxies.sh <count>
|
||||||
|
# bottom-right: create-gcp-proxies.sh <count>
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./run-demo.sh
|
||||||
|
#
|
||||||
|
# Optional environment:
|
||||||
|
# COUNT proxies each create script makes (default: 4)
|
||||||
|
# SESSION tmux session name (default: proxy-demo; an existing
|
||||||
|
# session with this name is killed and recreated)
|
||||||
|
# NETSHOOT_POD pod to exec into for the egress-IP loop (default: netshoot)
|
||||||
|
# DEMO_DIR where the demo scripts live (default: this script's dir)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
COUNT="${COUNT:-4}"
|
||||||
|
SESSION="${SESSION:-proxy-demo}"
|
||||||
|
NETSHOOT_POD="${NETSHOOT_POD:-netshoot}"
|
||||||
|
DEMO_DIR="${DEMO_DIR:-$(cd "$(dirname "$0")" && pwd)}"
|
||||||
|
BASE_URL="http://egress-proxies-operator-controller-manager-discovery-service.egress-proxies-operator-system.svc.cluster.local:8090"
|
||||||
|
|
||||||
|
for tool in tmux kubectl; do
|
||||||
|
if ! command -v "${tool}" &> /dev/null; then
|
||||||
|
echo "ERROR: ${tool} is required but not installed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if ! kubectl get pod "${NETSHOOT_POD}" &> /dev/null; then
|
||||||
|
echo "ERROR: pod ${NETSHOOT_POD} not found — start one with:" >&2
|
||||||
|
echo " kubectl run netshoot --image=nicolaka/netshoot -- sleep infinity" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Copying show-egress-ips-table.sh into pod ${NETSHOOT_POD} ..."
|
||||||
|
kubectl cp "${DEMO_DIR}/show-egress-ips-table.sh" "${NETSHOOT_POD}:/tmp/show-egress-ips-table.sh"
|
||||||
|
|
||||||
|
if tmux has-session -t "${SESSION}" 2> /dev/null; then
|
||||||
|
echo "Killing existing tmux session ${SESSION}"
|
||||||
|
tmux kill-session -t "${SESSION}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 2x2 grid: after these splits pane indexes are 0 top-left, 1 top-right,
|
||||||
|
# 2 bottom-left, 3 bottom-right; tiled layout evens them into quarters.
|
||||||
|
tmux new-session -d -s "${SESSION}"
|
||||||
|
tmux split-window -h -t "${SESSION}:0"
|
||||||
|
tmux split-window -v -t "${SESSION}:0.0"
|
||||||
|
tmux split-window -v -t "${SESSION}:0.1"
|
||||||
|
tmux select-layout -t "${SESSION}:0" tiled
|
||||||
|
|
||||||
|
loop_cmd="BASE_URL=${BASE_URL}; while true; do bash /tmp/show-egress-ips-table.sh \"\$BASE_URL\"; echo; sleep 10; done"
|
||||||
|
tmux send-keys -t "${SESSION}:0.0" "kubectl exec -it ${NETSHOOT_POD} -- bash -c '${loop_cmd}'" C-m
|
||||||
|
tmux send-keys -t "${SESSION}:0.1" "watch -n3 kubectl get px" C-m
|
||||||
|
tmux send-keys -t "${SESSION}:0.2" "bash ${DEMO_DIR}/create-kubernetes-proxies.sh ${COUNT}" C-m
|
||||||
|
tmux send-keys -t "${SESSION}:0.3" "bash ${DEMO_DIR}/create-gcp-proxies.sh ${COUNT}" C-m
|
||||||
|
|
||||||
|
tmux select-pane -t "${SESSION}:0.2"
|
||||||
|
if [ -n "${TMUX:-}" ]; then
|
||||||
|
tmux switch-client -t "${SESSION}"
|
||||||
|
else
|
||||||
|
tmux attach-session -t "${SESSION}"
|
||||||
|
fi
|
||||||
75
docs/demo/show-egress-ips-table.sh
Executable file
75
docs/demo/show-egress-ips-table.sh
Executable file
@@ -0,0 +1,75 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Demo: condensed-table variant of show-egress-ips.sh, made to fit a small
|
||||||
|
# tmux pane. One line per proxy: which proxy the request goes through, its
|
||||||
|
# endpoint, location (zone/geo attribute), and the egress IP the IP-echo
|
||||||
|
# site saw — or unhealthy/FAILED.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./show-egress-ips-table.sh <BASE_URL> e.g. ./show-egress-ips-table.sh localhost:8090
|
||||||
|
#
|
||||||
|
# Optional environment:
|
||||||
|
# TOKEN bearer token for the discovery API (see docs/api.md)
|
||||||
|
# IP_ECHO_URL site that returns the caller's IP as JSON with an "ip" field
|
||||||
|
# (default: https://api.ipify.org?format=json)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if [ $# -ne 1 ]; then
|
||||||
|
echo "usage: $(basename "$0") <BASE_URL> (e.g. localhost:8090)" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
BASE_URL="$1"
|
||||||
|
IP_ECHO_URL="${IP_ECHO_URL:-https://api.ipify.org?format=json}"
|
||||||
|
|
||||||
|
for tool in curl jq; do
|
||||||
|
if ! command -v "${tool}" &> /dev/null; then
|
||||||
|
echo "ERROR: ${tool} is required but not installed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
auth_args=()
|
||||||
|
if [ -n "${TOKEN:-}" ]; then
|
||||||
|
auth_args=(-H "Authorization: Bearer ${TOKEN}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! proxies_json=$(curl -sS --fail "${auth_args[@]}" "${BASE_URL}/v1/proxies") \
|
||||||
|
|| ! echo "${proxies_json}" | jq -e . > /dev/null 2>&1; then
|
||||||
|
echo "ERROR: could not fetch proxy list from ${BASE_URL}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
total=$(echo "${proxies_json}" | jq -r '.count')
|
||||||
|
fmt="%-31s %-21s %-21s %s\n"
|
||||||
|
|
||||||
|
echo "${total} proxies @ $(date +%H:%M:%S)"
|
||||||
|
# shellcheck disable=SC2059
|
||||||
|
printf "${fmt}" "PROXY" "ENDPOINT" "LOCATION" "EGRESS-IP"
|
||||||
|
|
||||||
|
probed=0
|
||||||
|
skipped=0
|
||||||
|
failed=0
|
||||||
|
while IFS= read -r proxy; do
|
||||||
|
id=$(echo "${proxy}" | jq -r '.id')
|
||||||
|
endpoint=$(echo "${proxy}" | jq -r '"\(.ip):\(.port)"')
|
||||||
|
location=$(echo "${proxy}" | jq -r '.attributes.zone // .attributes.geo // "-"')
|
||||||
|
healthy=$(echo "${proxy}" | jq -r '.healthy')
|
||||||
|
|
||||||
|
if [ "${healthy}" != "true" ]; then
|
||||||
|
# shellcheck disable=SC2059
|
||||||
|
printf "${fmt}" "${id}" "${endpoint}" "${location}" "(unhealthy)"
|
||||||
|
skipped=$((skipped + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
if response=$(curl -sS --max-time 10 -x "http://${endpoint}" "${IP_ECHO_URL}" 2> /dev/null); then
|
||||||
|
egress=$(echo "${response}" | jq -r '.ip // "?"' 2> /dev/null || echo "?")
|
||||||
|
probed=$((probed + 1))
|
||||||
|
else
|
||||||
|
egress="FAILED"
|
||||||
|
failed=$((failed + 1))
|
||||||
|
fi
|
||||||
|
# shellcheck disable=SC2059
|
||||||
|
printf "${fmt}" "${id}" "${endpoint}" "${location}" "${egress}"
|
||||||
|
done < <(echo "${proxies_json}" | jq -c '.proxies[]')
|
||||||
|
|
||||||
|
echo "-- ${probed} probed, ${skipped} unhealthy, ${failed} failed --"
|
||||||
74
docs/demo/show-egress-ips.sh
Executable file
74
docs/demo/show-egress-ips.sh
Executable file
@@ -0,0 +1,74 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Demo: list proxies from the discovery API and show the egress IP each
|
||||||
|
# healthy one provides, by calling an IP-echo site through it.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./show-egress-ips.sh <BASE_URL> e.g. ./show-egress-ips.sh localhost:8090
|
||||||
|
#
|
||||||
|
# Optional environment:
|
||||||
|
# TOKEN bearer token for the discovery API (see docs/api.md)
|
||||||
|
# IP_ECHO_URL site that returns the caller's IP as JSON
|
||||||
|
# (default: https://api.ipify.org?format=json)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if [ $# -ne 1 ]; then
|
||||||
|
echo "usage: $(basename "$0") <BASE_URL> (e.g. localhost:8090)" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
BASE_URL="$1"
|
||||||
|
IP_ECHO_URL="${IP_ECHO_URL:-https://api.ipify.org?format=json}"
|
||||||
|
|
||||||
|
for tool in curl jq; do
|
||||||
|
if ! command -v "${tool}" &> /dev/null; then
|
||||||
|
echo "ERROR: ${tool} is required but not installed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
auth_args=()
|
||||||
|
if [ -n "${TOKEN:-}" ]; then
|
||||||
|
auth_args=(-H "Authorization: Bearer ${TOKEN}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Fetching proxies from ${BASE_URL}/v1/proxies ..."
|
||||||
|
if ! proxies_json=$(curl -sS --fail "${auth_args[@]}" "${BASE_URL}/v1/proxies"); then
|
||||||
|
echo "ERROR: could not fetch proxy list from ${BASE_URL}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if ! echo "${proxies_json}" | jq -e . > /dev/null; then
|
||||||
|
echo "ERROR: response from ${BASE_URL}/v1/proxies is not valid JSON" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
total=$(echo "${proxies_json}" | jq -r '.count')
|
||||||
|
echo "Found ${total} proxies"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
probed=0
|
||||||
|
skipped=0
|
||||||
|
failed=0
|
||||||
|
while IFS= read -r proxy; do
|
||||||
|
id=$(echo "${proxy}" | jq -r '.id')
|
||||||
|
ip=$(echo "${proxy}" | jq -r '.ip')
|
||||||
|
port=$(echo "${proxy}" | jq -r '.port')
|
||||||
|
healthy=$(echo "${proxy}" | jq -r '.healthy')
|
||||||
|
|
||||||
|
if [ "${healthy}" != "true" ]; then
|
||||||
|
echo "--- skipping ${id} (unhealthy) ---"
|
||||||
|
echo ""
|
||||||
|
skipped=$((skipped + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "=== via ${id} — http://${ip}:${port} ==="
|
||||||
|
if response=$(curl -sS --max-time 10 -x "http://${ip}:${port}" "${IP_ECHO_URL}"); then
|
||||||
|
echo "${response}" | jq . 2> /dev/null || echo "${response}"
|
||||||
|
probed=$((probed + 1))
|
||||||
|
else
|
||||||
|
echo "WARNING: request through ${id} failed" >&2
|
||||||
|
failed=$((failed + 1))
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
done < <(echo "${proxies_json}" | jq -c '.proxies[]')
|
||||||
|
|
||||||
|
echo "Done: ${total} proxies — ${probed} probed, ${skipped} skipped (unhealthy), ${failed} failed"
|
||||||
55
docs/plans-executions/2026-08-11-2220-demo-scripts.md
Normal file
55
docs/plans-executions/2026-08-11-2220-demo-scripts.md
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# Execution: Demo scripts
|
||||||
|
|
||||||
|
Plan: [2026-08-11-2220-demo-scripts.md](../plans/2026-08-11-2220-demo-scripts.md)
|
||||||
|
|
||||||
|
- [x] Step 0 — Branch `feat/demo-scripts` + plan commit
|
||||||
|
- [x] Step 1 — `docs/demo/show-egress-ips.sh`
|
||||||
|
- [x] Extra (added iteratively, not in the original plan) — proxy-creation scripts
|
||||||
|
|
||||||
|
## Step 0 + Step 1
|
||||||
|
|
||||||
|
Branched off `main`, committed the plan alone, then wrote
|
||||||
|
`docs/demo/show-egress-ips.sh`: takes `BASE_URL` as its argument, lists
|
||||||
|
`/v1/proxies` (bearer auth via optional `TOKEN` env), probes each healthy
|
||||||
|
proxy with `curl -x http://ip:port` against an IP-echo site
|
||||||
|
(`IP_ECHO_URL`, default ipify JSON), banners which proxy each request goes
|
||||||
|
through, skips unhealthy ones, and ends with a probed/skipped/failed
|
||||||
|
summary. Per user request the script was left uncommitted for iteration
|
||||||
|
and no verification beyond `bash -n` was run.
|
||||||
|
|
||||||
|
## Extra — create-kubernetes-proxies.sh, create-gcp-proxies.sh
|
||||||
|
|
||||||
|
Added on the same branch before the first commit:
|
||||||
|
|
||||||
|
- `create-kubernetes-proxies.sh <count>` — creates
|
||||||
|
`proxy-kubernetes-demo-1..N` with the kubernetes provider, spec taken
|
||||||
|
from `config/samples/proxy_kubernetes.yaml`, applied via
|
||||||
|
`kubectl apply -f -` heredocs. Optional `NAMESPACE` env.
|
||||||
|
- `create-gcp-proxies.sh <count>` — creates `proxy-gcp-demo-1..N` from the
|
||||||
|
user-supplied gcp-eu manifest (e2-micro, Ubuntu 24.04, Squid
|
||||||
|
cloud-init), each with a zone picked randomly from a hardcoded list of
|
||||||
|
27 EU zones so the fleet gets egress IPs from different locations.
|
||||||
|
Env overrides: `ZONES`, `GCP_PROVIDER` (default `gcp-eu`), `NAMESPACE`.
|
||||||
|
|
||||||
|
## Extra — run-demo.sh, show-egress-ips-table.sh
|
||||||
|
|
||||||
|
Second round of iterative additions:
|
||||||
|
|
||||||
|
- `run-demo.sh` — tmux demo driver: 2x2 tiled grid with the egress-IP
|
||||||
|
table looping every 10s inside a netshoot pod (script `kubectl cp`'d
|
||||||
|
into the pod, `BASE_URL` set to the in-cluster Service FQDN),
|
||||||
|
`watch -n3 kubectl get px`, and both create scripts auto-running with
|
||||||
|
`COUNT` proxies each (default 4). Env knobs: `COUNT`, `SESSION`,
|
||||||
|
`NETSHOOT_POD`, `DEMO_DIR` (defaults to the script's own dir).
|
||||||
|
- `show-egress-ips-table.sh` — condensed one-line-per-proxy variant of
|
||||||
|
`show-egress-ips.sh` sized for a tmux pane: PROXY / ENDPOINT /
|
||||||
|
LOCATION (zone→geo attribute fallback) / EGRESS-IP columns, with
|
||||||
|
`(unhealthy)` and `FAILED` inline instead of verbose output. The
|
||||||
|
verbose script stays for standalone use; the demo driver uses the
|
||||||
|
table variant.
|
||||||
|
|
||||||
|
Worth noting: beyond the user's sample manifest, the gcp script also
|
||||||
|
writes the picked zone into `attributes.zone`, so the discovery API
|
||||||
|
exposes each proxy's location and leases can select on it. The zone list
|
||||||
|
is static — if a project lacks quota in some region, `ZONES` narrows the
|
||||||
|
pool; nothing validates zones against the live project.
|
||||||
33
docs/plans/2026-08-11-2220-demo-scripts.md
Normal file
33
docs/plans/2026-08-11-2220-demo-scripts.md
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
# Plan: Demo script — egress IP check through each healthy proxy
|
||||||
|
**Created:** 2026-08-11 22:20
|
||||||
|
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
First of a planned series of demo scripts under `docs/demo/`. This one showcases the discovery API end to end without leases: list proxies from `$BASE_URL/v1/proxies`, and for each **healthy** one, call an IP-echo site through it (`curl -x`) to show the egress IP that proxy provides — clearly labeling which proxy each request goes through. The user explicitly asked to start by switching to a new branch. The script will be iterated on: **write it but do not commit it** — the user wants to add things to it before anything is committed.
|
||||||
|
|
||||||
|
## Steps
|
||||||
|
|
||||||
|
### Step 0 — Branch
|
||||||
|
|
||||||
|
Create `feat/demo-scripts` off `main`. Commit only the plan copy (`docs/plans/<timestamp>-demo-scripts.md`, timestamp via `date "+%Y-%m-%d-%H%M"`) per CLAUDE.md — nothing else gets committed this round.
|
||||||
|
|
||||||
|
### Step 1 — `docs/demo/show-egress-ips.sh` (new file, executable, left uncommitted)
|
||||||
|
|
||||||
|
Style: match [.devcontainer/post-install.sh](.devcontainer/post-install.sh) — `#!/bin/bash`, `set -euo pipefail`, `ERROR:`/`WARNING:` messages, `${VAR}` braces.
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
1. **Args/env:** `BASE_URL` is `$1` (required; missing → usage text + exit 1, e.g. `usage: show-egress-ips.sh <BASE_URL> (e.g. localhost:8090)`). Optional env: `TOKEN` (bearer token, same name docs/api.md uses; when set, send `Authorization: Bearer $TOKEN`), `IP_ECHO_URL` (default `https://api.ipify.org?format=json` — returns `{"ip":"..."}`).
|
||||||
|
2. **Dependency check:** `command -v curl`, `command -v jq` → `ERROR` + exit 1 if missing.
|
||||||
|
3. **Fetch** `"$BASE_URL/v1/proxies"` once (no server-side `healthy` filter — fetch all so unhealthy ones can be shown as skipped, which makes the demo more informative). Fail with a clear error if curl or JSON parsing fails.
|
||||||
|
4. **Iterate** proxies with `jq -c '.proxies[]'`; for each, extract `id`, `ip`, `port`, `healthy`:
|
||||||
|
- unhealthy → print `--- skipping <id> (unhealthy) ---`
|
||||||
|
- healthy → print a clear banner naming the proxy before the request, e.g. `=== via <id> — http://<ip>:<port> ===`, then `curl -sS --max-time 10 -x "http://${ip}:${port}" "$IP_ECHO_URL"`; print the JSON response. A failed probe prints `WARNING: request through <id> failed` and continues (guard so `set -e` doesn't kill the loop).
|
||||||
|
5. Finish with a one-line summary: N proxies, M probed, K skipped/failed.
|
||||||
|
|
||||||
|
Reference for API shapes: [docs/api.md](docs/api.md) (`proxies[].id/ip/port/healthy`; `curl -x http://ip:port` usage is already documented there and in README).
|
||||||
|
|
||||||
|
### Deliberately deferred (user will iterate on the script first)
|
||||||
|
|
||||||
|
- No commit of the script, no push beyond the plan commit, no MR, no execution summary, no CHANGELOG — all wait until the user says the script (or script set) is ready.
|
||||||
Reference in New Issue
Block a user