Compare commits
3 Commits
0d68111bc2
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 5ccd317bec | |||
| 6ff9eebefc | |||
| f6b67006dc |
@@ -83,7 +83,17 @@
|
|||||||
"Bash(kubectl -n egress-proxies-operator-system get pods -o wide)",
|
"Bash(kubectl -n egress-proxies-operator-system get pods -o wide)",
|
||||||
"Bash(kubectl -n egress-proxies-operator-system get deploy egress-proxies-operator-controller-manager -o jsonpath='{.spec.template.spec.containers[0].args}')",
|
"Bash(kubectl -n egress-proxies-operator-system get deploy egress-proxies-operator-controller-manager -o jsonpath='{.spec.template.spec.containers[0].args}')",
|
||||||
"Bash(kubectl -n egress-proxies-operator-system logs deploy/egress-proxies-operator-controller-manager)",
|
"Bash(kubectl -n egress-proxies-operator-system logs deploy/egress-proxies-operator-controller-manager)",
|
||||||
"Bash(python3 -c \"import json; d=json.load\\(open\\('docs/deploy/sa_key.json'\\)\\); print\\(d.get\\('type'\\), d.get\\('client_email'\\)\\)\")"
|
"Bash(python3 -c \"import json; d=json.load\\(open\\('docs/deploy/sa_key.json'\\)\\); print\\(d.get\\('type'\\), d.get\\('client_email'\\)\\)\")",
|
||||||
|
"Bash(tea pr *)",
|
||||||
|
"Bash(git worktree *)",
|
||||||
|
"Bash(python3 -c \"import yaml; yaml.safe_load\\(open\\('.gitea/workflows/build.yaml'\\)\\); print\\('YAML OK'\\)\")",
|
||||||
|
"Bash(ruby -ryaml -e \"YAML.load_file\\('.gitea/workflows/build.yaml'\\); puts 'YAML OK'\")",
|
||||||
|
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator status --short --branch)",
|
||||||
|
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator log --oneline -1)",
|
||||||
|
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator tag 0.01)",
|
||||||
|
"Bash(git -C /Users/jan.novak/srv/go/egress-proxies-operator push origin 0.01)",
|
||||||
|
"Bash(chmod +x docs/demo/run-demo.sh)",
|
||||||
|
"Bash(bash -n docs/demo/run-demo.sh)"
|
||||||
],
|
],
|
||||||
"additionalDirectories": [
|
"additionalDirectories": [
|
||||||
"/Users/jan.novak/srv/go/egress-proxies-operator/.claude",
|
"/Users/jan.novak/srv/go/egress-proxies-operator/.claude",
|
||||||
|
|||||||
56
README.md
56
README.md
@@ -136,6 +136,62 @@ Cloud-init from a Secret: the Secret **must** carry the label
|
|||||||
labelled Secrets, so an unlabelled one is invisible (the Proxy reports
|
labelled Secrets, so an unlabelled one is invisible (the Proxy reports
|
||||||
`CloudInitError`). Rotating the Secret's content triggers VM replacement.
|
`CloudInitError`). Rotating the Secret's content triggers VM replacement.
|
||||||
|
|
||||||
|
## Providers
|
||||||
|
|
||||||
|
`Managed` proxies are provisioned by a provider — a small compute backend
|
||||||
|
behind one minimal interface. Providers are configured in the
|
||||||
|
`--providers-config` YAML file as **named instances**: `spec.provider` on a
|
||||||
|
Proxy refers to an entry's `name`, not its `type`, so `gcp-eu` and `gcp-us`
|
||||||
|
can be two differently-configured instances of the same `gcp` type (see
|
||||||
|
[config/samples/providers-config.yaml](config/samples/providers-config.yaml)).
|
||||||
|
|
||||||
|
### Implemented providers
|
||||||
|
|
||||||
|
| Type | Creates | Per-instance config | Notes |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `kubernetes` | A real Squid pod (`ubuntu/squid:6.6-24.04_edge` by default) in the same cluster the operator runs in | `image` (optional) | Needs no cloud account — local dev, CI, and the kind quickstart. Pods share the cluster's egress IP, so it exercises the full lifecycle but not distinct egress paths. |
|
||||||
|
| `gcp` | A Compute Engine VM with an ephemeral external IP | `project` (required), `network`, `networkTag`, `diskSizeGb` | The real egress fleet. Deliberately uses only four API calls (Insert / Get / Delete / AggregatedList), all fire-and-forget: `Create` returns as soon as the operation is submitted and the reconciler discovers progress by polling `Get`. Auth is Application Default Credentials — workload identity in-cluster, `gcloud` ADC locally; no key-file plumbing. |
|
||||||
|
|
||||||
|
### Adding a provider
|
||||||
|
|
||||||
|
A new backend (Hetzner, AWS, ...) is four pieces; the contract lives in
|
||||||
|
[internal/provider/provider.go](internal/provider/provider.go):
|
||||||
|
|
||||||
|
1. **Implement the 4-method `Provider` interface** in a new
|
||||||
|
`internal/provider/<type>/` package:
|
||||||
|
- `Create` submits and returns — it never blocks until the VM runs, and
|
||||||
|
must be idempotent keyed on `req.Name` (a deterministic name derived
|
||||||
|
from the Proxy's UID), so a repeat call after a crash finds the
|
||||||
|
existing instance instead of duplicating it.
|
||||||
|
- `Get` returns `provider.ErrNotFound` as a *normal* outcome — the
|
||||||
|
reconciler branches on it for replacement and adoption, so don't
|
||||||
|
treat it as exceptional.
|
||||||
|
- `Delete` is idempotent: deleting an already-gone instance is not an
|
||||||
|
error.
|
||||||
|
- `ListByTag` returns every instance the operator ever tagged, for
|
||||||
|
orphan GC.
|
||||||
|
2. **Tag every created resource** with `LabelManaged=true` and
|
||||||
|
`LabelUID=<Proxy UID>`, and report `CreatedAt` — orphan GC relies on
|
||||||
|
all three to find owned resources and skip in-flight creates.
|
||||||
|
3. **Classify every returned error** with `provider.Wrap` into the
|
||||||
|
four-sentinel taxonomy in
|
||||||
|
[internal/provider/errors.go](internal/provider/errors.go)
|
||||||
|
(`ErrNotFound` / `ErrQuotaExceeded` / `ErrTransient` / `ErrPermanent`)
|
||||||
|
— the reconciler decides retry, slow backoff, or latching `Failed`
|
||||||
|
purely from that classification, never from provider-specific types.
|
||||||
|
4. **Wire it up**: add a type-specific config block in
|
||||||
|
[internal/provider/config.go](internal/provider/config.go), and
|
||||||
|
register the constructor in the builtins map in
|
||||||
|
[cmd/main.go](cmd/main.go) (`"<type>": <pkg>.New`). The registry
|
||||||
|
([internal/provider/registry](internal/provider/registry/registry.go))
|
||||||
|
handles named instances, and the metrics wrapper is applied
|
||||||
|
automatically.
|
||||||
|
|
||||||
|
Test against a fake API seam rather than the real cloud — see the
|
||||||
|
`instancesAPI` seam in
|
||||||
|
[internal/provider/gcp/gcp.go](internal/provider/gcp/gcp.go) for the
|
||||||
|
pattern.
|
||||||
|
|
||||||
## Caveats — read these two
|
## Caveats — read these two
|
||||||
|
|
||||||
**Changing a proxy changes its IP.** Proxies are immutable cattle: editing
|
**Changing a proxy changes its IP.** Proxies are immutable cattle: editing
|
||||||
|
|||||||
67
docs/demo/run-demo.sh
Executable file
67
docs/demo/run-demo.sh
Executable file
@@ -0,0 +1,67 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Demo driver: opens a tmux session with a 2x2 pane grid:
|
||||||
|
#
|
||||||
|
# top-left: show-egress-ips-table.sh in a 10s loop, run inside the
|
||||||
|
# netshoot pod against the in-cluster discovery Service
|
||||||
|
# top-right: watch -n3 kubectl get px
|
||||||
|
# bottom-left: create-kubernetes-proxies.sh <count>
|
||||||
|
# bottom-right: create-gcp-proxies.sh <count>
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./run-demo.sh
|
||||||
|
#
|
||||||
|
# Optional environment:
|
||||||
|
# COUNT proxies each create script makes (default: 4)
|
||||||
|
# SESSION tmux session name (default: proxy-demo; an existing
|
||||||
|
# session with this name is killed and recreated)
|
||||||
|
# NETSHOOT_POD pod to exec into for the egress-IP loop (default: netshoot)
|
||||||
|
# DEMO_DIR where the demo scripts live (default: this script's dir)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
COUNT="${COUNT:-4}"
|
||||||
|
SESSION="${SESSION:-proxy-demo}"
|
||||||
|
NETSHOOT_POD="${NETSHOOT_POD:-netshoot}"
|
||||||
|
DEMO_DIR="${DEMO_DIR:-$(cd "$(dirname "$0")" && pwd)}"
|
||||||
|
BASE_URL="http://egress-proxies-operator-controller-manager-discovery-service.egress-proxies-operator-system.svc.cluster.local:8090"
|
||||||
|
|
||||||
|
for tool in tmux kubectl; do
|
||||||
|
if ! command -v "${tool}" &> /dev/null; then
|
||||||
|
echo "ERROR: ${tool} is required but not installed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if ! kubectl get pod "${NETSHOOT_POD}" &> /dev/null; then
|
||||||
|
echo "ERROR: pod ${NETSHOOT_POD} not found — start one with:" >&2
|
||||||
|
echo " kubectl run netshoot --image=nicolaka/netshoot -- sleep infinity" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Copying show-egress-ips-table.sh into pod ${NETSHOOT_POD} ..."
|
||||||
|
kubectl cp "${DEMO_DIR}/show-egress-ips-table.sh" "${NETSHOOT_POD}:/tmp/show-egress-ips-table.sh"
|
||||||
|
|
||||||
|
if tmux has-session -t "${SESSION}" 2> /dev/null; then
|
||||||
|
echo "Killing existing tmux session ${SESSION}"
|
||||||
|
tmux kill-session -t "${SESSION}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# 2x2 grid: after these splits pane indexes are 0 top-left, 1 top-right,
|
||||||
|
# 2 bottom-left, 3 bottom-right; tiled layout evens them into quarters.
|
||||||
|
tmux new-session -d -s "${SESSION}"
|
||||||
|
tmux split-window -h -t "${SESSION}:0"
|
||||||
|
tmux split-window -v -t "${SESSION}:0.0"
|
||||||
|
tmux split-window -v -t "${SESSION}:0.1"
|
||||||
|
tmux select-layout -t "${SESSION}:0" tiled
|
||||||
|
|
||||||
|
loop_cmd="BASE_URL=${BASE_URL}; while true; do bash /tmp/show-egress-ips-table.sh \"\$BASE_URL\"; echo; sleep 10; done"
|
||||||
|
tmux send-keys -t "${SESSION}:0.0" "kubectl exec -it ${NETSHOOT_POD} -- bash -c '${loop_cmd}'" C-m
|
||||||
|
tmux send-keys -t "${SESSION}:0.1" "watch -n3 kubectl get px" C-m
|
||||||
|
tmux send-keys -t "${SESSION}:0.2" "bash ${DEMO_DIR}/create-kubernetes-proxies.sh ${COUNT}" C-m
|
||||||
|
tmux send-keys -t "${SESSION}:0.3" "bash ${DEMO_DIR}/create-gcp-proxies.sh ${COUNT}" C-m
|
||||||
|
|
||||||
|
tmux select-pane -t "${SESSION}:0.2"
|
||||||
|
if [ -n "${TMUX:-}" ]; then
|
||||||
|
tmux switch-client -t "${SESSION}"
|
||||||
|
else
|
||||||
|
tmux attach-session -t "${SESSION}"
|
||||||
|
fi
|
||||||
75
docs/demo/show-egress-ips-table.sh
Executable file
75
docs/demo/show-egress-ips-table.sh
Executable file
@@ -0,0 +1,75 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Demo: condensed-table variant of show-egress-ips.sh, made to fit a small
|
||||||
|
# tmux pane. One line per proxy: which proxy the request goes through, its
|
||||||
|
# endpoint, location (zone/geo attribute), and the egress IP the IP-echo
|
||||||
|
# site saw — or unhealthy/FAILED.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# ./show-egress-ips-table.sh <BASE_URL> e.g. ./show-egress-ips-table.sh localhost:8090
|
||||||
|
#
|
||||||
|
# Optional environment:
|
||||||
|
# TOKEN bearer token for the discovery API (see docs/api.md)
|
||||||
|
# IP_ECHO_URL site that returns the caller's IP as JSON with an "ip" field
|
||||||
|
# (default: https://api.ipify.org?format=json)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
if [ $# -ne 1 ]; then
|
||||||
|
echo "usage: $(basename "$0") <BASE_URL> (e.g. localhost:8090)" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
BASE_URL="$1"
|
||||||
|
IP_ECHO_URL="${IP_ECHO_URL:-https://api.ipify.org?format=json}"
|
||||||
|
|
||||||
|
for tool in curl jq; do
|
||||||
|
if ! command -v "${tool}" &> /dev/null; then
|
||||||
|
echo "ERROR: ${tool} is required but not installed" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
auth_args=()
|
||||||
|
if [ -n "${TOKEN:-}" ]; then
|
||||||
|
auth_args=(-H "Authorization: Bearer ${TOKEN}")
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! proxies_json=$(curl -sS --fail "${auth_args[@]}" "${BASE_URL}/v1/proxies") \
|
||||||
|
|| ! echo "${proxies_json}" | jq -e . > /dev/null 2>&1; then
|
||||||
|
echo "ERROR: could not fetch proxy list from ${BASE_URL}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
total=$(echo "${proxies_json}" | jq -r '.count')
|
||||||
|
fmt="%-31s %-21s %-21s %s\n"
|
||||||
|
|
||||||
|
echo "${total} proxies @ $(date +%H:%M:%S)"
|
||||||
|
# shellcheck disable=SC2059
|
||||||
|
printf "${fmt}" "PROXY" "ENDPOINT" "LOCATION" "EGRESS-IP"
|
||||||
|
|
||||||
|
probed=0
|
||||||
|
skipped=0
|
||||||
|
failed=0
|
||||||
|
while IFS= read -r proxy; do
|
||||||
|
id=$(echo "${proxy}" | jq -r '.id')
|
||||||
|
endpoint=$(echo "${proxy}" | jq -r '"\(.ip):\(.port)"')
|
||||||
|
location=$(echo "${proxy}" | jq -r '.attributes.zone // .attributes.geo // "-"')
|
||||||
|
healthy=$(echo "${proxy}" | jq -r '.healthy')
|
||||||
|
|
||||||
|
if [ "${healthy}" != "true" ]; then
|
||||||
|
# shellcheck disable=SC2059
|
||||||
|
printf "${fmt}" "${id}" "${endpoint}" "${location}" "(unhealthy)"
|
||||||
|
skipped=$((skipped + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
if response=$(curl -sS --max-time 10 -x "http://${endpoint}" "${IP_ECHO_URL}" 2> /dev/null); then
|
||||||
|
egress=$(echo "${response}" | jq -r '.ip // "?"' 2> /dev/null || echo "?")
|
||||||
|
probed=$((probed + 1))
|
||||||
|
else
|
||||||
|
egress="FAILED"
|
||||||
|
failed=$((failed + 1))
|
||||||
|
fi
|
||||||
|
# shellcheck disable=SC2059
|
||||||
|
printf "${fmt}" "${id}" "${endpoint}" "${location}" "${egress}"
|
||||||
|
done < <(echo "${proxies_json}" | jq -c '.proxies[]')
|
||||||
|
|
||||||
|
echo "-- ${probed} probed, ${skipped} unhealthy, ${failed} failed --"
|
||||||
@@ -31,6 +31,23 @@ Added on the same branch before the first commit:
|
|||||||
27 EU zones so the fleet gets egress IPs from different locations.
|
27 EU zones so the fleet gets egress IPs from different locations.
|
||||||
Env overrides: `ZONES`, `GCP_PROVIDER` (default `gcp-eu`), `NAMESPACE`.
|
Env overrides: `ZONES`, `GCP_PROVIDER` (default `gcp-eu`), `NAMESPACE`.
|
||||||
|
|
||||||
|
## Extra — run-demo.sh, show-egress-ips-table.sh
|
||||||
|
|
||||||
|
Second round of iterative additions:
|
||||||
|
|
||||||
|
- `run-demo.sh` — tmux demo driver: 2x2 tiled grid with the egress-IP
|
||||||
|
table looping every 10s inside a netshoot pod (script `kubectl cp`'d
|
||||||
|
into the pod, `BASE_URL` set to the in-cluster Service FQDN),
|
||||||
|
`watch -n3 kubectl get px`, and both create scripts auto-running with
|
||||||
|
`COUNT` proxies each (default 4). Env knobs: `COUNT`, `SESSION`,
|
||||||
|
`NETSHOOT_POD`, `DEMO_DIR` (defaults to the script's own dir).
|
||||||
|
- `show-egress-ips-table.sh` — condensed one-line-per-proxy variant of
|
||||||
|
`show-egress-ips.sh` sized for a tmux pane: PROXY / ENDPOINT /
|
||||||
|
LOCATION (zone→geo attribute fallback) / EGRESS-IP columns, with
|
||||||
|
`(unhealthy)` and `FAILED` inline instead of verbose output. The
|
||||||
|
verbose script stays for standalone use; the demo driver uses the
|
||||||
|
table variant.
|
||||||
|
|
||||||
Worth noting: beyond the user's sample manifest, the gcp script also
|
Worth noting: beyond the user's sample manifest, the gcp script also
|
||||||
writes the picked zone into `attributes.zone`, so the discovery API
|
writes the picked zone into `attributes.zone`, so the discovery API
|
||||||
exposes each proxy's location and leases can select on it. The zone list
|
exposes each proxy's location and leases can select on it. The zone list
|
||||||
|
|||||||
Reference in New Issue
Block a user