kacerr/egress-proxies-operator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Housekeeping: review docs into docs/, refreshed agent guide and settings

Browse Source 
Four unrelated working-tree changes, none of them touching operator
behaviour:

- review/*.md -> docs/reviews/2026-09-17-kvm2-provider-review/, byte for
  byte, so the kvm2 review sits with the rest of the docs under a name
  that says which review it was.
- AGENTS.md: replaced the generic kubebuilder scaffold guide with what is
  actually true of this repo — the real component layout, the lint/test
  commands that work here, and the behavioural gotchas (replace-on-change
  hashing, the cloud-init Secret label, in-memory leases).
- .claude/settings.json: the allow list still carried /Users/jan.novak
  paths from the macOS checkout; re-pointed at /home/novakj and collapsed
  the one-off entries into prefixes.
- config/samples/proxy_kvm2_crawl_only_one.yaml: a one-VM cut of the crawl
  pool, the rest of the members left commented out.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kacerr
2026-09-18 08:12:27 +00:00
co-authored by Claude Opus 5
parent 1fa3e55b9c
commit 8fa9e8aad9
8 changed files with 319 additions and 422 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+42 -320
View File
@@ -1,320 +1,42 @@
# egress-proxies-operator - AI Agent Guide
## Project Structure
**Single-group layout (default):**
```
cmd/main.go Manager entry (registers controllers/webhooks)
api/<version>/*_types.go CRD schemas (+kubebuilder markers)
api/<version>/zz_generated.* Auto-generated (DO NOT EDIT)
internal/controller/* Reconciliation logic
internal/webhook/* Validation/defaulting (if present)
config/crd/bases/* Generated CRDs (DO NOT EDIT)
config/rbac/role.yaml Generated RBAC (DO NOT EDIT)
config/samples/* Example CRs (edit these)
Makefile Build/test/deploy commands
PROJECT Kubebuilder metadata Auto-generated (DO NOT EDIT)
```
**Multi-group layout** (for projects with multiple API groups):
```
api/<group>/<version>/*_types.go CRD schemas by group
internal/controller/<group>/* Controllers by group
internal/webhook/<group>/<version>/* Webhooks by group and version (if present)
```
Multi-group layout organizes APIs by group name (e.g., `batch`, `apps`). Check the `PROJECT` file for `multigroup: true`.
**To convert to multi-group layout:**
1. Run: `kubebuilder edit --multigroup=true`
2. Move APIs: `mkdir -p api/<group> && mv api/<version> api/<group>/`
3. Move controllers: `mkdir -p internal/controller/<group> && mv internal/controller/*.go internal/controller/<group>/`
4. Move webhooks (if present): `mkdir -p internal/webhook/<group> && mv internal/webhook/<version> internal/webhook/<group>/`
5. Update import paths in all files
6. Fix `path` in `PROJECT` file for each resource
7. Update test suite CRD paths (add one more `..` to relative paths)
## Critical Rules
### Never Edit These (Auto-Generated)
- `config/crd/bases/*.yaml` - from `make manifests`
- `config/rbac/role.yaml` - from `make manifests`
- `config/webhook/manifests.yaml` - from `make manifests`
- `**/zz_generated.*.go` - from `make generate`
- `PROJECT` - from `kubebuilder [OPTIONS]`
### Never Remove Scaffold Markers
Do NOT delete `// +kubebuilder:scaffold:*` comments. CLI injects code at these markers.
### Keep Project Structure
Do not move files around. The CLI expects files in specific locations.
### Always Use CLI Commands
Always use `kubebuilder create api` and `kubebuilder create webhook` to scaffold. Do NOT create files manually.
### E2E Tests Require an Isolated Kind Cluster
The e2e tests are designed to validate the solution in an isolated environment (similar to GitHub Actions CI).
Ensure you run them against a dedicated [Kind](https://kind.sigs.k8s.io/) cluster (not your “real” dev/prod cluster).
## After Making Changes
**After editing `*_types.go` or markers:**
```
make manifests # Regenerate CRDs/RBAC from markers
make generate # Regenerate DeepCopy methods
```
**After editing `*.go` files:**
```
make lint-fix # Auto-fix code style
make test # Run unit tests
```
## CLI Commands Cheat Sheet
### Create API (your own types)
```bash
kubebuilder create api --group <group> --version <version> --kind <Kind>
```
### Deploy Image Plugin (scaffold to deploy/manage ANY container image)
Generate a controller that deploys and manages a container image (nginx, redis, memcached, your app, etc.):
```bash
# Example: deploying memcached
kubebuilder create api --group example.com --version v1alpha1 --kind Memcached \
--image=memcached:alpine \
--plugins=deploy-image.go.kubebuilder.io/v1-alpha
```
Scaffolds good-practice code: reconciliation logic, status conditions, finalizers, RBAC. Use as a reference implementation.
### Create Webhooks
```bash
# Validation + defaulting
kubebuilder create webhook --group <group> --version <version> --kind <Kind> \
--defaulting --programmatic-validation
# Conversion webhook (for multi-version APIs)
kubebuilder create webhook --group <group> --version v1 --kind <Kind> \
--conversion --spoke v2
```
### Controller for Core Kubernetes Types
```bash
# Watch Pods
kubebuilder create api --group core --version v1 --kind Pod \
--controller=true --resource=false
# Watch Deployments
kubebuilder create api --group apps --version v1 --kind Deployment \
--controller=true --resource=false
```
### Controller for External Types (e.g., from other operators)
Watch resources from external APIs (cert-manager, Argo CD, Istio, etc.):
```bash
# Example: watching cert-manager Certificate resources
kubebuilder create api \
--group cert-manager --version v1 --kind Certificate \
--controller=true --resource=false \
--external-api-path=github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1 \
--external-api-domain=io \
--external-api-module=github.com/cert-manager/cert-manager
```
**Note:** Use `--external-api-module=<module>@<version>` only if you need a specific version. Otherwise, omit `@<version>` to use what's in go.mod.
### Webhook for External Types
```bash
# Example: validating external resources
kubebuilder create webhook \
--group cert-manager --version v1 --kind Issuer \
--defaulting \
--external-api-path=github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1 \
--external-api-domain=io \
--external-api-module=github.com/cert-manager/cert-manager
```
## Testing & Development
```bash
make test # Run unit tests (uses envtest: real K8s API + etcd)
make run # Run locally (uses current kubeconfig context)
```
Tests use **Ginkgo + Gomega** (BDD style). Check `suite_test.go` for setup.
## Deployment Workflow
```bash
# 1. Regenerate manifests
make manifests generate
# 2. Build & deploy
export IMG=<registry>/<project>:tag
make docker-build docker-push IMG=$IMG # Or: kind load docker-image $IMG --name <cluster>
make deploy IMG=$IMG
# 3. Test
kubectl apply -k config/samples/
# 4. Debug
kubectl logs -n <project>-system deployment/<project>-controller-manager -c manager -f
```
### API Design
**Key markers for** `api/<version>/*_types.go`:
```go
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:scope=Namespaced
// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=".status.conditions[?(@.type=='Ready')].status"
// On fields:
// +kubebuilder:validation:Required
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:MaxLength=100
// +kubebuilder:validation:Pattern="^[a-z]+$"
// +kubebuilder:default="value"
```
- **Use** `metav1.Condition` for status (not custom string fields)
- **Use predefined types**: `metav1.Time` instead of `string` for dates
- **Follow K8s API conventions**: Standard field names (`spec`, `status`, `metadata`)
### Controller Design
**RBAC markers in** `internal/controller/*_controller.go`:
```go
// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=mygroup.example.com,resources=mykinds/finalizers,verbs=update
// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
```
**Implementation rules:**
- **Idempotent reconciliation**: Safe to run multiple times
- **Re-fetch before updates**: `r.Get(ctx, req.NamespacedName, obj)` before `r.Update` to avoid conflicts
- **Structured logging**: `log := log.FromContext(ctx); log.Info("msg", "key", val)`
- **Owner references**: Enable automatic garbage collection (`SetControllerReference`)
- **Watch secondary resources**: Use `.Owns()` or `.Watches()`, not just `RequeueAfter`
- **Finalizers**: Clean up external resources (buckets, VMs, DNS entries)
### Logging
**Follow Kubernetes logging message style guidelines:**
- Start from a capital letter
- Do not end the message with a period
- Active voice: subject present (`"Deployment could not create Pod"`) or omitted (`"Could not create Pod"`)
- Past tense: `"Could not delete Pod"` not `"Cannot delete Pod"`
- Specify object type: `"Deleted Pod"` not `"Deleted"`
- Balanced key-value pairs
```go
log.Info("Starting reconciliation")
log.Info("Created Deployment", "name", deploy.Name)
log.Error(err, "Failed to create Pod", "name", name)
```
**Reference:** https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md#message-style-guidelines
### Webhooks
- **Create all types together**: `--defaulting --programmatic-validation --conversion`
- **When`--force`is used**: Backup custom logic first, then restore after scaffolding
- **For multi-version APIs**: Use hub-and-spoke pattern (`--conversion --spoke v2`)
- Hub version: Usually oldest stable version (v1)
- Spoke versions: Newer versions that convert to/from hub (v2, v3)
- Example: `--group crew --version v1 --kind Captain --conversion --spoke v2` (v1 is hub, v2 is spoke)
### Learning from Examples
The **deploy-image plugin** scaffolds a complete controller following good practices. Use it as a reference implementation:
```bash
kubebuilder create api --group example --version v1alpha1 --kind MyApp \
--image=<your-image> --plugins=deploy-image.go.kubebuilder.io/v1-alpha
```
Generated code includes: status conditions (`metav1.Condition`), finalizers, owner references, events, idempotent reconciliation.
## Distribution Options
### Option 1: YAML Bundle (Kustomize)
```bash
# Generate dist/install.yaml from Kustomize manifests
make build-installer IMG=<registry>/<project>:tag
```
**Key points:**
- The `dist/install.yaml` is generated from Kustomize manifests (CRDs, RBAC, Deployment)
- Commit this file to your repository for easy distribution
- Users only need `kubectl` to install (no additional tools required)
**Example:** Users install with a single command:
```bash
kubectl apply -f https://raw.githubusercontent.com/<org>/<repo>/<tag>/dist/install.yaml
```
### Option 2: Helm Chart
```bash
kubebuilder edit --plugins=helm/v2-alpha # Generates dist/chart/ (default)
kubebuilder edit --plugins=helm/v2-alpha --output-dir=charts # Generates charts/chart/
```
**For development:**
```bash
make helm-deploy IMG=<registry>/<project>:<tag> # Deploy manager via Helm
make helm-deploy IMG=$IMG HELM_EXTRA_ARGS="--set ..." # Deploy with custom values
make helm-status # Show release status
make helm-uninstall # Remove release
make helm-history # View release history
make helm-rollback # Rollback to previous version
```
**For end users/production:**
```bash
helm install my-release ./<output-dir>/chart/ --namespace <ns> --create-namespace
```
**Important:** If you add webhooks or modify manifests after initial chart generation:
1. Backup any customizations in `<output-dir>/chart/values.yaml` and `<output-dir>/chart/manager/manager.yaml`
2. Re-run: `kubebuilder edit --plugins=helm/v2-alpha --force` (use same `--output-dir` if customized)
3. Manually restore your custom values from the backup
### Publish Container Image
```bash
export IMG=<registry>/<project>:<version>
make docker-build docker-push IMG=$IMG
```
## References
### Essential Reading
- **Kubebuilder Book**: https://book.kubebuilder.io (comprehensive guide)
- **controller-runtime FAQ**: https://github.com/kubernetes-sigs/controller-runtime/blob/main/FAQ.md (common patterns and questions)
- **Good Practices**: https://book.kubebuilder.io/reference/good-practices.html (why reconciliation is idempotent, status conditions, etc.)
- **Logging Conventions**: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md#message-style-guidelines (message style, verbosity levels)
### API Design & Implementation
- **API Conventions**: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md
- **Operator Pattern**: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
- **Markers Reference**: https://book.kubebuilder.io/reference/markers.html
### Tools & Libraries
- **controller-runtime**: https://github.com/kubernetes-sigs/controller-runtime
- **controller-tools**: https://github.com/kubernetes-sigs/controller-tools
- **Kubebuilder Repo**: https://github.com/kubernetes-sigs/kubebuilder
# egress-proxies-operator
## Sources and architecture
- Trust `go.mod`, `Makefile`, and current code over version tables or bootstrap prose. `CLAUDE.md`'s opening architecture/commands sections are stale; this is a working operator, not an empty scaffold.
- `cmd/main.go` assembles the provider registry, reconciler, proxy health engine, in-memory lease store, discovery HTTP API, orphan GC, and provider-health monitor on one controller-runtime manager.
- `api/v1alpha1` contains namespaced `Proxy` and cluster-scoped `ProviderHealth` resources (`crawl.example.com`). Validation/defaulting uses CRD markers/CEL; there are no admission webhooks.
- `internal/provider` defines the backend contract independently of CRD types. Constructors are registered in `cmd/main.go`; `spec.provider` selects a configured **instance name**, not a provider type. `kvm` and `kvm2` are separate implementations and do not adopt each other's VMs.
- Before changing a provider, read `docs/provider-contract.md`; use `docs/provider-spec-template.md` for a new one. Callers rely on self-contained provider IDs, truthful `ErrNotFound`, idempotent create/delete, and persisted UID/creation time for orphan GC. See `docs/providers-existing.md` for implementation-specific exceptions.
- `docs/architecture.md` explains component ownership and state transitions; `docs/api.md` documents discovery/leases; `docs/testing.md` covers test boundaries and real-infrastructure setup.
## Commands and verification
- Go version comes from `go.mod` (currently 1.26.0). Make installs development tools into `bin/`.
- After Go changes: `make lint-fix`, then `make test`. The latter regenerates manifests/DeepCopy, formats, vets, sets up envtest, then runs non-e2e tests with `-race` and writes `cover.out`.
- Fast checks: `go test -short -race ./...` skips envtest. Focused package/test: `go test -race ./internal/lease/` or `go test -race -run TestAcquire ./internal/lease/`.
- Envtest needs local API-server/etcd binaries; use `make setup-envtest` before targeted runs. For the controller suite with the current Kubernetes minor:
```sh
KUBEBUILDER_ASSETS="$(./bin/setup-envtest use 1.36 --bin-dir "$PWD/bin" -p path)" go test -race ./internal/controller/ -run TestControllers -ginkgo.focus='<spec regex>'
```
The Makefile derives the envtest **minor** from `go.mod`; do not replace it with a dependency patch version that may have no published binary bundle.
- Tests are mostly stdlib `testing`; controller envtest and e2e use Ginkgo/Gomega. Fake clients do not execute CEL or schema defaults: test those with envtest, and use the API's `*OrDefault` helpers in unit tests. Envtest has no kubelet, so it cannot prove Squid starts or serves traffic.
- `make lint` / `make lint-fix` builds a custom golangci-lint with the `logcheck` plugin from `.custom-gcl.yml`; a stock global binary is insufficient. `.golangci.yml` forbids `sort` (use `slices`). Logging uses controller-runtime/logr, with Kubernetes-style messages.
- Real-host tests are opt-in: `make test-integration-kvm` / `make test-integration-kvm2` require the corresponding `KVM_TEST_*` / `KVM2_TEST_*` variables from `docs/testing.md`. After provider refactors, compile-check these with `go vet -tags=integration ./...`.
- E2E requires Docker, kind, kubectl, and an isolated cluster. `make test-e2e` uses `KIND_CLUSTER=egress-proxies-operator-test-e2e`; ensure kubectl targets that cluster, especially if it already exists. On failure, run `make cleanup-test-e2e` explicitly. Compile-check with `go vet -tags=e2e ./...`. Tracing specs additionally require both `TEMPO_URL` and `OTLP_ENDPOINT`.
- `.gitea/workflows/build.yaml` runs only on tags/manual dispatch; its gate is `go vet ./...`, `go build ./...`, `go test -short ./...`. It does not replace local lint, race, or envtest verification.
## Code generation
- After API types or Kubebuilder markers change, run `make manifests generate`. Edit source markers, never generated `config/crd/bases/*.yaml`, `config/rbac/role.yaml`, or `**/zz_generated.*.go`.
- Keep `+kubebuilder:rbac` markers in standalone package-level comment blocks, outside function/type documentation; otherwise controller-gen can silently ignore them. Inspect `git diff config/rbac/role.yaml` after generation.
- Preserve `+kubebuilder:scaffold:*` markers and the single-group layout. Use `kubebuilder create api` / `kubebuilder create webhook` for new scaffolding; `PROJECT` is CLI-managed.
## Behaviour and operational gotchas
- Managed proxies are replace-on-change: `internal/controller/spechash.go` hashes placement, resolved cloud-init content, and effective port. Do not hash all of `ProxySpec`: attributes, lease limits, and health settings must not churn VMs.
- Cloud-init Secrets must carry `crawl.example.com/cloud-init: "true"`; the manager's Secret cache excludes unlabelled Secrets. Rotating labelled Secret content triggers VM replacement.
- Proxy health checks fetch **through** the proxy. The health engine supplies snapshots/events and the reconciler writes status; status-affecting transitions drive writes, so `lastHealthCheckTime` is not a last-probe heartbeat.
- Leases/cooldowns are in-memory and disappear on restart. Discovery is not leader-elected; the deployment uses one replica. Adding replicas alone does not provide shared lease capacity.
- Use `make run-dev` locally: plain `make run` omits the required `--providers-config`. Host-side probes generally cannot route to kind pod IPs; deploy in-cluster using the README quickstart to verify kubernetes-provider health.
- `make deploy IMG=...` and `make build-installer IMG=...` modify `config/manager/kustomization.yaml` in place. Check that diff before including it in a change.
- Namespace-restricted mode disables orphan GC by default. `--gc-allow-namespaced` bypasses that guard and can classify instances owned outside the visible namespace as orphans.