# Runtime Flows Deep-dive reference for every control path in the gateway-cert-operator. Complements the user-facing overview in `docs/README.md`. --- ## 1. Overview The operator sits between two Kubernetes APIs and keeps them in sync: ``` +--------------------+ +------------------------+ | cert-manager | | gateway-cert-operator | | Certificate |----(watches)-------->| (this binary) | | (source of | | | | truth for TLS) | +----------+-------------+ +--------------------+ | | patches Spec.Listeners v +--------------------+ +------------------------+ | Gateway API | | Gateway | | controller |<----(watches)--------| (target object) | | (routes traffic) | +------------------------+ +--------------------+ ``` The operator makes no changes to cert-manager objects. Its only write action is a `PATCH` on `Gateway.Spec.Listeners`. All other interactions are reads. --- ## 2. Startup Flow Executed once at process start (`main.go`): ``` process start │ ├─ init() [main.go:22-26] │ ├─ clientgoscheme.AddToScheme(scheme) # core/v1: Secrets, Events, … │ ├─ cmv1.AddToScheme(scheme) # cert-manager.io/v1 Certificate │ └─ gwapiv1.Install(scheme) # gateway.networking.k8s.io/v1 Gateway │ └─ main() [main.go:28] │ ├─ flag.StringVar --metrics-bind-address default ":8080" [main.go:32] ├─ flag.StringVar --health-probe-bind-address default ":8081" [main.go:33] ├─ flag.Parse() [main.go:34] │ ├─ ctrl.SetLogger(zap, UseDevMode=true, AddCaller) [main.go:36-41] │ dev-mode console logger with file:line call site │ ├─ ctrl.GetConfigOrDie() # KUBECONFIG or in-cluster config │ on err ──► os.Exit(1) │ ├─ ctrl.NewManager(cfg, Options{Scheme, HealthProbeBindAddress: ":8081"}) │ on err ──► log + os.Exit(1) [main.go:44-47] │ NOTE: metricsAddr is parsed above but NOT passed here; │ the manager falls back to its own default, which │ is also ":8080" (see §10, item 2) │ ├─ controller.SetupCertificateReconciler(mgr) [main.go:53] │ registers field index + watches (see §3) │ on err ──► log + os.Exit(1) │ ├─ mgr.AddHealthzCheck("healthz", healthz.Ping) → GET /healthz:8081 ├─ mgr.AddReadyzCheck("readyz", healthz.Ping) → GET /readyz:8081 │ on err ──► log + os.Exit(1) │ └─ mgr.Start(ctrl.SetupSignalHandler()) [main.go:68] SetupSignalHandler installs SIGINT / SIGTERM cancel mgr.Start blocks; on shutdown → graceful stop on err ──► log + os.Exit(1) ``` --- ## 3. Watch Wiring and Event Sources `SetupCertificateReconciler` registers two watches before starting the controller (`certificate_controller.go:59-63`): ``` controller-runtime manager │ ├─ For(&cmv1.Certificate{}) primary watch │ every Certificate create/update/delete │ ──► enqueue Reconcile(cert.Namespace/cert.Name) │ └─ Watches(&gwapiv1.Gateway{}, EnqueueRequestsFromMapFunc(r.gatewayToCertificates)) secondary watch every Gateway create/update/delete ──► gatewayToCertificates maps Gateway → []Certificates ──► enqueue Reconcile for each matching Certificate ``` Full trigger table: | ID | Event | Enqueued object | Code path | |----|-------|-----------------|-----------| | E1 | Certificate **create** | that Certificate | primary watch | | E2 | Certificate **update** (any field — no predicate filter) | that Certificate | primary watch | | E3 | Certificate **delete** | that Certificate | primary watch | | E4 | Gateway **create / update / delete** | every Certificate targeting it (via field index) | secondary watch + `gatewayToCertificates` | | E5 | Reconcile returns non-nil error | same Certificate | controller-runtime rate-limited backoff re-queue | | E6 | Controller-runtime cache resync (~10 h default) | every known Certificate | implicit periodic replay | --- ## 4. Field Index The field index is the only mechanism that lets the operator find all Certificates pointing at a given Gateway without scanning the entire namespace. It is registered during setup and queried in two places. ``` IndexFieldGatewayTarget = ".metadata.annotations.gatewayTarget" (identifier used by controller-runtime; no annotation by this name exists) Indexer function (certificate_controller.go:44-55): ┌─────────────────────────────────────────────────────────┐ │ for each Certificate obj: │ │ gwName = obj.Annotations["…/gateway-name"] │ │ if gwName == "" → return nil (not indexed) │ │ gwNs = obj.Annotations["…/gateway-namespace"] │ │ if gwNs == "" → gwNs = obj.Namespace (fallback) │ │ return [ "/" ] │ └─────────────────────────────────────────────────────────┘ Key shape: "/" Two query sites: ┌──────────────────────────────────────────────────────┐ │ Reconcile [certificate_controller.go:95-99] │ │ List(Certs, MatchingFields{index: "/"})│ │ returns all sibling Certs targeting the same GW │ └──────────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────────┐ │ gatewayToCertificates [certificate_controller.go:138]│ │ List(Certs, MatchingFields{index: "/"})│ │ used by the secondary watch to fan-out GW events │ │ to the relevant Certs │ └──────────────────────────────────────────────────────┘ ``` The namespace-fallback rule (`gwNs = obj.Namespace` when the annotation is absent) is duplicated in two places: the indexer (`:51-53`) and `Reconcile` (`:89-91`). Both must apply the same rule or the index key used during write will differ from the key used during lookup. --- ## 5. Reconcile Decision Tree Every enqueued item arrives here. Inputs: one `Reconcile.Request` containing the namespace/name of a Certificate. Side effects: 0 or 1 `PATCH` on a Gateway. ``` Reconcile(ctx, req) │ ├─(A) r.Get(ctx, req.NamespacedName, &cert) [controller:71] │ │ │ ├─ NotFound ──► client.IgnoreNotFound → nil err │ │ return Result{}, nil ►► STOP — NO ACTION │ │ │ │ ┌─ IMPORTANT: the deleted cert's listener is NOT removed here. │ │ │ The controller has no record of which Gateway was targeted. │ │ │ Cleanup happens on the next E2/E4/E6 event for any sibling │ │ │ cert that still references the same Gateway — at that point │ │ │ BuildListenersFromCertificates omits the deleted cert and │ │ │ MergeListeners replaces all auto-* listeners. Until that │ │ │ event fires, the stale listener remains in the Gateway. │ │ │ (See §10, item 3 for details.) │ │ └─────────────────────────────────────────────────────────── │ │ │ └─ other error ──► return Result{}, err ►► REQUEUE (E5) │ ├─(B) gwName, ok := cert.Annotations["…/gateway-name"] [controller:82] │ │ │ └─ !ok ──► log V(1) "missing gateway annotation" │ return Result{}, nil ►► STOP — NO ACTION │ ├─(C) gwNamespace := cert.Annotations["…/gateway-namespace"] [controller:88] │ └─ if "" ──► gwNamespace = cert.Namespace (DEFAULT FALLBACK) │ ├─(D) r.List(ctx, &certList, [controller:95] │ MatchingFields{IndexFieldGatewayTarget: "/"}) │ │ │ └─ err ──► return Result{}, err ►► REQUEUE │ ├─(E) desiredListeners := │ gw.BuildListenersFromCertificates(certList.Items, "auto-") │ (see §6a) [controller:102] │ ├─(F) r.Get(ctx, {gwNs, gwName}, &gateway) [controller:106] │ │ │ └─ err (including NotFound) ──► return Result{}, err ►► REQUEUE │ │ NOTE: Gateway-not-found is NOT specially handled — the │ reconciler will retry with backoff indefinitely. A typo'd │ gateway-name annotation produces a perpetual retry loop. │ (See §10, item 5.) │ ├─(G) updated := gw.MergeListeners( [controller:111] │ gateway.Spec.Listeners, │ desiredListeners, "auto-") │ (see §6b) │ ├─(H) gw.ListenersEqual(gateway.Spec.Listeners, updated) [controller:113] │ │ │ └─ true ──► log V(1) "already up to date" │ return Result{}, nil ►► SHORT-CIRCUIT — NO PATCH │ ├─(I) patch := client.MergeFrom(gateway.DeepCopy()) [controller:118] │ gateway.Spec.Listeners = updated │ r.Patch(ctx, &gateway, patch) [controller:120] │ │ │ └─ err ──► return Result{}, err ►► REQUEUE │ └─(J) log Info "patched gateway listeners" [controller:124] return Result{}, nil ►► DONE Cluster I/O per reconcile: Reads: 1 Certificate Get (step A) 1 Certificate List via index (step D) 1 Gateway Get (step F) Writes: 0 or 1 Gateway Patch (step I, only when listeners changed) ``` --- ## 6. Build / Merge / Equal Pipeline The three pure functions in `internal/gateway/patch.go` do all computation. `Reconcile` only calls them; it holds no business logic of its own. ``` certList.Items │ v BuildListenersFromCertificates(certs, "auto-") [patch.go:22] │ └──► desired []gwapiv1.Listener │ │ gateway.Spec.Listeners (current state from API server) │ │ v v MergeListeners(existing, desired, "auto-") [patch.go:76] │ └──► updated []gwapiv1.Listener │ v ListenersEqual(gateway.Spec.Listeners, updated) [patch.go:93] │ ├─ true ──► short-circuit, no patch └─ false ──► Patch(gateway) ``` ### 6a. BuildListenersFromCertificates ``` for cert in certs: │ ├─ if cert.DeletionTimestamp != nil ──► skip [patch.go:26] │ (cert is being deleted; omitting it removes its listeners │ from the desired set and MergeListeners will drop them) │ ├─ if cert.Spec.SecretName == "" ──► skip [patch.go:30] │ └─ for dns in cert.Spec.DNSNames: [patch.go:34] │ └─ emit Listener{ Name: "auto--" Hostname: dns Port: 443 (hardcoded, see §8) Protocol: HTTPS AllowedRoutes: { From: All } TLS: { Mode: Terminate CertificateRefs: [{ Group: "" (core) Kind: "Secret" Name: cert.Spec.SecretName Namespace: cert.Namespace (same as cert) }] } } sort by Name ascending [patch.go:67] (deterministic order prevents spurious diffs on every reconcile) ``` ### 6b. MergeListeners ``` existing listeners │ ├─ Name starts with "auto-" ──► DROP (managed, will be replaced) └─ Name does NOT start with "auto-" ──► KEEP (user-managed) merged = [kept user listeners] + [all desired auto-* listeners] ``` The result: user-created listeners appear first in their original order; operator-managed listeners follow, sorted by name. **Edge case:** a user-created listener whose name happens to start with `auto-` will be treated as managed and will be deleted on the next reconcile. ### 6c. ListenersEqual `ListenersEqual` is a **partial comparison** — only the fields the operator actively manages are checked: | Compared | Ignored | |----------|---------| | listener `Name` | `Protocol` | | `Hostname` | `AllowedRoutes` | | `Port` | `TLS.Options` | | first `CertificateRef` (`/`) | cert refs beyond index 0 | | | `Group`, `Kind` of ref | Drift in an ignored field will not trigger a corrective patch. For example: if someone manually flips the Protocol of an auto-managed listener from HTTPS to HTTP, the operator will consider it "equal" and leave it alone. The comparison is done via a name-keyed map (`listenerMap`, `patch.go:118`), so listener order does not affect the result. --- ## 7. Listener Naming and Sanitization `listenerNameFor` (`patch.go:143`): ``` input: prefix="auto-" certName="my-app-cert" dnsName="app.example.com" step 1: replace every "." with "-" "app.example.com" ──► "app-example-com" step 2: concatenate "auto-" + "my-app-cert" + "-" + "app-example-com" ──► "auto-my-app-cert-app-example-com" step 3: if len > 253 bytes: name = name[:253] (hard byte slice) ``` Example outputs for a cert with two DNS names: | `dnsName` | listener `Name` | |-----------|-----------------| | `app.example.com` | `auto-my-app-cert-app-example-com` | | `*.example.com` | `auto-my-app-cert-*-example-com` | The second entry is a risk: `*` is not valid in a Gateway API `SectionName` (which must match RFC 1123). The API server will reject the patch and the reconciler will requeue forever (see §10, item 7). The sanitization is also lossy: `a.b.com` and `a-b-com` produce the same result. If a certificate lists both, two listeners with the same name are appended; the API server rejects the patch (listener names must be unique), so no listener on that Gateway is updated. --- ## 8. Defaults Baked In All of these are hardcoded in `internal/gateway/patch.go`. None are configurable via annotations (the `listener-port` annotation exists as a constant but is not read — see §10, item 1). | Property | Value | Source | |----------|-------|--------| | Listener port | `443` | `defaultPort` `patch.go:13` | | TLS mode | `TLSModeTerminate` | `tlsModeTerminate` `patch.go:14` | | Allowed routes | `NamespacesFromAll` | `namespacesFromAll` `patch.go:15` | | Protocol | `HTTPSProtocolType` | inline `patch.go:45` | | CertificateRef Group | `""` (core API group) | `patch.go:55` | | CertificateRef Kind | `"Secret"` | `patch.go:56` | | Secret namespace | cert's own namespace | `patch.go:39,58` | | Listener name prefix | `"auto-"` | `ListenerPrefix` `controller:28` | | SectionName max length | 253 bytes | `patch.go:147` | | Gateway namespace fallback | cert's own namespace | `controller:51-53` and `controller:89-91` | --- ## 9. End-to-End Scenarios ### Scenario 1 — First Certificate created, Gateway has no auto-listeners ``` 1. User creates Certificate "my-cert" (namespace: default) Annotations: gateway-name=my-gateway Spec: dnsNames=[app.example.com], secretName=my-cert-tls 2. (E1) Cert create event fires → Reconcile(default/my-cert) 3. (A) Get cert → found (B) gateway-name="my-gateway" → ok (C) gateway-namespace missing → fallback to "default" (D) List certs by index "default/my-gateway" → [my-cert] (E) BuildListeners → [Listener{Name:"auto-my-cert-app-example-com", Port:443, …}] (F) Get gateway "default/my-gateway" → found, Listeners=[] (G) MergeListeners([], desired) → [auto-my-cert-app-example-com] (H) ListenersEqual([], updated) → false (len differs) (I) Patch gateway → Spec.Listeners=[auto-my-cert-app-example-com] (J) Done. Log: "patched gateway listeners" Cluster writes: 1 Gateway PATCH ``` ### Scenario 2 — Second Certificate added for the same Gateway ``` 1. User creates Certificate "other-cert" targeting the same Gateway dnsNames=[api.example.com], secretName=other-cert-tls 2. (E1) Reconcile(default/other-cert) 3. (D) List certs by index "default/my-gateway" → [my-cert, other-cert] (E) BuildListeners → [ auto-my-cert-app-example-com, auto-other-cert-api-example-com ] (sorted alphabetically by name) (F) Gateway still has [auto-my-cert-app-example-com] (G) MergeListeners → [auto-my-cert-app-example-com, auto-other-cert-api-example-com] (H) ListenersEqual(old 1-item, new 2-item) → false (I) Patch → Gateway now has both listeners Cluster writes: 1 Gateway PATCH ``` ### Scenario 3 — Certificate dnsNames updated ``` 1. User edits "my-cert": dnsNames=[app.example.com] → [app.example.com, www.example.com] 2. (E2) Cert update event → Reconcile(default/my-cert) 3. (E) BuildListeners produces TWO listeners for my-cert now: auto-my-cert-app-example-com auto-my-cert-www-example-com (plus any from other-cert if still present) (H) ListenersEqual: len changed → false (I) Patch gateway with new full set Cluster writes: 1 Gateway PATCH ``` ### Scenario 4 — Certificate deleted ``` 1. User deletes "my-cert" 2. (E3) Cert delete event → Reconcile(default/my-cert) 3. (A) Get cert → NotFound return Result{}, nil ──► STOP, NO ACTION 4. Gateway still contains "auto-my-cert-app-example-com" — stale. 5. Cleanup happens only when the next relevant event fires: - If "other-cert" is updated or created → (E2) Reconcile(other-cert) (D) List by index returns [other-cert] only (my-cert is gone) (E) BuildListeners → only other-cert's listeners (G) MergeListeners drops the stale auto-my-cert-* listener (I) Patch removes it - OR if the Gateway is manually edited → (E4) all targeting Certs re-enqueued, same convergence path - OR on the next cache resync (E6, ~10 h) Every path above needs at least one surviving Certificate that targets the Gateway. If my-cert was the last one, nothing is ever enqueued for it and the stale listener stays until someone removes it by hand. Cluster writes: 0 at deletion time; 1 Gateway PATCH at next sibling event ``` --- ## 10. Known Limitations Recorded as-found. None are fixed by this document. **1. `AnnotationListenerPort` is dead code** `certificate_controller.go:25` declares the constant; `BuildListenersFromCertificates` (`patch.go:44`) always uses `defaultPort = 443`. Setting `gateway-cert-operator.io/listener-port` on a Certificate has no effect. **2. `--metrics-bind-address` flag is parsed but not wired** `main.go:32` parses the flag; `main.go:44-47` never passes `MetricsBindAddress` to `ctrl.Options`. controller-runtime defaults an empty bind address to `:8080`, so metrics are served and the Service in `deploy/service_gateway-cert-operator-metrics.yaml` works; only changing the flag has no effect. **3. No finalizer → cert deletion leaves stale listeners** When a Certificate is deleted, Reconcile returns early at branch (A) with no cluster write. The stale `auto-*` listener persists in the Gateway until a sibling event triggers re-convergence (see Scenario 4 above), or permanently if no other Certificate targets that Gateway. Adding a finalizer to the Certificate would close this gap by allowing cleanup before the object is removed from the API server. **4. `ListenersEqual` is a partial comparison** Described in §6c. Drift in `Protocol`, `AllowedRoutes`, `TLS.Options`, or any `CertificateRef` beyond the first will not be detected or corrected. **5. Gateway-not-found retries forever** Branch (F) of Reconcile treats all Gateway `Get` errors the same (including `NotFound`) and returns them as re-queue-able errors. A Certificate with a typo'd `gateway-name` annotation will retry with rate-limited backoff indefinitely, generating log noise and API load. There is no terminal failure path. **6. No leader election** `ctrl.NewManager` at `main.go:44` does not set `LeaderElection: true`. The `deploy/deployment_gateway-cert-operator.yaml` runs `replicas: 1` which makes this safe today. Scaling to multiple replicas without enabling leader election would result in concurrent Gateway patches. **7. `listenerNameFor` edge cases** - **Wildcard hostnames**: `*.example.com` produces `SectionName` `auto-…-*-example-com`. The `*` character is not valid in an RFC 1123 label; the Gateway API server will reject the Patch, causing an error-retry loop. - **Collisions after sanitization**: `a.b.com` and `a-b-com` produce identical sanitized names. The same happens across certificates (cert `a` with `b-c.com` and cert `a-b` with `c.com`) and across namespaces, since the name omits the cert namespace. Both listeners are appended and the API server rejects the patch for the whole Gateway. - **Truncation at 253 bytes**: `patch.go:147` slices on bytes, not runes (safe for ASCII DNS names). Two DNS names sharing the first 253 characters after sanitization will map to the same listener name. **Dead symbols in `patch.go`** `gatewayGroup` and `gatewayKind` are declared at `patch.go:16-17` but never referenced. They can be removed without effect.