From 6f28c9731910da137ae22ac279a60ba4784113a5 Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Tue, 4 Aug 2026 23:23:04 +0200 Subject: [PATCH] cks: swap kind CNI to Cilium, fix broken ingress-nginx setup kindnet's NetworkPolicy enforcement isn't guaranteed, undermining the Task 1 lab; Cilium (with kube-proxy replacement) enforces it deterministically. Also fixes ingress-nginx, which was silently broken: missing ingress-ready node label left the controller Pending forever, and hostPorts 80/443 were never mapped in kind-config. Co-Authored-By: Claude Opus 5 --- kubernetes/cks/exam-by-claude/README.md | 7 ++-- kubernetes/cks/exam-by-claude/bootstrap.sh | 36 ++++++++++++++++--- kubernetes/cks/exam-by-claude/exam.md | 2 +- .../cks/exam-by-claude/kind-config.yaml | 16 ++++++++- kubernetes/cks/exam-by-claude/solutions.md | 9 +++-- 5 files changed, 59 insertions(+), 11 deletions(-) diff --git a/kubernetes/cks/exam-by-claude/README.md b/kubernetes/cks/exam-by-claude/README.md index c632414..4aa5b3b 100644 --- a/kubernetes/cks/exam-by-claude/README.md +++ b/kubernetes/cks/exam-by-claude/README.md @@ -3,8 +3,8 @@ Timed practice that mirrors the current CKS (17 tasks, 2h, six domains). Runs on kind (k8s v1.35.5) with the host-side / kernel bits on your Ubuntu VM. ## Prereqs (Ubuntu VM) -- docker, `kind >= v0.32.0`, `kubectl` v1.35, `helm` -- host tools installed latest: `kube-bench`, `trivy`, `kubesec`, `falco` (v0.44.x), optional `cosign` +- docker, `kind >= v0.32.0`, `kubectl` v1.35, `helm` (used to install Cilium — load-bearing, not optional) +- host tools installed latest: `kube-bench`, `trivy`, `kubesec`, `falco` (v0.44.x), optional `cosign`, optional `cilium-cli` for `cilium status`/`cilium connectivity test` - host must have AppArmor enabled (default on Ubuntu) for Task 7 ## Quickstart @@ -32,6 +32,9 @@ kind delete cluster --name cks && ./bootstrap.sh && ./seed.sh ## Notes / kind caveats - **Version:** exam is on v1.35 per the Linux Foundation page. k8s 1.36 shipped ~May 2026; the exam env aligns "within 4–8 weeks" of a release, so it *may* have rolled to 1.36 by your date — verify on the LF exam page. Nothing in this set changes between 1.35/1.36. To bump: swap the digest in `kind-config.yaml` for a `kindest/node:v1.36.x` one. +- **CNI is Cilium**, not kindnet — `disableDefaultCNI: true` in `kind-config.yaml`, installed by `bootstrap.sh` via Helm. It runs with the **kube-proxy replacement** (`kubeProxyMode: none`), so there is no `kube-proxy` DaemonSet in `kube-system`; kube-bench's kube-proxy CIS checks are N/A (Task 3's four target checks — kubelet ×3, apiserver ×1 — are unaffected). This also means Task 1's NetworkPolicy is genuinely enforced, not a maybe. +- Between `kind create cluster` and the Cilium install finishing, nodes sit **NotReady** and CoreDNS is **Pending** — expected, not a failure; `bootstrap.sh` waits it out. +- Ingress reaches the cluster on host ports **80/443** (mapped in `kind-config.yaml`) via the control-plane node, which is labelled `ingress-ready=true` so the ingress-nginx kind manifest schedules onto it. - **AppArmor** needs a real AppArmor host; works on Ubuntu, not on Docker Desktop/macOS. - **gVisor (Task 11)** pod stays Pending in kind (no `runsc` in kind's containerd) — the config is the graded artifact. - **ImagePolicyWebhook / audit / encryption** edit the live apiserver: always `cp` the manifest first so you can revert fast. diff --git a/kubernetes/cks/exam-by-claude/bootstrap.sh b/kubernetes/cks/exam-by-claude/bootstrap.sh index 40bffe4..f609648 100644 --- a/kubernetes/cks/exam-by-claude/bootstrap.sh +++ b/kubernetes/cks/exam-by-claude/bootstrap.sh @@ -1,32 +1,60 @@ #!/usr/bin/env bash # CKS mock — cluster bootstrap. Run on your Ubuntu VM (needs docker, kind >= v0.32.0, kubectl v1.35, helm). +# CNI is Cilium (kube-proxy replacement, no kindnet, no kube-proxy) — see kind-config.yaml's networking: block. set -euo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" cd "$HERE" +# ---- pinned versions ---- +CILIUM_VERSION="1.20.0" +INGRESS_NGINX_REF="controller-v1.15.1" + mkdir -p exam-files echo "[*] Creating kind cluster 'cks' (1 cp + 2 workers, k8s v1.35.5)..." +# No --wait here: with disableDefaultCNI, nodes are legitimately NotReady (and CoreDNS Pending) +# until Cilium is installed below — waiting now would just time out. kind create cluster --config kind-config.yaml kubectl config use-context kind-cks +echo "[*] Installing Cilium v${CILIUM_VERSION} (CNI + kube-proxy replacement)..." +CP_IP="$(docker inspect -f '{{.NetworkSettings.Networks.kind.IPAddress}}' cks-control-plane)" +helm repo add cilium https://helm.cilium.io/ >/dev/null +helm repo update cilium >/dev/null +helm upgrade --install cilium cilium/cilium \ + --version "$CILIUM_VERSION" --namespace kube-system \ + --set ipam.mode=kubernetes \ + --set kubeProxyReplacement=true \ + --set k8sServiceHost="$CP_IP" --set k8sServicePort=6443 \ + --set operator.replicas=1 \ + --set image.pullPolicy=IfNotPresent \ + --wait --timeout=5m + +echo "[*] Waiting for Cilium and nodes to be Ready..." +kubectl -n kube-system rollout status ds/cilium --timeout=300s +kubectl wait --for=condition=Ready nodes --all --timeout=300s +command -v cilium >/dev/null 2>&1 && cilium status --wait || true + echo "[*] Installing an ingress controller (needed for the TLS task)..." -kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml -kubectl -n ingress-nginx wait --for=condition=Ready pod \ - -l app.kubernetes.io/component=controller --timeout=180s || true +kubectl label node cks-control-plane ingress-ready=true --overwrite +kubectl apply -f "https://raw.githubusercontent.com/kubernetes/ingress-nginx/${INGRESS_NGINX_REF}/deploy/static/provider/kind/deploy.yaml" +kubectl -n ingress-nginx rollout status deploy/ingress-nginx-controller --timeout=300s cat <<'EOF' [*] Cluster up. Sanity check: kubectl get nodes -o wide + cilium status + kubectl -n kube-system get ds kube-proxy # expected: not found — Cilium replaces it [*] Host-side tooling you want on the VM (install latest): - kube-bench : https://github.com/aquasecurity/kube-bench/releases (or run as a Job in-cluster) - trivy : https://github.com/aquasecurity/trivy (apt: aquasecurity repo) - kubesec : https://github.com/controlplaneio/kubesec/releases - falco : apt install falco (v0.44.1) — used in the runtime task on the VM - - helm : for Kyverno / Falco chart installs + - helm : for Kyverno / Falco chart installs / Cilium (already used above) + - cilium-cli : optional, https://github.com/cilium/cilium-cli/releases — `cilium status`/`cilium connectivity test` - cosign : optional, image signing task [*] Now run ./seed.sh to plant the vulnerable/target objects for the kubectl-only tasks. diff --git a/kubernetes/cks/exam-by-claude/exam.md b/kubernetes/cks/exam-by-claude/exam.md index a1fc526..a5ae889 100644 --- a/kubernetes/cks/exam-by-claude/exam.md +++ b/kubernetes/cks/exam-by-claude/exam.md @@ -43,7 +43,7 @@ Expose the `hello` Service in namespace `web` over HTTPS. 1. Create a TLS Secret `hello-tls` in `web` from a self-signed cert for host `hello.cks.local` (you generate the cert/key on the VM). 2. Create an Ingress `hello` in `web` routing host `hello.cks.local` path `/` to Service `hello:80`, using `hello-tls` for TLS. -> Verify: `curl -k --resolve hello.cks.local:443:127.0.0.1 https://hello.cks.local/` returns `hello-tls` (adjust port to the ingress-nginx NodePort/hostPort in your setup). +> Verify: `curl -k --resolve hello.cks.local:443:127.0.0.1 https://hello.cks.local/` returns `hello-tls`. --- diff --git a/kubernetes/cks/exam-by-claude/kind-config.yaml b/kubernetes/cks/exam-by-claude/kind-config.yaml index 22052c5..3a5ae8d 100644 --- a/kubernetes/cks/exam-by-claude/kind-config.yaml +++ b/kubernetes/cks/exam-by-claude/kind-config.yaml @@ -1,9 +1,16 @@ # CKS mock cluster — 1 control-plane + 2 workers, k8s v1.35 # Pinned digest from kind v0.32.0 (default 1.36) — we force 1.35.5 to match the exam. # If `kind load`/image pull complains, use kind >= v0.32.0. +# +# CNI is Cilium (installed by bootstrap.sh), not kindnet — see networking: below. kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: cks +networking: + disableDefaultCNI: true # no kindnet — Cilium takes over, and it enforces NetworkPolicy deterministically + kubeProxyMode: none # Cilium eBPF kube-proxy replacement (no kube-proxy DaemonSet) + podSubnet: "10.244.0.0/16" # Cilium runs with ipam.mode=kubernetes, honoring this + serviceSubnet: "10.96.0.0/16" nodes: - role: control-plane image: kindest/node:v1.35.5@sha256:ce977ae6d65918d0b58a5f8b5e940429c2ce42fa3a5619ec2bbc60b949c0ac95 @@ -12,7 +19,14 @@ nodes: - hostPath: ./exam-files containerPath: /etc/kubernetes/exam extraPortMappings: - # so you can curl the Ingress task from the VM + # ingress-nginx's kind provider manifest hostPorts — Task 2 (TLS Ingress) needs 80/443 reachable + - containerPort: 80 + hostPort: 80 + protocol: TCP + - containerPort: 443 + hostPort: 443 + protocol: TCP + # so you can curl the Ingress task from the VM via NodePort too, if you set one up - containerPort: 30443 hostPort: 30443 protocol: TCP diff --git a/kubernetes/cks/exam-by-claude/solutions.md b/kubernetes/cks/exam-by-claude/solutions.md index dd1b44d..7397f4e 100644 --- a/kubernetes/cks/exam-by-claude/solutions.md +++ b/kubernetes/cks/exam-by-claude/solutions.md @@ -34,7 +34,11 @@ Verify: kubectl -n prod exec deploy/frontend -- curl -s --max-time 3 db:5432 # -> db-ok kubectl -n prod exec deploy/attacker -- curl -s --max-time 3 db:5432 # -> timeout ``` -Gotcha: kind's default CNI (kindnetd) enforces NetworkPolicy in recent versions; if your kindnet build doesn't, swap to Calico. On the exam CNI always enforces. +Gotcha: this cluster runs Cilium (not kindnetd), which enforces NetworkPolicy deterministically — no "if your build doesn't enforce it" caveat. If the attacker curl doesn't actually time out, don't blame the CNI; debug it: +```bash +cilium status # agents healthy on all 3 nodes? +kubectl -n kube-system exec ds/cilium -- cilium-dbg endpoint list # policy enforcement column for the db/attacker/frontend endpoints +``` --- @@ -64,9 +68,8 @@ spec: pathType: Prefix backend: {service: {name: hello, port: {number: 80}}} ``` -Verify (port depends on how ingress-nginx is exposed in kind — NodePort or the hostPort mapping): +Verify: ```bash -kubectl -n ingress-nginx get svc ingress-nginx-controller curl -k --resolve hello.cks.local:443:127.0.0.1 https://hello.cks.local/ ```