docs: add Kubernetes CKS study notes
This commit is contained in:
120
kubernetes/level 2/task-10.md
Normal file
120
kubernetes/level 2/task-10.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# Assignment
|
||||
|
||||
Last week, the Nautilus DevOps team deployed a redis app on Kubernetes cluster, which was working fine so far. This morning one of the team members was making some changes in this existing setup, but he made some mistakes and the app went down. We need to fix this as soon as possible. Please take a look.
|
||||
|
||||
|
||||
|
||||
The deployment name is redis-deployment. The pods are not in running state right now, so please look into the issue and fix the same.
|
||||
|
||||
|
||||
Note: The kubectl utility on the jump-host has been configured to work with the Kubernetes cluster.
|
||||
|
||||
# Solution
|
||||
|
||||
```bash
|
||||
# - incorrect config map name
|
||||
# - incorrect image on redis
|
||||
```
|
||||
|
||||
# Troubleshoot Deployment — `redis-deployment`
|
||||
|
||||
The redis pods won't start after a bad edit. This walks the diagnosis, then the fix for whichever
|
||||
root cause `describe` reveals.
|
||||
|
||||
## Step 1 — Diagnose
|
||||
|
||||
```bash
|
||||
# Deployment + pod state
|
||||
kubectl get deployment redis-deployment
|
||||
kubectl get pods -l app=redis # adjust selector if different
|
||||
|
||||
# The authoritative source: per-container state + Events at the bottom
|
||||
kubectl describe pod -l app=redis
|
||||
# (or describe a specific pod name from the get output)
|
||||
|
||||
# The full spec, to spot the bad edit
|
||||
kubectl get deployment redis-deployment -o yaml
|
||||
```
|
||||
|
||||
Read the **container state** and **Events** in `describe`. The status tells you the class of bug:
|
||||
|
||||
| Symptom in `describe` | Root cause |
|
||||
|-----------------------|------------|
|
||||
| `ErrImagePull` / `ImagePullBackOff` | **Image name/tag typo** (e.g. `redis:alpin` → `redis:alpine`). |
|
||||
| `CreateContainerConfigError` | **Bad reference** — a `configMapKeyRef` / `configMap` volume / `secretKeyRef` pointing at a name or key that doesn't exist. |
|
||||
| `CrashLoopBackOff` | Container starts then exits — bad command/args or a config the app rejects. |
|
||||
| `Pending` / `FailedScheduling` | An unschedulable request (e.g. `resources.requests` too high, bad nodeSelector). |
|
||||
|
||||
## Step 2 — Fix (match to what you found)
|
||||
|
||||
### A) Image typo → correct the image
|
||||
|
||||
```bash
|
||||
# Confirm the container name and the wrong image
|
||||
kubectl get deployment redis-deployment \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].name}{" "}{.spec.template.spec.containers[0].image}{"\n"}'
|
||||
|
||||
# Fix it (use the real container name = left of '='; correct the image/tag)
|
||||
kubectl set image deployment/redis-deployment redis-container=redis:alpine
|
||||
```
|
||||
|
||||
### B) Bad configMap / volume reference → correct the name
|
||||
|
||||
If a `configMap` volume or `configMapKeyRef` points at a mistyped name, fix it in place:
|
||||
|
||||
```bash
|
||||
kubectl edit deployment redis-deployment
|
||||
# find the wrong reference (e.g. name: redis-cofig) and correct it (redis-config),
|
||||
# matching an existing ConfigMap:
|
||||
kubectl get configmaps
|
||||
```
|
||||
|
||||
### C) Unschedulable resource request → lower it
|
||||
|
||||
```bash
|
||||
# Example: a request of "2" CPU on a small node leaves the pod Pending
|
||||
kubectl edit deployment redis-deployment
|
||||
# correct spec.template.spec.containers[].resources.requests to a sane value
|
||||
```
|
||||
|
||||
Any of these edits changes the pod template and triggers a fresh rollout automatically.
|
||||
|
||||
```bash
|
||||
kubectl rollout status deployment/redis-deployment
|
||||
```
|
||||
|
||||
## How it works
|
||||
|
||||
### Why `describe` is the first move, not a guess
|
||||
|
||||
A broken pod advertises its exact failure in its container state and Events. `ImagePullBackOff`
|
||||
names the image it couldn't pull (revealing a typo); `CreateContainerConfigError` names the
|
||||
missing ConfigMap/Secret; `FailedScheduling` states why no node fits. Reading that first tells you
|
||||
*which* field the "mistake" touched, so you fix one thing instead of shotgunning changes.
|
||||
|
||||
### Why in-place edits over re-apply here
|
||||
|
||||
The fix is a single-field correction on a deployment you didn't author and whose full spec you may
|
||||
not have cleanly. `kubectl set image` (for the image) and `kubectl edit` (for a reference or
|
||||
resource value) touch exactly the broken field and leave the rest intact — safer than
|
||||
reconstructing the whole manifest. Each edit updates the pod template, so the Deployment rolls out
|
||||
corrected pods on its own.
|
||||
|
||||
### Why the pods recover automatically
|
||||
|
||||
A Deployment continuously reconciles toward its spec. Once the template is valid (real image,
|
||||
existing ConfigMap, schedulable requests), the ReplicaSet successfully creates pods and they reach
|
||||
`Running` — no manual pod deletion needed, though you can delete a stuck pod to speed replacement.
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
kubectl get deployment redis-deployment # READY should match desired
|
||||
kubectl get pods -l app=redis # all Running
|
||||
kubectl describe deployment redis-deployment | sed -n '/Events/,$p'
|
||||
```
|
||||
|
||||
Expected — `redis-deployment` `READY N/N`, all pods `Running`, and no recurring error events.
|
||||
|
||||
> Paste the output of `kubectl get deployment redis-deployment -o yaml` if you want the exact
|
||||
> one-line patch — the fix depends on which field was mistyped.
|
||||
Reference in New Issue
Block a user