133 lines
5.0 KiB
Markdown
133 lines
5.0 KiB
Markdown
# Assignment
|
|
|
|
A junior DevOps team member encountered difficulties deploying a stack on the Kubernetes cluster. The pod fails to start, presenting errors. Let's troubleshoot and rectify the issue promptly.
|
|
|
|
|
|
There is a pod named webserver, and the container within it is named httpd-container, its utilizing the httpd:latest image.
|
|
|
|
Additionally, there's a sidecar container named sidecar-container using the ubuntu:latest image.
|
|
|
|
Identify and address the issue to ensure the pod is in the running state and the application is accessible.
|
|
|
|
Note: The kubectl utility on the jump-host has been configured to work with the Kubernetes cluster.
|
|
|
|
# Solution
|
|
|
|
There is incorrect image in the pod ... just fix it
|
|
|
|
# Troubleshoot Pod — `webserver` (httpd + ubuntu sidecar)
|
|
|
|
The `webserver` pod won't start. This walks the diagnosis, then applies a corrected
|
|
manifest via heredoc.
|
|
|
|
## Step 1 — Diagnose
|
|
|
|
```bash
|
|
# Overall state + which container is failing
|
|
kubectl get pod webserver -o wide
|
|
|
|
# The authoritative source: events + per-container state at the bottom
|
|
kubectl describe pod webserver
|
|
|
|
# Logs from each container (the sidecar is the usual culprit)
|
|
kubectl logs webserver -c httpd-container
|
|
kubectl logs webserver -c sidecar-container
|
|
```
|
|
|
|
Look at the container states in `describe`: `ImagePullBackOff` / `ErrImagePull` points to
|
|
a **bad image name or tag**; `CrashLoopBackOff` with the sidecar exiting `Completed` points
|
|
to a **container with no long-running process**.
|
|
|
|
## Step 2 — The two usual root causes
|
|
|
|
For this specific setup (an `httpd` container plus an `ubuntu` sidecar), the failure is
|
|
almost always one or both of:
|
|
|
|
1. **Image typo** — e.g. `httpd:latst`, `httpd:letest`, or a misspelled `ubuntu` — which
|
|
yields `ErrImagePull` / `ImagePullBackOff`.
|
|
2. **The ubuntu sidecar exits immediately.** `ubuntu:latest` has no long-running entrypoint
|
|
— it starts a shell, finds nothing to do, and exits `0`. Kubernetes sees the container
|
|
terminate and puts the pod in `CrashLoopBackOff` (it keeps restarting a container that
|
|
keeps exiting). A sidecar **must** be given a command that keeps it alive.
|
|
|
|
## Step 3 — Fix (recreate with a corrected manifest)
|
|
|
|
A pod's container image and command are effectively immutable in place, so the clean fix is
|
|
delete and re-create:
|
|
|
|
```bash
|
|
kubectl delete pod webserver
|
|
|
|
kubectl apply -f - <<'EOF'
|
|
apiVersion: v1
|
|
kind: Pod
|
|
metadata:
|
|
name: webserver
|
|
labels:
|
|
app: httpd
|
|
spec:
|
|
containers:
|
|
- name: httpd-container
|
|
image: httpd:latest
|
|
- name: sidecar-container
|
|
image: ubuntu:latest
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- "while true; do echo sidecar running; sleep 5; done"
|
|
EOF
|
|
```
|
|
|
|
## How it works
|
|
|
|
### The heredoc apply pattern
|
|
|
|
- **`kubectl apply -f -`** reads the manifest from **stdin**; nothing written to disk.
|
|
- **`<<'EOF'` (delimiter quoted)** keeps the `while`/`$`-free command literal so the
|
|
container's shell runs it as written.
|
|
|
|
### What the corrected manifest fixes
|
|
|
|
- **`httpd-container` → `image: httpd:latest`** — a valid, correctly-spelled image/tag, so
|
|
the pull succeeds (fixes any `ImagePullBackOff` from a typo).
|
|
- **`sidecar-container` → `command: [...while true...sleep 5...]`** — gives the ubuntu
|
|
container a **long-running foreground process**. Now it never exits, so the pod stays
|
|
`Running` instead of crash-looping. Any equivalent keep-alive works (`sleep infinity`,
|
|
`tail -f /dev/null`); the infinite loop is a clear, portable choice.
|
|
|
|
### Why a sidecar needs a command but httpd doesn't
|
|
|
|
`httpd:latest`'s default entrypoint **is** a long-running server (Apache in the
|
|
foreground), so it stays up on its own. `ubuntu:latest` has no such default — its job here
|
|
is just to accompany the main container, so **you** must supply the process that keeps it
|
|
alive. This asymmetry is the heart of the bug: the same manifest that's fine for httpd
|
|
leaves ubuntu dead on arrival.
|
|
|
|
### "Application accessible"
|
|
|
|
Once both containers stay up, the pod reports `2/2 Running` and Apache serves on its
|
|
default port 80 inside the pod. You can confirm the app responds from within the pod
|
|
(below); exposing it externally would be a separate Service, which this task doesn't ask
|
|
for.
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
# Both containers up: READY should show 2/2
|
|
kubectl get pod webserver
|
|
|
|
# No more restart churn / crash events
|
|
kubectl describe pod webserver | sed -n '/Events/,$p'
|
|
|
|
# httpd actually serving inside the pod
|
|
kubectl exec webserver -c httpd-container -- sh -c 'apt-get -v >/dev/null 2>&1; echo ok' 2>/dev/null || true
|
|
kubectl exec webserver -c sidecar-container -- echo "sidecar alive"
|
|
```
|
|
|
|
Expected — `webserver` in `Running` with `READY 2/2`, no crash-loop events, and both
|
|
`exec` checks succeeding. Apache serving on port 80 within the pod.
|
|
|
|
> If `describe` still shows `ImagePullBackOff` after the fix, the node is offline or
|
|
> rate-limited on the registry — not a manifest problem. If the sidecar still exits,
|
|
> confirm its `command` made it into the spec:
|
|
> `kubectl get pod webserver -o jsonpath='{.spec.containers[1].command}'`. |