Assignment

Last week, the Nautilus DevOps team deployed a redis app on Kubernetes cluster, which was working fine so far. This morning one of the team members was making some changes in this existing setup, but he made some mistakes and the app went down. We need to fix this as soon as possible. Please take a look.

The deployment name is redis-deployment. The pods are not in running state right now, so please look into the issue and fix the same.

Note: The kubectl utility on the jump-host has been configured to work with the Kubernetes cluster.

Solution

# - incorrect config map name
# - incorrect image on redis

Troubleshoot Deployment — `redis-deployment`

The redis pods won't start after a bad edit. This walks the diagnosis, then the fix for whichever root cause describe reveals.

Step 1 — Diagnose

# Deployment + pod state
kubectl get deployment redis-deployment
kubectl get pods -l app=redis          # adjust selector if different

# The authoritative source: per-container state + Events at the bottom
kubectl describe pod -l app=redis
# (or describe a specific pod name from the get output)

# The full spec, to spot the bad edit
kubectl get deployment redis-deployment -o yaml

Read the container state and Events in describe. The status tells you the class of bug:

Symptom in `describe`	Root cause
`ErrImagePull` / `ImagePullBackOff`	Image name/tag typo (e.g. `redis:alpin` → `redis:alpine`).
`CreateContainerConfigError`	Bad reference — a `configMapKeyRef` / `configMap` volume / `secretKeyRef` pointing at a name or key that doesn't exist.
`CrashLoopBackOff`	Container starts then exits — bad command/args or a config the app rejects.
`Pending` / `FailedScheduling`	An unschedulable request (e.g. `resources.requests` too high, bad nodeSelector).

Step 2 — Fix (match to what you found)

A) Image typo → correct the image

# Confirm the container name and the wrong image
kubectl get deployment redis-deployment \
  -o jsonpath='{.spec.template.spec.containers[0].name}{"  "}{.spec.template.spec.containers[0].image}{"\n"}'

# Fix it (use the real container name = left of '='; correct the image/tag)
kubectl set image deployment/redis-deployment redis-container=redis:alpine

B) Bad configMap / volume reference → correct the name

If a configMap volume or configMapKeyRef points at a mistyped name, fix it in place:

kubectl edit deployment redis-deployment
# find the wrong reference (e.g. name: redis-cofig) and correct it (redis-config),
# matching an existing ConfigMap:
kubectl get configmaps

C) Unschedulable resource request → lower it

# Example: a request of "2" CPU on a small node leaves the pod Pending
kubectl edit deployment redis-deployment
# correct spec.template.spec.containers[].resources.requests to a sane value

Any of these edits changes the pod template and triggers a fresh rollout automatically.

kubectl rollout status deployment/redis-deployment

How it works

Why `describe` is the first move, not a guess

A broken pod advertises its exact failure in its container state and Events. ImagePullBackOff names the image it couldn't pull (revealing a typo); CreateContainerConfigError names the missing ConfigMap/Secret; FailedScheduling states why no node fits. Reading that first tells you which field the "mistake" touched, so you fix one thing instead of shotgunning changes.

Why in-place edits over re-apply here

The fix is a single-field correction on a deployment you didn't author and whose full spec you may not have cleanly. kubectl set image (for the image) and kubectl edit (for a reference or resource value) touch exactly the broken field and leave the rest intact — safer than reconstructing the whole manifest. Each edit updates the pod template, so the Deployment rolls out corrected pods on its own.

Why the pods recover automatically

A Deployment continuously reconciles toward its spec. Once the template is valid (real image, existing ConfigMap, schedulable requests), the ReplicaSet successfully creates pods and they reach Running — no manual pod deletion needed, though you can delete a stuck pod to speed replacement.

Verify

kubectl get deployment redis-deployment          # READY should match desired
kubectl get pods -l app=redis                    # all Running
kubectl describe deployment redis-deployment | sed -n '/Events/,$p'

Expected — redis-deployment READY N/N, all pods Running, and no recurring error events.

Paste the output of kubectl get deployment redis-deployment -o yaml if you want the exact one-line patch — the fix depends on which field was mistyped.

4.7 KiB Raw Blame History