Day 9 - Fix a Failing Pod

Description

Your pod is stuck in CrashLoopBackOff status. The application worked fine locally with Docker, but something is wrong in the Kubernetes deployment. Time to put on your debugging hat and figure out what's broken.

Task

Debug and fix a failing Kubernetes pod.

Requirements:

• Identify the root cause of pod failure
• Use kubectl debugging commands effectively
• Fix the underlying issue
• Verify the pod runs successfully
• Document the troubleshooting process

Target

• ✅ Pod status changes from CrashLoopBackOff to Running
• ✅ All containers in the pod are ready
• ✅ Application responds to requests
• ✅ Root cause identified and documented

Sample App

Broken Deployment

# broken-deployment.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: broken-app
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: broken-app
data:
  DATABASE_URL: "postgresql://db:5432/mydb"
  API_KEY: ""  # Empty! Will cause issues
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: broken-app
  namespace: broken-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: broken-app
  template:
    metadata:
      labels:
        app: broken-app
    spec:
      containers:
      - name: app
        image: nginx:wrongtag  # Image doesn't exist!
        ports:
        - containerPort: 3000

        env:
        - name: DATABASE_URL
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: DATABASE_URL
        - name: API_KEY
          valueFrom:
            configMapKeyRef:
              name: app-config
              key: API_KEY

        # Wrong port for probe!
        livenessProbe:
          httpGet:
            path: /health
            port: 8080  # App listens on 3000!
          initialDelaySeconds: 5
          periodSeconds: 10

        resources:
          requests:
            memory: "10Gi"  # Too much memory!
            cpu: "8"        # Too many CPUs!
          limits:
            memory: "10Gi"
            cpu: "8"
---
apiVersion: v1
kind: Service
metadata:
  name: broken-app
  namespace: broken-app
spec:
  selector:
    app: broken-app
  ports:
  - port: 80
    targetPort: 3000

Result

Apply Fixed Deployment

# Delete broken deployment
kubectl delete -f broken-deployment.yaml

# Apply fixed version
kubectl apply -f fixed-deployment.yaml

# Watch pods come up
kubectl get pods -n broken-app -w

# Output:
# NAME                         READY   STATUS    RESTARTS   AGE
# fixed-app-5d7f8c9b4d-abc12   1/1     Running   0          15s
# fixed-app-5d7f8c9b4d-def34   1/1     Running   0          15s

Verify Fix

# Check pod status
kubectl get pods -n broken-app
# All should show Running and Ready 1/1

# Check logs
kubectl logs -l app=fixed-app -n broken-app
# Should show normal startup logs

# Test the app
kubectl port-forward -n broken-app svc/fixed-app 8080:80
curl http://localhost:8080
# Should return nginx welcome page

# Check resource usage
kubectl top pod -n broken-app
# Should show reasonable CPU/memory usage

Validation

Complete Debugging Checklist

# 1. Pod is running
kubectl get pod -n broken-app -l app=fixed-app
# STATUS should be "Running"

# 2. No recent restarts
kubectl get pod -n broken-app -l app=fixed-app
# RESTARTS should be 0 or very low

# 3. All containers ready
kubectl get pod -n broken-app -l app=fixed-app
# READY should be 1/1 (or X/X for multi-container)

# 4. No error events
kubectl get events -n broken-app --field-selector type=Warning
# Should be empty or no recent warnings

# 5. Logs show healthy startup
kubectl logs -l app=fixed-app -n broken-app --tail=20
# Should show normal application logs

# 6. Probes passing
kubectl describe pod -n broken-app -l app=fixed-app | grep -A5 "Liveness\|Readiness"
# Should show successful probe results

# 7. Application responds
kubectl run test --rm -it --image=curlimages/curl -- curl http://fixed-app.broken-app.svc.cluster.local
# Should get successful response

Advanced Debugging

Debug with Ephemeral Containers

# Add debug container to running pod (K8s 1.23+)
kubectl debug -it broken-app-xxx -n broken-app --image=busybox --target=app

# Debug with different image
kubectl debug broken-app-xxx -n broken-app --image=ubuntu --share-processes

Debug Node Issues

# Check node status
kubectl get nodes
kubectl describe node <node-name>

# SSH to node (if possible)
kubectl debug node/<node-name> -it --image=ubuntu

Network Debugging

# Run network debug pod
kubectl run netdebug --rm -it --image=nicolaka/netshoot -- /bin/bash

# Inside pod, test connectivity
nslookup fixed-app.broken-app.svc.cluster.local
curl http://fixed-app.broken-app.svc.cluster.local
traceroute fixed-app.broken-app.svc.cluster.local

Best Practices

✅ Do's

1. Check logs first: Most issues show up in logs
2. Use describe: Shows events and state
3. Check resource usage: OOMKilled is common
4. Validate probes: Wrong port is frequent issue
5. Test locally first: Catch issues before K8s
6. Set proper timeouts: Give apps time to start

❌ Don'ts

1. Don't skip previous logs: Crash info is there
2. Don't ignore events: They tell the story
3. Don't guess: Use systematic debugging
4. Don't forget namespaces: Easy to overlook
5. Don't use :latest: Hard to debug versions

Links

• Kubernetes Debugging
• Debug Pods
• Debug Services
• kubectl Cheat Sheet
• Troubleshooting Applications

Share Your Success

Fixed your broken pod? Share the detective work!

Tag @thedevopsdaily on X with:

• What was broken
• How you diagnosed it
• The fix you applied
• Screenshot of healthy pods

Use hashtags: #AdventOfDevOps #Kubernetes #Debugging #Day9