Day 5 - Debug a Broken Container

Description

Your application container keeps crashing immediately after startup. The logs show cryptic errors, and you need to figure out what's wrong. This is a common scenario in production environments where containers fail due to configuration issues, missing dependencies, or runtime errors.

Task

Debug and fix a broken Docker container that crashes on startup.

Requirements:

• Identify why the container is crashing
• Fix the underlying issue
• Verify the container runs successfully
• Document the root cause

Target

• ✅ Container starts and stays running
• ✅ Application responds to health checks
• ✅ No errors in container logs
• ✅ Root cause identified and documented

Sample App

Broken Dockerfile

FROM node:18-alpine

WORKDIR /app

# Missing package.json copy!
RUN npm install

COPY . .

# Wrong port exposed
EXPOSE 8080

# Incorrect startup command
CMD ["node", "server.js"]

Application Code (server.js)

const express = require('express');
const app = express();

// Port from environment variable
const PORT = process.env.PORT || 3000;

app.get('/health', (req, res) => {
  res.json({ status: 'healthy' });
});

app.get('/', (req, res) => {
  res.send('Hello from fixed container!');
});

app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

package.json

{
  "name": "broken-app",
  "version": "1.0.0",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.18.0"
  }
}

Result

After fixing the issues:

# Build the fixed image
docker build -t fixed-app:latest .

# Run the container
docker run -d -p 3000:3000 --name my-app fixed-app:latest

# Verify it's running
docker ps
# CONTAINER ID   IMAGE              STATUS                    PORTS
# xyz789abc123   fixed-app:latest   Up 30 seconds (healthy)   0.0.0.0:3000->3000/tcp

# Test the application
curl http://localhost:3000
# Hello from fixed container!

curl http://localhost:3000/health
# {"status":"healthy"}

# Check logs
docker logs my-app
# Server running on port 3000

Validation

Test Checklist

# 1. Container starts successfully
docker ps | grep my-app
# Should show "Up" status

# 2. Health check passes
docker inspect my-app | jq '.[0].State.Health.Status'
# Should return "healthy"

# 3. Application responds
curl -f http://localhost:3000/health
# Should return 200 OK

# 4. No errors in logs
docker logs my-app --tail 50
# Should show normal startup messages

# 5. Container handles restart
docker restart my-app && sleep 5 && curl http://localhost:3000
# Should respond after restart

Advanced Debugging

Use Multi-Stage Builds for Debugging

# Debug stage
FROM node:18-alpine AS debug
WORKDIR /app
COPY package*.json ./
RUN npm install  # Include dev dependencies
COPY . .
CMD ["npm", "run", "dev"]

# Production stage
FROM node:18-alpine AS production
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
CMD ["npm", "start"]

Build debug version:

docker build --target debug -t app:debug .

Docker Events Monitoring

# Monitor container events
docker events --filter container=my-app

# Monitor all events
docker events --since '2023-12-05T00:00:00' --until '2023-12-05T23:59:59'

Performance Debugging

# Limit resources for testing
docker run -d \
  --memory="256m" \
  --cpus="0.5" \
  --name resource-limited \
  fixed-app:latest

# Monitor resource usage
docker stats resource-limited

Best Practices

✅ Do's

1. Use health checks: Detect issues early
2. Check logs first: Most issues are logged
3. Use .dockerignore: Prevent unnecessary files
4. Layer caching: Order Dockerfile for optimal caching
5. Run as non-root: Security best practice

❌ Don'ts

1. Don't ignore exit codes: They indicate failure types
2. Don't use :latest in prod: Pin versions
3. Don't run as root: Security risk
4. Don't store secrets in images: Use secrets management
5. Don't use ADD when COPY works: Less confusion

Links

• Docker Debugging Guide
• Dockerfile Best Practices
• Docker Health Checks
• Container Troubleshooting
• Docker CLI Reference

Share Your Success

Fixed your broken container? Share your debugging journey!

Tag @thedevopsdaily on X with:

• What was broken
• How you debugged it
• The fix you applied
• Screenshot of healthy container

Use hashtags: #AdventOfDevOps #Docker #Debugging #Day5