Exploring a Docker Container's File System
When your containerized application starts behaving unexpectedly - crashing on startup, serving 500 errors, or consuming too much memory - you need to look inside the container to understand what's happening. Unlike traditional servers where you can SSH in and explore, containers require different approaches to inspect their file systems, examine running processes, and analyze logs. This guide will show you several techniques for investigating container internals, from basic file exploration to advanced debugging strategies.
Prerequisites
To follow along with this guide, you'll need:
- Docker installed and running on your system
- At least one running container to practice with
- Basic familiarity with Linux command-line tools
- Understanding of file system navigation concepts
If you don't have a container running, start one for practice:
# Start a simple web server container for testing
docker run -d --name debug-practice nginx:alpine
Understanding Container File System Architecture
Before diving into exploration techniques, it's helpful to understand how Docker organizes container file systems. Containers use a layered approach where each layer represents changes from the previous layer:
Container File System Structure:
┌─────────────────────────────┐
│ Writable Layer │ ← Runtime changes, logs, temp files
├─────────────────────────────┤
│ Application Layer │ ← Your app code, configs
├─────────────────────────────┤
│ Dependencies Layer │ ← npm install, pip install results
├─────────────────────────────┤
│ Base OS Layer │ ← Ubuntu, Alpine, etc.
└─────────────────────────────┘
This layered structure affects where files appear and how changes persist. Files in lower layers are read-only, while the top writable layer captures all runtime modifications.
Interactive Shell Access
The most direct way to explore a container is launching an interactive shell inside it. This gives you the same experience as logging into a Linux server, allowing you to navigate directories, examine files, and run diagnostic commands.
Basic Shell Access
When you need to quickly check file contents, examine directory structures, or run commands inside a container, docker exec provides immediate access. This command works with any running container and doesn't affect the container's primary process.
# Launch an interactive shell in a running container
docker exec -it debug-practice /bin/sh
# For containers with bash available, use bash for better features
docker exec -it debug-practice /bin/bash
The -i flag keeps STDIN open for interaction, while -t allocates a pseudo-TTY for a proper terminal experience. Together, they create the familiar command-line interface you expect.
Once inside, you can navigate and explore like any Linux system:
# Inside the container shell
pwd # See current directory
ls -la # List files with permissions
ps aux # View running processes
cat /etc/os-release # Check the base OS
Troubleshooting Shell Access Issues
Sometimes the standard shell paths don't work, especially in minimal containers. Different base images use different shells, and some distroless images might not include a shell at all. Here's how to handle various scenarios:
# Try different shell locations if /bin/sh fails
docker exec -it myapp-container /bin/bash
docker exec -it myapp-container /bin/ash
docker exec -it myapp-container sh
# For Alpine-based containers, ash is usually available
docker exec -it alpine-container /bin/ash
# Check what shells are available in the container
docker exec myapp-container ls -la /bin/*sh
If no shell is available (common in distroless images), you can still run individual commands:
# Run single commands without shell access
docker exec myapp-container ls -la /app
docker exec myapp-container cat /etc/passwd
docker exec myapp-container ps aux
File and Directory Navigation
Once you have shell access, effective navigation helps you quickly locate relevant files. Understanding common directory structures in containerized applications speeds up your debugging process.
Common Container Directory Layouts
Most containerized applications follow predictable directory patterns. Web applications typically store code in /app, system services use /usr/local, and logs often appear in /var/log. Understanding these conventions helps you navigate faster.
# Navigate to common application directories
cd /app # Many applications install here
ls -la # Check application files
cd /var/log # Standard location for log files
ls -la # Look for application-specific logs
cd /etc # Configuration files
ls -la # Check for custom configs
cd /tmp # Temporary files and runtime data
ls -la # Might contain debug files or temp data
Examining File Permissions and Ownership
File permission issues cause many container problems. Applications might fail to read configuration files, write logs, or create temporary files due to permission misconfigurations. Examining ownership and permissions helps identify these issues.
# Check file ownership and permissions in detail
ls -la /app/
# Look for permission issues with specific files
ls -la /var/log/app.log
ls -la /etc/myapp/config.json
# Check what user the application runs as
whoami
id
# See all users in the container
cat /etc/passwd
Look for mismatches between file ownership and the user running your application. If your app runs as user node but files are owned by root, you might have permission problems.
Finding Recently Modified Files
When debugging issues that started recently, finding files that changed around the time problems began can provide valuable clues. This technique helps you identify configuration changes, log rotations, or corrupted files.
# Find files modified in the last 24 hours
find /app -type f -mtime -1 -ls
# Find files modified in the last hour
find /var/log -type f -mmin -60 -ls
# Look for recently created temporary files
find /tmp -type f -mtime -1 -ls
# Check for large files that might indicate issues
find /app -type f -size +100M -ls
This is particularly useful when investigating memory leaks (look for growing log files) or configuration issues (check for recently modified config files).
Copying Files Between Container and Host
Sometimes you need to analyze files with tools that aren't available inside the container, or you want to preserve logs and configuration files for later analysis. Docker's copy command enables bidirectional file transfer without stopping the container.
Extracting Files for Analysis
When you need to examine files with external tools - perhaps analyzing logs with advanced text processors, checking binary files with hexdump, or comparing configuration files with known good versions - copying files to your host system provides more analysis options.
# Copy a single file from container to host
docker cp debug-practice:/var/log/nginx/access.log ./nginx-access.log
# Copy entire directory structures
docker cp debug-practice:/etc/nginx ./container-nginx-config
# Copy files with preserved directory structure
docker cp debug-practice:/var/log ./container-logs
The copy command preserves file permissions and ownership information when possible, giving you an exact replica of the container's file state.
Copying Multiple Files Efficiently
When investigating complex issues, you often need multiple files - logs, configuration files, and application data. Instead of copying files individually, you can create archives inside the container and copy the archive out in one operation.
# Create an archive of multiple important directories inside the container
docker exec debug-practice tar czf /tmp/debug-info.tar.gz \
/var/log \
/etc/nginx \
/app/config
# Copy the archive to your host
docker cp debug-practice:/tmp/debug-info.tar.gz ./
# Extract and examine on your host system
tar -xzf debug-info.tar.gz
This approach is faster for large amounts of data and maintains relative directory structures.
Copying Files Into Containers
Sometimes you need to temporarily add diagnostic tools, configuration files, or test data to a running container. While this changes the container's state, it can be valuable for debugging without rebuilding images.
# Copy a diagnostic script into the container
docker cp ./debug-script.sh debug-practice:/tmp/
# Copy updated configuration for testing
docker cp ./test-config.json debug-practice:/app/config/
# Make copied scripts executable
docker exec debug-practice chmod +x /tmp/debug-script.sh
Remember that files copied into containers are lost when the container is removed, so this technique is for temporary debugging only.
Analyzing Container Layers and History
Understanding how your container was built helps you locate files and understand why certain configurations exist. Docker's layering system means files might come from base images, dependency installations, or your application code, and knowing the source helps target your investigation.
Examining Image Build History
Every Docker image contains metadata about how it was constructed. This build history shows you the commands that created each layer, helping you understand where files originated and why certain configurations exist.
# View the complete build history of an image
docker history nginx:alpine
# Get detailed information about each layer
docker history --no-trunc nginx:alpine
# See only the layers without truncation for better readability
docker history --format "table {{.CreatedBy}}\t{{.Size}}" nginx:alpine
The history output shows you commands like RUN apt-get install, COPY . /app, and ENV declarations. This information helps you understand where specific files or configurations came from.
Understanding Layer Composition
Each layer in a Docker image represents a filesystem changeset. When debugging, knowing which layer contains specific files helps you understand whether issues come from the base image, dependencies, or your application code.
# Inspect detailed image information including layers
docker inspect nginx:alpine
# Use jq to extract just the layer information (if jq is available)
docker inspect nginx:alpine | grep -A 50 "RootFS"
You can also explore individual layers by creating temporary containers from specific points in the build history, though this is an advanced technique typically used for deep debugging.
Comparing Images
When debugging issues that appeared after an image update, comparing the old and new image configurations can reveal what changed. This helps you identify whether issues stem from base image updates, dependency changes, or application modifications.
# Compare configurations between image versions
docker inspect myapp:v1.0 > old-config.json
docker inspect myapp:v1.1 > new-config.json
diff old-config.json new-config.json
# Compare image histories to see build differences
docker history myapp:v1.0 > old-history.txt
docker history myapp:v1.1 > new-history.txt
diff old-history.txt new-history.txt
Advanced File System Investigation
Beyond basic navigation and file copying, several advanced techniques help you understand complex container behavior, especially when dealing with performance issues, resource constraints, or subtle configuration problems.
Monitoring File System Usage
Container file systems can fill up due to growing log files, temporary data, or memory-mapped files. Monitoring disk usage helps identify space-related issues before they cause application failures.
# Check disk usage in the container
docker exec debug-practice df -h
# Find the largest directories
docker exec debug-practice du -sh /* | sort -hr
# Identify large files specifically
docker exec debug-practice find / -type f -size +10M -exec ls -lh {} \; 2>/dev/null
# Monitor real-time file system changes
docker exec debug-practice inotifywait -m -r -e create,modify,delete /app
These commands help you understand whether disk space issues are causing problems or if certain directories are growing unexpectedly.
Examining Running Processes and Open Files
Understanding what processes are running inside your container and what files they have open provides insight into application behavior, resource usage, and potential issues like file handle leaks.
# View all running processes with detailed information
docker exec debug-practice ps aux
# See what files processes have open
docker exec debug-practice lsof
# Check network connections
docker exec debug-practice netstat -tulpn
# Monitor process activity in real-time
docker exec debug-practice top
This information helps you understand whether your application is behaving as expected and can reveal issues like memory leaks, hanging processes, or unexpected network connections.
Investigating Environment and Configuration
Many container issues stem from environment variable problems, missing configuration files, or incorrect PATH settings. Examining the runtime environment helps identify these configuration-related problems.
# Check all environment variables
docker exec debug-practice env | sort
# Verify PATH configuration
docker exec debug-practice echo $PATH
# Check specific application environment variables
docker exec debug-practice printenv | grep -i app
# Look for configuration files in common locations
docker exec debug-practice find /etc -name "*.conf" -o -name "*.cfg" -o -name "*.json"
File System Forensics
When investigating security incidents or understanding how an application modifies its environment, examining file access times, modification patterns, and file integrity can provide valuable insights.
# Check file access times to see what's being read
docker exec debug-practice find /app -type f -atime -1 -ls
# Look for recently modified configuration files
docker exec debug-practice find /etc -type f -mtime -7 -ls
# Check for files with unusual permissions
docker exec debug-practice find /app -type f \( -perm -4000 -o -perm -2000 \) -ls
# Verify file checksums if you have known good values
docker exec debug-practice sha256sum /app/important-file.txt
Debugging Common File System Issues
Understanding common container file system problems and their solutions helps you quickly resolve issues when they occur.
Permission Problems
Permission issues frequently cause container failures. Applications might not be able to read configuration files, write logs, or create temporary files due to user or group misconfigurations.
# Identify the user your application runs as
docker exec myapp-container ps aux | grep -v grep
# Check ownership of important files
docker exec myapp-container ls -la /app/config/
docker exec myapp-container ls -la /var/log/
# Test write permissions in directories your app uses
docker exec myapp-container touch /tmp/test-write
docker exec myapp-container touch /var/log/test-write
If you find permission issues, you might need to rebuild your image with proper user configuration or adjust file ownership in your Dockerfile.
Missing Files and Dependencies
Applications sometimes fail because expected files don't exist or dependencies weren't properly installed. Systematic checking helps identify these issues.
# Verify application files exist
docker exec myapp-container ls -la /app/
# Check for required configuration files
docker exec myapp-container test -f /app/config.json && echo "Config exists" || echo "Config missing"
# Verify shared libraries are available
docker exec myapp-container ldd /app/myapp
# Check for required system tools
docker exec myapp-container which curl
docker exec myapp-container which npm
Resource Constraints
File system issues can stem from resource constraints - insufficient disk space, too many open files, or inode exhaustion. These problems often manifest as seemingly unrelated application errors.
# Check available disk space
docker exec myapp-container df -h
# Monitor inode usage
docker exec myapp-container df -i
# Check file descriptor limits
docker exec myapp-container ulimit -n
# See how many files are currently open
docker exec myapp-container lsof | wc -l
Best Practices for Container File System Investigation
Following consistent practices makes your debugging sessions more effective and helps you build institutional knowledge about your containerized applications.
Maintain Investigation Logs
Keep notes about what you find during investigations. This documentation helps with future debugging sessions and can reveal patterns across different incidents.
# Create investigation notes with timestamps
echo "$(date): Started investigating container myapp-prod" >> investigation.log
echo "$(date): Found high CPU usage in process PID 1234" >> investigation.log
# Capture command outputs for later analysis
docker exec myapp-container ps aux > processes-$(date +%Y%m%d-%H%M).txt
docker exec myapp-container df -h > disk-usage-$(date +%Y%m%d-%H%M).txt
Use Read-Only Investigation
When possible, avoid making changes to running containers during investigation. This preserves the original state for analysis and prevents accidental modifications from masking the real problem.
# Use read-only commands for investigation
docker exec myapp-container cat /var/log/app.log
docker exec myapp-container ls -la /app/
# If you must make changes, document them
echo "$(date): Added debug flag to config" >> investigation.log
Automate Common Investigation Tasks
Create scripts for common debugging tasks to speed up future investigations and reduce the chance of forgetting important checks.
#!/bin/bash
# container-debug.sh
CONTAINER_NAME=$1
echo "=== Container Information ==="
docker inspect $CONTAINER_NAME | grep -E "(State|Image|Created)"
echo "=== Process Information ==="
docker exec $CONTAINER_NAME ps aux
echo "=== Disk Usage ==="
docker exec $CONTAINER_NAME df -h
echo "=== Recent Log Entries ==="
docker logs --tail 50 $CONTAINER_NAME
echo "=== Environment Variables ==="
docker exec $CONTAINER_NAME env | sort
Understanding how to navigate and investigate Docker container file systems is essential for maintaining containerized applications. These techniques help you quickly identify and resolve issues, from simple configuration problems to complex performance issues. The key is combining multiple approaches - interactive exploration, file analysis, and systematic investigation - to build a complete picture of what's happening inside your containers. As you become more comfortable with these tools, you'll find that container debugging becomes much more manageable and systematic.
Found an issue?