Looping Through File Content Line by Line in Bash

Reading file content line by line is a fundamental operation in Bash scripting. Whether you're processing log files, configuration files, or data sets, understanding the different approaches and their nuances will help you write more robust scripts.

The Standard while read Loop

The most common and reliable method uses a while loop with the read command:

while IFS= read -r line; do
    echo "Processing: $line"
done < "data.txt"

Let's break down this syntax:

• IFS= prevents leading/trailing whitespace from being trimmed
• -r prevents backslash escapes from being interpreted
• < "data.txt" redirects the file content to the loop

This method handles most edge cases correctly and is the recommended approach for file processing.

Processing Files with Special Characters

When your file contains special characters, backslashes, or unusual whitespace, the standard method preserves the content exactly:

# Create a test file with special content
cat > special_content.txt << 'EOF'
Line with    multiple spaces
Line with\backslashes
Line with "quotes" and 'apostrophes'
    Indented line
EOF

# Process while preserving formatting
while IFS= read -r line; do
    echo "[$line]"
done < special_content.txt

The square brackets in the output will show you exactly what each line contains, including preserved whitespace.

Using Process Substitution

For complex file processing where you need to maintain variables outside the loop, use process substitution:

counter=0
total_lines=0

while IFS= read -r line; do
    counter=$((counter + 1))
    total_lines=$((total_lines + 1))
    echo "Line $counter: $line"
done < <(cat "data.txt")

echo "Processed $total_lines lines total"

This approach prevents the subshell issue where variables modified inside a pipeline don't persist outside it.

Reading from Command Output

You can also loop through the output of commands:

# Process the output of a command line by line
while IFS= read -r line; do
    echo "File: $line"
    # Get file size
    size=$(stat -f%z "$line" 2>/dev/null || stat -c%s "$line" 2>/dev/null)
    echo "  Size: $size bytes"
done < <(find /home/user/documents -name "*.txt")

This technique is useful for processing lists generated by commands like find, grep, or ls.

Handling Different File Formats

For CSV files or structured data, you can parse fields within each line:

# Process a CSV file
while IFS=',' read -r name age city; do
    echo "Name: $name"
    echo "Age: $age"
    echo "City: $city"
    echo "---"
done < "users.csv"

For files with different delimiters:

# Process colon-separated data (like /etc/passwd)
while IFS=':' read -r username password uid gid comment home shell; do
    echo "User: $username (UID: $uid, Home: $home)"
done < "/etc/passwd"

Error Handling and File Validation

Always verify that files exist and are readable before processing:

filename="data.txt"

if [[ ! -f "$filename" ]]; then
    echo "Error: File $filename does not exist"
    exit 1
fi

if [[ ! -r "$filename" ]]; then
    echo "Error: File $filename is not readable"
    exit 1
fi

line_count=0
while IFS= read -r line || [[ -n "$line" ]]; do
    line_count=$((line_count + 1))

    # Skip empty lines
    [[ -z "$line" ]] && continue

    # Skip comment lines
    [[ "$line" =~ ^[[:space:]]*# ]] && continue

    echo "Processing line $line_count: $line"
done < "$filename"

echo "Successfully processed $line_count lines"

The || [[ -n "$line" ]] condition ensures the last line is processed even if it doesn't end with a newline character.

Processing Large Files Efficiently

For very large files, you might want to add progress indicators or process files in chunks:

filename="large_data.txt"
total_lines=$(wc -l < "$filename")
current_line=0

while IFS= read -r line; do
    current_line=$((current_line + 1))

    # Show progress every 1000 lines
    if (( current_line % 1000 == 0 )); then
        percentage=$((current_line * 100 / total_lines))
        echo "Progress: $current_line/$total_lines lines ($percentage%)"
    fi

    # Your processing logic here
    process_line "$line"

done < "$filename"

Real-World Example: Log Analysis

Here's a practical example that analyzes web server logs:

#!/bin/bash

log_file="/var/log/apache2/access.log"
error_count=0
request_count=0

while IFS= read -r line; do
    request_count=$((request_count + 1))

    # Extract status code (assuming standard Apache log format)
    status_code=$(echo "$line" | awk '{print $9}')

    # Count error responses (4xx and 5xx)
    if [[ "$status_code" =~ ^[45][0-9][0-9]$ ]]; then
        error_count=$((error_count + 1))
        echo "Error found: $line"
    fi

    # Extract IP address for further analysis
    ip_address=$(echo "$line" | awk '{print $1}')

    # Log suspicious activity (example: too many requests from same IP)
    # This would require additional logic to track IPs over time

done < "$log_file"

echo "Analysis complete:"
echo "Total requests: $request_count"
echo "Error responses: $error_count"
echo "Error rate: $(( error_count * 100 / request_count ))%"

Alternative Approaches

While the while read loop is generally preferred, you can also use other methods for specific use cases:

# Using mapfile (Bash 4+) to read entire file into array
mapfile -t lines < "data.txt"
for line in "${lines[@]}"; do
    echo "Processing: $line"
done

# Using for loop with command substitution (not recommended for large files)
for line in $(cat "data.txt"); do
    echo "Word: $line"  # Note: this splits on whitespace, not lines
done

The while read approach remains the most reliable method for line-by-line file processing, especially when dealing with files containing spaces, special characters, or varying line lengths.