How to Loop Through File Content in Bash

Processing file content line by line is a common requirement in Bash scripting, whether you're analyzing log files, processing configuration data, or batch-processing lists of items. Bash provides several methods to iterate through file content, each with specific advantages depending on your use case and the characteristics of your data.

Prerequisites

You'll need access to a Linux terminal with basic Bash scripting knowledge. These examples work on all systems with Bash, including Linux distributions and macOS.

Basic Line-by-Line Reading

The most straightforward approach uses a while loop with the read command. This script processes each line of a file:

while IFS= read -r line; do
    echo "Processing: $line"
done < input.txt

The IFS= prevents the shell from trimming whitespace, and -r prevents backslash interpretation. This combination preserves the original line content exactly as it appears in the file.

Processing Files with Error Handling

Adding error handling makes your scripts more robust. This example checks if the file exists before processing:

filename="data.txt"

if [[ ! -f "$filename" ]]; then
    echo "Error: File $filename not found"
    exit 1
fi

while IFS= read -r line; do
    # Skip empty lines
    [[ -z "$line" ]] && continue

    echo "Processing: $line"
    # Add your processing logic here
done < "$filename"

This approach prevents common errors and provides clear feedback when files are missing or inaccessible.

Reading Files with Field Separation

When your file contains structured data with delimiters, you can split lines into fields automatically:

while IFS=',' read -r name email department; do
    echo "Name: $name"
    echo "Email: $email"
    echo "Department: $department"
    echo "---"
done < employees.csv

The IFS (Internal Field Separator) set to comma automatically splits each line into variables based on the delimiter. This approach works well with CSV files or any delimited data format.

Processing Large Files Efficiently

For large files, you might want to add progress indicators or process files in chunks:

total_lines=$(wc -l < large_file.txt)
current_line=0

while IFS= read -r line; do
    ((current_line++))

    # Show progress every 1000 lines
    if ((current_line % 1000 == 0)); then
        printf "Progress: %d/%d lines processed\r" "$current_line" "$total_lines"
    fi

    # Process the line
    echo "$line" | grep "pattern" >> filtered_output.txt

done < large_file.txt

echo -e "\nProcessing complete!"

This provides visual feedback during long-running operations and helps you monitor script progress.

Handling Files with Different Encodings

When processing files that might contain special characters or different encodings, ensure your script handles them correctly:

while IFS= read -r line || [[ -n "$line" ]]; do
    # Process the line, handling the last line even if it doesn't end with newline
    echo "Line: $line"
done < <(iconv -f utf-8 -t utf-8 input.txt)

The || [[ -n "$line" ]] condition ensures the last line gets processed even if it doesn't end with a newline character. The iconv command handles character encoding conversion.

Reading Files into Arrays

Sometimes you need to store all lines in memory for random access or multiple processing passes:

readarray -t lines < input.txt

echo "Total lines: ${#lines[@]}"

for i in "${!lines[@]}"; do
    echo "Line $((i+1)): ${lines[i]}"
done

The readarray command loads the entire file into an array, allowing you to access lines by index or process them multiple times without re-reading the file.

Processing Command Output Line by Line

You can apply the same techniques to process command output instead of files:

ps aux | while IFS= read -r line; do
    if [[ "$line" == *"nginx"* ]]; then
        echo "Found nginx process: $line"
    fi
done

This example processes the output of the ps command to find specific processes. The pipe creates a subshell, so variables modified inside the loop won't persist outside it.

Using Process Substitution to Avoid Subshells

To maintain variable scope when processing command output, use process substitution:

count=0

while IFS= read -r line; do
    if [[ "$line" == *"error"* ]]; then
        ((count++))
    fi
done < <(grep -i "error" /var/log/messages)

echo "Found $count error lines"

The < <(...) syntax creates a process substitution that avoids the subshell problem, allowing the count variable to retain its value after the loop.

Reading Files with Custom Record Separators

Sometimes files use delimiters other than newlines to separate records:

while IFS= read -d ';' -r record; do
    # Process each record separated by semicolons
    echo "Record: $record"
done < data.txt

The -d ';' option changes the delimiter from newline to semicolon, allowing you to process files with custom record separators.

Parallel Processing for Performance

For CPU-intensive processing of large files, consider parallel execution:

process_chunk() {
    local chunk_file="$1"
    while IFS= read -r line; do
        # Intensive processing here
        echo "Processed: $line" | tr '[:lower:]' '[:upper:]'
    done < "$chunk_file"
}

# Split file into chunks and process in parallel
split -l 1000 large_file.txt chunk_
for chunk in chunk_*; do
    process_chunk "$chunk" &
done
wait  # Wait for all background processes to complete

# Clean up chunk files
rm chunk_*

This approach splits large files into smaller chunks and processes them simultaneously, significantly reducing processing time for CPU-bound operations.

Error Recovery and Logging

For production scripts, implement comprehensive error handling and logging:

log_file="processing.log"

process_line() {
    local line="$1"
    local line_number="$2"

    # Simulate processing that might fail
    if [[ "$line" == *"invalid"* ]]; then
        echo "Error processing line $line_number: $line" >> "$log_file"
        return 1
    fi

    echo "Successfully processed line $line_number" >> "$log_file"
    return 0
}

line_number=0
error_count=0

while IFS= read -r line; do
    ((line_number++))

    if ! process_line "$line" "$line_number"; then
        ((error_count++))
    fi
done < input.txt

echo "Processing complete. Errors: $error_count. Check $log_file for details."

This framework provides detailed logging and error tracking, making it easier to debug issues and monitor script performance.

Reading Configuration Files

A practical example of processing configuration files with key-value pairs:

declare -A config

while IFS='=' read -r key value; do
    # Skip comments and empty lines
    [[ "$key" =~ ^[[:space:]]*# ]] && continue
    [[ -z "$key" ]] && continue

    # Remove leading/trailing whitespace
    key=$(echo "$key" | xargs)
    value=$(echo "$value" | xargs)

    config["$key"]="$value"
done < config.ini

# Use the configuration
echo "Database host: ${config[database_host]}"
echo "Port: ${config[port]}"

This example demonstrates parsing configuration files while handling comments, whitespace, and storing values in an associative array for easy access.

Next Steps

You can now process file content efficiently using various Bash techniques. Consider exploring more advanced topics like stream processing with awk or sed, handling binary files, or implementing sophisticated parsing for complex file formats like JSON or XML using specialized tools.

Good luck with your file processing scripts!