Script It: Summarize an Access Log
Finish a half-written Bash script that turns an nginx access log into a short report: requests, status classes, top clients and top paths. ShellCheck has to be clean and the edge cases right.
The situation
When the site is slow, the first question on the incident call is always the same: who is hitting us, which pages, and how many errors?
Someone started summarize.sh to answer it from an nginx access log, then got pulled onto something else. Right now it counts lines and not much more, and ShellCheck is not happy with it.
Your task
- Print the report described in the challenge README for a log in the combined format: requests, unique IPs, counts per status class, skipped lines, and the top IPs and paths.
- Read the log from a file, or from stdin when no file is given or the file is -.
- Support -n N for the length of the top lists, and exit 2 on usage errors and 1 on unreadable files, with messages on stderr.
- Keep ShellCheck clean, without disable comments.
- Summarize 100,000 lines in under 10 seconds.
What the grader checks
| Check | Passes when |
|---|---|
| ShellCheck is clean | ShellCheck 0.11 reports no issues at any severity, and the script does not switch checks off with disable directives. |
| Report for a known log | Prints the exact report for a small log with a known answer. |
| Edge cases | Skips malformed lines, ignores empty lines, strips query strings, breaks ties in byte order, counts a last line with no newline and handles an empty log. |
| File or stdin | Reads the log from a file, from stdin when no file is given or the file is -, and from a file name with a space in it. |
| Options and usage errors | Supports -n N for the length of the top lists, and exits 2 with a message on stderr for a bad -n value, an unknown option or more than one file. |
| Unreadable input | Exits 1 with a message on stderr, and prints nothing on stdout, when the log file is missing, unreadable or a directory. |
| Random logs | Gives the right report for three logs generated at random during grading. Their names and answers are not visible to the script. |
| Speed | Summarizes a random 100,000-line log correctly in under 10 seconds, including about a second to start the container. |
When a check fails, the grader's comment says what it saw, not how to fix it.
Files you may change
challenges/script-it/summarize.sh
A pull request that changes any other file, including the tests or the workflows, is not graded.
Hints
Getting the edge cases right
Match each line against the whole pattern before you use any of its fields, so a malformed line is skipped instead of counted with wrong values.
Speed
A while read loop that does the work in bash is much slower than awk on 100,000 lines, and starting cut or grep once per line is slower still. Let one awk program do the reading and counting, and sort do the ranking.
The ShellCheck warnings in the starter
Look up each code on the ShellCheck wiki. The fix is in the code, not in a disable comment.
set -o pipefail and head
sort | head -n 5 can kill sort with SIGPIPE when there is a lot of output. With pipefail and set -e that ends the script early. Test with a big log.
Test it locally
From the challenge folder, with bash, Python 3, ShellCheck and bats-core 1.5 or later:
./summarize.sh sample/access.log
shellcheck summarize.sh
SANDBOX=none SOLUTION_DIR="$PWD" bats grade/summarize.batsSubmit
- Fork The-DevOps-Daily/challenges and create a branch.
- Commit your changes to the files listed above, and nothing else.
- Open a pull request against main in The-DevOps-Daily/challenges.
- Read the grader's comment. If a check fails, push another commit to the same branch and it grades again.
- When every check passes, the PR gets the passed label and is closed. It is never merged, and your pass is usually counted on this page within a few minutes.
The workflow takes the tests and the grader from the repository's main branch and copies only the files above out of your PR. Your code runs in a container or a throwaway kind cluster, and some inputs are random at grading time. A second workflow checks the result against your PR's real commit before it comments.
Learn the basics first
Exercise
Linux Server Monitoring and Maintenance Essentials
Learn to monitor system resources, manage logs, configure automatic security updates, and set up basic alerting with cron jobs on your Ubuntu server.
45 minutes
Exercise
Linux Performance Tuning and Troubleshooting
Master essential Linux performance analysis and optimization techniques to diagnose and resolve system bottlenecks.
80 minutes
Guide
Introduction to Bash
Learn Bash scripting step by step with real output: expansions, quoting, arrays, functions, arguments, strict mode, traps, ShellCheck and bats tests.
Guide
Introduction to Linux
Learn Linux step by step: the command line, files and permissions, apt and dnf, users and sudo, processes, networking, Bash scripts and systemd.
More challenges
Docker · Starter
Shrink the Image
A tiny Flask app ships in a container image of more than 1 GB that runs as root. Rewrite the Dockerfile so the image is under 200 MB, runs as a normal user and still works.
About 30 min
Kubernetes · Intermediate
Fix the Deployment
An inventory API never comes up on Kubernetes. Find and fix the bugs in its Deployment, Service and ConfigMap until the rollout finishes and the Service answers.
About 45 min