Write the Alerts
The checkout service has metrics but no useful alerts. Write four Prometheus alert rules that page for real outages and stay quiet for blips, quiet nights and other services.
The situation
Last week the checkout service had a bad night. One instance was down for an hour, then a deploy made one checkout in five fail with a 500. Customers noticed before anyone on the team did.
Prometheus scrapes the service and there is a dashboard, but alerts.yml has only two rules. One pages for every missed scrape of anything Prometheus watches. The other was copied from a blog post years ago, and nobody trusts it.
The service exports up, a counter of requests by instance, route and status code (http_requests_total), and a histogram of request durations (http_request_duration_seconds). Other services are scraped by the same Prometheus.
Your task
- CheckoutInstanceDown (severity page): a checkout instance fails its scrapes for 3 minutes. One alert per instance.
- CheckoutTargetsMissing (severity page): there has been no up series for the checkout job at all for 5 minutes.
- CheckoutHighErrorRate (severity page): over 5 minutes, more than 5% of all checkout requests got a 5xx answer, at 1 request per second or more, and it has held for 5 minutes. One alert for the whole service.
- CheckoutSlowRequests (severity ticket): the 95th percentile duration of /api/checkout, for all instances together, has been above 500 ms for 10 minutes.
- Every alert has a summary annotation and a runbook_url annotation with an https link. Every selector names one of the given metrics or a recording rule, and label values are plain text.
What the grader checks
| Check | Passes when |
|---|---|
| The rules load | promtool check rules passes, and alerts.yml defines CheckoutInstanceDown, CheckoutTargetsMissing, CheckoutHighErrorRate and CheckoutSlowRequests once each, with the severity label from the brief and a summary and an https runbook_url annotation. No rule uses keep_firing_for, reads a series other than the metrics in the brief and its own recording rules (ALERTS included), reads the clock, uses @ or offset, or looks back more than an hour (also through recording rules), no recording rule reads itself, no group sets its own interval, and label values are plain text, not templates. |
| Instance down | Random scenarios: an instance that stays down alerts after 3 minutes, one alert per instance, even while the others fail and are slow; a 2-minute blip does not alert; an instance of another job does not alert. |
| Targets missing | Random scenarios: when every checkout target disappears, or was never scraped, it alerts after 5 minutes; instances that are down but still scraped and another job's targets do not count. |
| Error rate | Random scenarios: a service-wide 5xx share over 5% across 5 minutes, even just over, at 1 request per second or more, alerts once it has lasted 5 minutes, not sooner or later, and stops after the errors end; whatever the 5xx codes and routes, for a flaky dependency, next to a busy healthy service, and while an instance is down and requests are slow. Under 5%, many 4xx, under 1 request per second, and one small failing route or instance do not alert. |
| Slow requests | Random scenarios: a p95 for checkout's /api/checkout over 500 ms, even only about 750 ms, alerts once it has lasted 10 minutes, not sooner or later, with severity ticket, even next to a busy fast service and while an instance is down and many requests fail. A p95 under 500 ms (even about 375 ms), a slow route or job elsewhere, one small slow instance and a 7-minute slowdown do not alert. |
When a check fails, the grader's comment says what it saw, not how to fix it.
Files you may change
challenges/write-the-alerts/alerts.yml
A pull request that changes any other file, including the tests or the workflows, is not graded.
Hints
One missed scrape is not an outage
An alert can wait before it fires. Read about the for field in the Prometheus alerting rules documentation.
A target that is gone has no up series
up == 0 needs an up series to compare. When every target disappears, there is nothing to compare. Look at absent().
A ratio for the whole service
rate() gives requests per second per series. Add up the 5xx rates and divide by the sum of all rates to get one ratio for the service. Two conditions can be joined with and.
Percentiles from a histogram
histogram_quantile() works on bucket rates. When you sum the buckets of several instances, keep the le label.
Test it locally
From the challenge folder, with promtool (it ships with Prometheus) and Python 3:
promtool check rules alerts.yml
promtool test rules tests/example.test.yml
python3 grade/gen_tests.py --out /tmp/alert-tests --rules "$PWD/alerts.yml"
for f in /tmp/alert-tests/*.yml; do promtool test rules "$f" > /dev/null || echo "FAILED: $f"; doneSubmit
- Fork The-DevOps-Daily/challenges and create a branch.
- Commit your changes to the files listed above, and nothing else.
- Open a pull request against main in The-DevOps-Daily/challenges.
- Read the grader's comment. If a check fails, push another commit to the same branch and it grades again.
- When every check passes, the PR gets the passed label and is closed. It is never merged, and your pass is usually counted on this page within a few minutes.
The workflow takes the tests and the grader from the repository's main branch and copies only the files above out of your PR. Your code runs in a container or a throwaway kind cluster, and some inputs are random at grading time. A second workflow checks the result against your PR's real commit before it comments.
Learn the basics first
Exercise
Complete Monitoring Stack Setup
Deploy and configure Prometheus and Grafana for comprehensive application and infrastructure monitoring.
150 minutes
Exercise
Microservices Observability with OpenTelemetry
Implement comprehensive observability for microservices using OpenTelemetry for distributed tracing, metrics, and logging across multiple services.
140 minutes
We earn commissions when you shop through the links below.
More challenges
Docker · Starter
Shrink the Image
A tiny Flask app ships in a container image of more than 1 GB that runs as root. Rewrite the Dockerfile so the image is under 200 MB, runs as a normal user and still works.
About 30 min
Bash · Starter
Script It: Summarize an Access Log
Finish a half-written Bash script that turns an nginx access log into a short report: requests, status classes, top clients and top paths. ShellCheck has to be clean and the edge cases right.
About 45 min
Kubernetes · Intermediate
Fix the Deployment
An inventory API never comes up on Kubernetes. Find and fix the bugs in its Deployment, Service and ConfigMap until the rollout finishes and the Service answers.
About 45 min
PostgreSQL · Intermediate
Speed Up the Slow Queries
Three queries behind the shop's busiest pages read whole tables. Add the right PostgreSQL indexes, and fix the query an index cannot help, so each one reads a few pages and returns the same rows.
About 45 min
Terraform · Intermediate
Fix the Network Module
A Terraform module for a VPC and its subnets broke in a refactor: it does not validate, its subnets overlap and it accepts any input. Fix it until its tests and random plans pass, with no AWS account.
About 60 min