On-Call: The Daily Incident
A new production incident every day. Six moves to find the root cause: read logs, metrics and kubectl output, then name the cause. Same incident for everyone. Keep a streak and share a spoiler-free result, like a daily word puzzle for on-call engineers.
Category: SRE
What You Will Learn
- Practise diagnosing production incidents with a limited budget of moves
- Gather evidence before you guess: logs, metrics, kubectl, dig, curl and openssl output
- Tell a symptom apart from its root cause
- Recognise connection pool exhaustion, lock queues, replication lag and full disks
- Spot DNS, TLS, networking, queue, cache and deploy failures from their signals
- A new incident every day, with a streak and a spoiler-free result to share
Topics covered: game, sre, on-call, incident-response, troubleshooting, debugging, daily-puzzle, interactive
// simulator
On-Call: The Daily Incident
A new production incident every day. Six moves to find the root cause: read logs, metrics and kubectl output, then name the cause. Same incident for everyone. Keep a streak and share a spoiler-free result, like a daily word puzzle for on-call engineers.
About On-Call: The Daily Incident
What you'll practise
- Reading an alert and deciding which signal to check first
- Reading real-looking output from kubectl, psql, dig, curl, openssl and logs
- Telling a symptom, such as high CPU, apart from what caused it
- Ruling causes out with evidence instead of guessing
- Learning how each failure is fixed and how to stop it coming back
How it works
- One incident a day: everyone gets the same incident, and a new one arrives at 00:00 UTC.
- Six moves: opening a signal costs a move, and so does naming a cause. Your last move is kept for a diagnosis.
- Feedback: green is the root cause, yellow is the right area but not the root cause, and grey is the wrong area.
- Your streak: it stays in your browser. Copy the backup code to keep it if you clear your browser or switch devices.
Evidence first, then a diagnosis
Most wrong turns during an incident come from naming a cause too early. A busy database, a recent deploy or a noisy log line all look guilty. The fastest responders pick the one signal that would prove or rule out their first idea, read it, and only then act. The move budget in this game rewards exactly that.
Read more
To run a whole incident as the person in charge, try the Incident Commander Simulator. For the process around the pager, read how to build an on-call rotation and escalation policy and how to write a postmortem people read. If a connection pool incident catches you out, read what a connection pool really saves you.
Overview
Most on-call work is reading signals under pressure. A page fires, an error rate jumps, and you have to choose which logs, metrics and commands to check before you name a cause. On-Call: The Daily Incident turns that into a daily puzzle. Every day, everyone gets the same production incident: the alert that paged you and a few symptoms. You have six moves. Opening a signal, such as kubectl output, a Postgres query, dig, curl, openssl or a load balancer log, costs one move, and naming a cause costs one move too. Green means you found the root cause, yellow means the cause is in the right area but did not cause this incident, and grey means the wrong area. After the round you see what happened and how to stop it happening again. The incidents cover Kubernetes, DNS, TLS certificates, databases, CI/CD, cloud networking, memory, cron and time zones, rate limits, caches, queues, feature flags, and deploys and rollbacks. The habit it trains is to gather the right evidence before you guess, and to tell a symptom apart from its cause.
Try this
- Read the page and the symptoms, then open only the signal that would best prove or rule out your first idea before you name a cause.
- If a cause comes back yellow, use the area it names to narrow your next pick instead of guessing again.
- Finish the round, open "Review every signal" to see the evidence you skipped, then copy your spoiler-free result with Share result.
Frequently asked questions
- How does the daily incident work?
- Everyone gets the same incident on the same UTC day, and a new one starts at 00:00 UTC. You have six moves: each signal you open and each cause you name uses one.
- What do the colors mean?
- Green is the root cause. Yellow is a cause in the same area as the root cause, for example another database problem, but not the one behind this incident. Grey is a cause in a different area. A magnifying glass marks a move spent on a signal.
- Where is my streak stored?
- Only in your browser. There is no account. Under "Stats and streak backup" you can copy a short backup code, then paste it later to get your streak back after clearing your browser or on another device.
- Are the incidents real?
- They are made up, but each one is built from a failure that teams hit in production, such as connection pool exhaustion, an expired certificate, a DNSSEC mistake or a replication slot that fills a disk. The signals are written to look like real command output.
Try next
// game
Uptime Defender
Fast-paced SRE game where you defend your infrastructure uptime! Handle incoming incidents by adding nodes, rotating logs, failing over, restarting pods, and scaling databases.
// simulator
Incident Commander Simulator
Take the page as incident commander. Direct two responders, read the evidence they bring back, choose a mitigation, keep customers updated, and confirm the recovery holds before you resolve. Turn-based, with a review of every decision at the end.
// simulator
Preview Environment Simulator
See how a pull request becomes a temporary copy of your app, complete with a private URL, safe test data, review checks, and automatic cleanup.