Log Aggregation Pipeline Simulator
Follow production logs from applications through collection, parsing, filtering, buffering, sharded indexing, and search. Trigger traffic spikes, parser failures, and slow indexing to see where backpressure builds and logs can be lost.
Category: Observability
What You Will Learn
- How application logs move through collectors, processors, buffers, indexes, and search
- Why parsing and filtering happen before durable indexing
- How bounded buffers absorb short bursts but eventually overflow
- How traffic spikes, parser mismatches, and slow indexing create different failure signals
- How document routing distributes indexed logs across search shards
- Why filtered, rejected, and dropped events never appear in search results
Topics covered: logs, observability, elasticsearch, fluent-bit, parsing, backpressure, monitoring, sre, educational, interactive
// simulator
Log Aggregation Pipeline Simulator
Follow production logs from applications through collection, parsing, filtering, buffering, sharded indexing, and search. Trigger traffic spikes, parser failures, and slow indexing to see where backpressure builds and logs can be lost.
Production log pipeline
Capacity is keeping up and accepted logs are progressing toward search.
Cycle 1 · next stage
Select a stage to inspect what it does.
Next: Apps & hosts
Ready. Generate a batch to start the pipeline.
Generated
0
all incoming
Buffered
0
waiting to index
Indexed
0
searchable
Rejected
0
parser mismatch
Dropped
0
lost to overflow
Stage inspector
Apps & hosts
Applications, containers, and hosts emit raw log events.
Operator signal: Watch the incoming rate. A sudden increase is often the first sign of an incident.
Changes require a fresh run.
Reduces storage volume before indexing.
Three primary shards
Events are distributed by document id. Balanced bars mean balanced indexing work.
How a log pipeline stays reliable
1. Collect close to the source
Lightweight agents tail files or container output and forward events. Local queues keep a brief network problem from immediately losing logs.
2. Shape before storage
Parsers extract searchable fields, enrichment adds context, and filters remove known noise. Reject counters expose format changes before they become blind spots.
3. Buffer the mismatch
Collection and indexing rarely run at exactly the same rate. A durable bounded buffer absorbs bursts while backlog alerts give operators time to restore capacity.
What to alert on in production
Monitor input rate, queue age and depth, parser rejection rate, buffer utilization, dropped events, indexing latency, and shard balance together. A single healthy search query does not prove the pipeline is complete: missing logs may have been rejected long before they reached the index.
Try next
// simulator
Prometheus Query Builder (PromQL Playground)
Learn Prometheus Query Language (PromQL) with an interactive playground. Master label filtering, rate calculations, aggregations, and histogram quantiles with real-time query execution.
// simulator
How Docker Works Under the Hood
Watch what really happens when you run docker run -p 8080:80 nginx, one layer at a time. Step down the whole stack: the CLI, the daemon, the registry pull, containerd, the OCI runtime bundle, runc, the running container, and the shared Linux kernel. Every stage shows the real low-level command you can run yourself, so it doubles as a tour of the primitives that make a container: namespaces, cgroups, and runc. A container is not a small VM, and this shows you why.
// simulator
Webhook Delivery Simulator
Send a webhook through a production-style delivery flow. Change endpoint responses, watch retries and backoff, inspect real HMAC-SHA256 signatures, and redeliver the same message to see receiver-side deduplication.