CASE 03 / Prometheus / Grafana / failure simulation
Production Observability & Alert Validation
Testing whether monitoring detects real failure, including deliberately creating load and proving the complete alert path rather than trusting dashboards.
Installing Prometheus and Grafana is easy. Knowing whether the monitoring system will detect a real failure is a different problem. I built coverage across hosts, containers, endpoints, databases and logs, then deliberately created failure conditions.
The symptom is not the layer.
During one high-CPU test, metrics correctly showed the load but the expected alert did not fire. That distinction mattered: metric collection was working, while the alerting logic was not. PromQL label matching prevented the rule from evaluating as intended.
How I worked the problem.
- 01
Generated a controlled failure instead of waiting for an incident.
- 02
Verified the underlying metrics before blaming the alert delivery path.
- 03
Diagnosed the PromQL rule, corrected label matching and repeated the same failure.
- 04
Validated the full chain from failure generation through metrics, rule evaluation and alert firing.
- 05
Used blackbox-style probes so application reachability is checked from outside rather than inferred from process state alone.
Make the system observable enough to explain.
The monitoring system became testable infrastructure rather than a collection of dashboards. The core principle is that observability itself needs controlled tests, expected behavior, reproducible failure and verification.
Artifacts that carry the technical story.
The case study is structured around real engineering evidence rather than generic product imagery.