A Green Dashboard Can Hide a Dead Data Pipeline
A panel can render old data successfully after the collector behind it has already stopped.
TOPIC
Observability & Monitoring
Lessons, explainers, experiments, and implementation notes.
Separate Raw Sensor Data from Interpreted State
Presence, motion and occupancy are conclusions; retain enough underlying evidence to debug those conclusions.
Calibration Defines Zero; Detection Measures Deviation
The useful baseline is the room's normal operating condition, not an imaginary perfectly quiet environment.
Ingress Telemetry Is a Boundary You Must Test
A topology UI can pass synthetic tests and still misread production traffic at the first adapter.
Prometheus Labels Are Part of the API
Label names and cardinality are query contracts for dashboards, recording rules and alerts.
Monitoring Tax: Why Exporters Need Resource Budgets
A production-engineering deep dive into monitoring tax: why exporters need resource budgets, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Temporary I/O Is a Query-Behavior Signal
A production-engineering deep dive into temporary i/o is a query-behavior signal, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Disk Full and Inodes Full Are Two Different Outages
A production-engineering deep dive into disk full and inodes full are two different outages, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Page Cache Is Not a Memory Leak
A production-engineering deep dive into page cache is not a memory leak, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
One PostgreSQL Deadlock Is Worth Recording
A production-engineering deep dive into one postgresql deadlock is worth recording, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
The VoIP Dashboard Must Follow a Call Across Layers
A production-engineering deep dive into the voip dashboard must follow a call across layers, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Desired State, Observed State and User-Visible State Are Three Different Things
A production-engineering deep dive into desired state, observed state and user-visible state are three different things, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Waiting Sessions Tell Me More Than CPU During Database Contention
A production-engineering deep dive into waiting sessions tell me more than cpu during database contention, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Disk Throughput Is Not Disk Latency
A production-engineering deep dive into disk throughput is not disk latency, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Buffer Hits, Disk Reads and the Shape of PostgreSQL Cache Pressure
A production-engineering deep dive into buffer hits, disk reads and the shape of postgresql cache pressure, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Swap Usage Alone Is a Bad Memory Alert
A production-engineering deep dive into swap usage alone is a bad memory alert, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Thermals on a 2014 Mac mini Are a Production Signal
A production-engineering deep dive into thermals on a 2014 mac mini are a production signal, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
One Machine, Many Failure Domains: Mapping the 2014 Mac mini
A production-engineering deep dive into one machine, many failure domains: mapping the 2014 mac mini, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
The Server Cannot Report Its Own Death
A production-engineering deep dive into the server cannot report its own death, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Monitoring TCP Retransmission on a Server That Also Runs VoIP
A production-engineering deep dive into monitoring tcp retransmission on a server that also runs voip, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
What Production-Grade Means on Decade-Old Hardware
A production-engineering deep dive into what production-grade means on decade-old hardware, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
How cAdvisor Became One of My Largest Workloads
A production-engineering deep dive into how cadvisor became one of my largest workloads, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Grouping and Inhibition Stop One Failure Becoming Twenty Pages
A production-engineering deep dive into grouping and inhibition stop one failure becoming twenty pages, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why 99.9% Uptime Does Not Tell Me When to Wake Someone Up
A production-engineering deep dive into why 99.9% uptime does not tell me when to wake someone up, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Linux PSI Changed How I Think About Saturation
A production-engineering deep dive into linux psi changed how i think about saturation, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Fast Burn and Slow Burn Are Different Incidents
A production-engineering deep dive into fast burn and slow burn are different incidents, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Clock Synchronization Is an Availability Dependency
A production-engineering deep dive into clock synchronization is an availability dependency, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Container Running, Healthy and Useful Are Three Different States
A production-engineering deep dive into container running, healthy and useful are three different states, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Restart Counts Without Context Create Noise
A production-engineering deep dive into restart counts without context create noise, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
OOM Events Are Better Evidence Than “High Memory” Alone
A production-engineering deep dive into oom events are better evidence than “high memory” alone, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Memory Limit Utilization vs Working Set: Which One Should Page Me?
A production-engineering deep dive into memory limit utilization vs working set: which one should page me?, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Container Block I/O and I/O Pressure Tell Different Stories
A production-engineering deep dive into container block i/o and i/o pressure tell different stories, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Docker Storage Cache Needs Its Own Freshness Metric
A production-engineering deep dive into a docker storage cache needs its own freshness metric, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Prometheus Is My Metrics Control Plane, Not Just a Scraper
A production-engineering deep dive into prometheus is my metrics control plane, not just a scraper, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why Every Target Should Not Be Scraped Every 15 Seconds
A production-engineering deep dive into why every target should not be scraped every 15 seconds, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
27.5k Active Series on a Small Server: What That Number Actually Means
A production-engineering deep dive into 27.5k active series on a small server: what that number actually means, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Cardinality Is Usually a Schema Problem
A production-engineering deep dive into cardinality is usually a schema problem, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Metric Relabeling Is Resource Control
A production-engineering deep dive into metric relabeling is resource control, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Recording Rules Are a CPU Trade: Compute Once or Query Repeatedly
A production-engineering deep dive into recording rules are a cpu trade: compute once or query repeatedly, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
30-Day Retention and a 15 GB Cap Were Capacity Decisions
A production-engineering deep dive into 30-day retention and a 15 gb cap were capacity decisions, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
WAL, Head Block and Why Prometheus Memory Does Not Equal Stored Data
A production-engineering deep dive into wal, head block and why prometheus memory does not equal stored data, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Rule Evaluation Failures Mean Monitoring Logic Is Broken
A production-engineering deep dive into rule evaluation failures mean monitoring logic is broken, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
How Much Prometheus Is Too Much for an 8 GB Machine?
A production-engineering deep dive into how much prometheus is too much for an 8 gb machine?, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Dashboards Should Answer Questions, Not Display Metrics
A production-engineering deep dive into dashboards should answer questions, not display metrics, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Building a NOC Dashboard for a Phone Screen
A production-engineering deep dive into building a noc dashboard for a phone screen, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Overview vs Drill-Down: One Dashboard Cannot Do Both Well
A production-engineering deep dive into overview vs drill-down: one dashboard cannot do both well, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why Green Dashboards Can Still Lie
A production-engineering deep dive into why green dashboards can still lie, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Alert Panels and Investigation Panels Serve Different Humans
A production-engineering deep dive into alert panels and investigation panels serve different humans, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Information Density Without Dashboard Wall Art
A production-engineering deep dive into information density without dashboard wall art, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
The Observability Dashboard That Watches the Observability Stack
A production-engineering deep dive into the observability dashboard that watches the observability stack, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Metrics Tell Me That Something Happened; Logs Tell Me What Happened
A production-engineering deep dive into metrics tell me that something happened; logs tell me what happened, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why Loki Labels Need to Stay Boring
A production-engineering deep dive into why loki labels need to stay boring, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
IP Addresses Belong in Log Content More Often Than Labels
A production-engineering deep dive into ip addresses belong in log content more often than labels, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Turning SSH Failures into a Production Security Signal
A production-engineering deep dive into turning ssh failures into a production security signal, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Journald Cursor Persistence Prevents Duplicate or Missing Logs
A production-engineering deep dive into journald cursor persistence prevents duplicate or missing logs, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
The Stale Docker Container Problem in Alloy
A production-engineering deep dive into the stale docker container problem in alloy, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
How Logging Itself Can Become an I/O Workload
A production-engineering deep dive into how logging itself can become an i/o workload, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Seven Days of Logs Was an Engineering Choice, Not a Default
A production-engineering deep dive into seven days of logs was an engineering choice, not a default, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Alert Fatigue Is an Architecture Bug
A production-engineering deep dive into alert fatigue is an architecture bug, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Threshold Alerts vs Symptom Alerts
A production-engineering deep dive into threshold alerts vs symptom alerts, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
What Prometheus `for:` Really Buys You—and What It Cannot
A production-engineering deep dive into what prometheus `for:` really buys you—and what it cannot, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Warning and Critical Are Different Response Contracts
A production-engineering deep dive into warning and critical are different response contracts, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Monitoring Is a Failure-Modeling Problem, Not a Dashboard Problem
A production-engineering deep dive into monitoring is a failure-modeling problem, not a dashboard problem, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
The Observability Tax on a 7.1 GiB Linux Server
A production-engineering deep dive into the observability tax on a 7.1 gib linux server, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why Every Monitoring Component Had to Justify Its RAM
A production-engineering deep dive into why every monitoring component had to justify its ram, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why My Monitoring System Needed Monitoring Too
A production-engineering deep dive into why my monitoring system needed monitoring too, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why “Up” Is One of the Weakest Signals in Production
A production-engineering deep dive into why “up” is one of the weakest signals in production, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Redis Eviction Means Policy Is Already Affecting Data
A production-engineering deep dive into redis eviction means policy is already affecting data, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Redis Fragmentation Can Look Like a Memory Leak
A production-engineering deep dive into redis fragmentation can look like a memory leak, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
MySQL Slow Queries Belong in Infrastructure Monitoring
A production-engineering deep dive into mysql slow queries belong in infrastructure monitoring, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Four Database Engines on One Small Server: Observability Without Exporter Sprawl
A production-engineering deep dive into four database engines on one small server: observability without exporter sprawl, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why a 302 Redirect Can Mean the Service Is Healthy
A production-engineering deep dive into why a 302 redirect can mean the service is healthy, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Internal Probe and Public Probe Answer Different Questions
A production-engineering deep dive into internal probe and public probe answer different questions, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
DNS Is a Production Dependency, Not Plumbing
A production-engineering deep dive into dns is a production dependency, not plumbing, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
TLS Certificate Expiration Is an Availability Incident Waiting to Happen
A production-engineering deep dive into tls certificate expiration is an availability incident waiting to happen, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Building a Dead-Man Monitor Outside the Machine It Watches
A production-engineering deep dive into building a dead-man monitor outside the machine it watches, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
GitHub Actions as Five-Minute External Infrastructure
A production-engineering deep dive into github actions as five-minute external infrastructure, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
I Deliberately Broke the Probe to Test the Monitoring System
A production-engineering deep dive into i deliberately broke the probe to test the monitoring system, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Dead-Man Monitoring Should Not Need a Credential from the Dead System
A production-engineering deep dive into dead-man monitoring should not need a credential from the dead system, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Monitoring SIP Is Not Monitoring RTP
A production-engineering deep dive into monitoring sip is not monitoring rtp, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Healthy FreeSWITCH Process Can Still Mean Broken Calls
A production-engineering deep dive into a healthy freeswitch process can still mean broken calls, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Heartbeat Freshness Is More Useful Than “Container Running” for Voice Workers
A production-engineering deep dive into heartbeat freshness is more useful than “container running” for voice workers, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
No Healthy Worker Is a User-Facing Signal
A production-engineering deep dive into no healthy worker is a user-facing signal, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
OpenSIPS 5xx Spikes Are More Useful Than Total SIP Traffic
A production-engineering deep dive into opensips 5xx spikes are more useful than total sip traffic, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
RTPengine Timeout Reasons Tell Different Media Stories
A production-engineering deep dive into rtpengine timeout reasons tell different media stories, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Sealed Secret Server Can Be Secure and Still Be Down
A production-engineering deep dive into a sealed secret server can be secure and still be down, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
How I Exposed OpenBao Metrics Without Exposing OpenBao
A production-engineering deep dive into how i exposed openbao metrics without exposing openbao, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Config Drift Is a Monitoring Signal
A production-engineering deep dive into config drift is a monitoring signal, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Monitoring Secrets Without Monitoring Secret Values
A production-engineering deep dive into monitoring secrets without monitoring secret values, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
SSH, sudo, Authelia and Docker Errors as One Security Telemetry Plane
A production-engineering deep dive into ssh, sudo, authelia and docker errors as one security telemetry plane, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Backup Success Is Not Recoverability
A production-engineering deep dive into backup success is not recoverability, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Fresh Backup Can Still Be Corrupt
A production-engineering deep dive into a fresh backup can still be corrupt, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Restore Verification Is the Most Important Backup Metric I Have
A production-engineering deep dive into restore verification is the most important backup metric i have, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Monitoring the System That Is Supposed to Save the System
A production-engineering deep dive into monitoring the system that is supposed to save the system, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
RPO, RTO and Evidence: When a Home Server Becomes a Production System
A production-engineering deep dive into rpo, rto and evidence: when a home server becomes a production system, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
How I Took cAdvisor from ~428 MiB to ~20–28 MiB
A production-engineering deep dive into how i took cadvisor from ~428 mib to ~20–28 mib, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Multi-Window Burn-Rate Alerts on a Tiny Self-Hosted Stack
A production-engineering deep dive into multi-window burn-rate alerts on a tiny self-hosted stack, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Resolved Notifications Are Part of the Incident Lifecycle
A production-engineering deep dive into resolved notifications are part of the incident lifecycle, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Why I Monitor MemAvailable Instead of “Free RAM”
A production-engineering deep dive into why i monitor memavailable instead of “free ram”, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Docker Filesystem Scanning Was More Expensive Than the Metric Was Worth
A production-engineering deep dive into docker filesystem scanning was more expensive than the metric was worth, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Load Average Without CPU Count Is Almost Meaningless
A production-engineering deep dive into load average without cpu count is almost meaningless, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Metrics, Logs, Probes and State Checks Answer Different Questions
A production-engineering deep dive into metrics, logs, probes and state checks answer different questions, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Every Alert Should Suggest the Next Investigation
A production-engineering deep dive into every alert should suggest the next investigation, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
A Successful TCP Connection Does Not Mean the Database Is Healthy
A production-engineering deep dive into a successful tcp connection does not mean the database is healthy, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
PostgreSQL Connection Utilization Needs a Denominator
A production-engineering deep dive into postgresql connection utilization needs a denominator, grounded in the 2014 Mac mini hserver observability stack and its accepted runtime evidence.
Freshness Guards Prevent Stale Metrics from Firing Misleading Alerts
A target that disappeared can leave its last sample available long enough for threshold expressions to evaluate against stale data.
Memory Usage Is Not the Same as Memory Pressure
The host looked busy enough that a simple percentage could easily become the whole diagnosis, but Linux memory reclaim makes that misleading.
Swap Usage Needs Duration and Context
A few megabytes of swap on an old Linux host did not automatically mean an incident, especially after long uptime.
Normalize Load Average by CPU Count
A load average of four means something very different on a two-core system than on an eight-core system.
CPU Utilization Without CPU Modes Hides the Bottleneck
The host can report high CPU usage even when the useful question is whether time is going to user work, system work, steal, or I/O wait.
PSI Made Short Resource Spikes Easier to Interpret
Some periods looked acceptable in average CPU and RAM graphs while interactive services still felt slow.
Clock Synchronization Is a Monitoring Dependency
Logs, TLS checks, backup ages and distributed event ordering all become harder to trust when the host clock drifts.
Temperature Monitoring Belongs Beside Load
A small Mac mini running many containers can hit thermal constraints before ordinary CPU graphs explain why performance changed.
Context Switches and Interrupts Explain Some Invisible CPU Cost
CPU percentage alone did not reveal when the kernel was spending more work scheduling tasks or servicing device activity.
Critical systemd Units Need Their Own Health Signal
Container monitoring does not cover host services such as Docker, networking, tunnels, backup timers or other systemd-managed dependencies.
SMART Health Is Different from Filesystem Free Space
A disk can have plenty of free capacity while the underlying SSD is reporting temperature or device-health problems.
Running Containers Are Not Healthy Containers
The Docker daemon can report a container as running even when the application inside it has stopped serving useful traffic.
Container Restart Counts Are Incident Breadcrumbs
A service can look healthy now and still have restarted repeatedly overnight, erasing the evidence from a simple current-state view.
OOM Events Need Container and Host Context
An application disappearing under load can be either a container memory-limit event or a host-wide memory emergency.
CPU by Container Is Useful Only Beside Limits
A container using one full CPU may be expected on an unrestricted worker and catastrophic for a service capped at a fraction of a core.
Working Set Is More Useful Than a Single Memory Number
Container memory graphs become noisy when cache and reclaimable pages are treated exactly like unreclaimable application working memory.
Container Network Errors Matter More Than Throughput Alone
RX and TX graphs looked busy enough, but throughput by itself could not tell whether traffic was healthy.
Block I/O per Container Exposes Noisy Neighbors
Host disk latency can rise because one container is performing heavy reads or writes while every other service only sees the consequence.
Container I/O Pressure Shows Waiting, Not Just Bytes
A workload can perform modest disk throughput and still suffer because requests are waiting behind slow storage or competing I/O.
Docker Security Posture Belongs in Monitoring
Privileged containers, Docker socket mounts, root users and published ports are configuration facts that can silently drift after deployment.
Healthcheck Duration Can Warn Before Healthcheck Failure
A probe may keep returning success while taking progressively longer, showing a dependency slowdown before Docker marks the container unhealthy.
Disk Usage Alerts Need Warning and Critical Bands
A single disk threshold gives operators no distinction between early cleanup work and a filesystem that is close to stopping writes.
Inode Exhaustion Can Break a Filesystem with Free Gigabytes
Free-space graphs can remain green while a workload creates huge numbers of tiny files and consumes the available inode table.
Disk Latency Explains More Than Disk Throughput
The host can show modest megabytes per second while applications still wait because each storage request takes too long.
IOPS and Throughput Answer Different Storage Questions
A database doing many tiny synchronous operations and a backup streaming large files can show similar disk utilization with very different access patterns.
Disk Busy Time Needs Latency Beside It
A device at high utilization can be handling work efficiently, while a lower-utilization device can still be returning slow requests.
Writable Layer Size Should Not Be Polled Expensively Every Scrape
Docker storage size is useful, but asking the daemon for deep filesystem inventory on every fast scrape consumed too much monitoring overhead.
Volume Size Monitoring Needs Ownership Context
A large Docker volume is not actionable if the dashboard cannot tell which service owns it or whether that growth is expected.
Image Storage Growth Is Different from Runtime Data Growth
Docker images can quietly accumulate through repeated deployments even when application volumes and databases remain stable.
The cAdvisor Disk Scanner Was More Expensive Than the Data Was Worth
The production monitoring stack measured roughly 428 MiB of cAdvisor memory with filesystem disk collection enabled on a small host.
A Storage Cache Needs Its Own Freshness Metric
Caching Docker storage inventory reduced overhead, but a silent refresh failure could otherwise leave old values looking current indefinitely.
Public and Internal Probes Answer Different Availability Questions
A service can be healthy on the LAN and unreachable through Cloudflare, TLS, DNS or authentication at the public edge.
Certificate Expiry Is an Availability Signal
TLS can work perfectly today and still have a known future outage date embedded in the certificate.
DNS Lookup Time Belongs in an Edge Dashboard
A slow public request is not always slow application code; DNS resolution can consume a meaningful part of the user-visible path.
HTTPS Latency Needs More Than One Number
A single total probe duration hides whether slowness came from name resolution, TCP connection, TLS negotiation or server response.
Packet Drops Are More Actionable Than Raw Bandwidth
High interface traffic can be completely healthy, while a small but sustained drop rate can damage voice, APIs and tunnel reliability.
TCP Retransmit Ratio Is Better Than Counting Retransmits Alone
A few retransmissions during heavy traffic may be normal, while the same count during low traffic can represent a serious quality problem.
Wi-Fi Signal Needed a Fallback Path
The host collector originally relied on `iw`, but driver and interface behavior did not always expose the active connection signal consistently.
Cloudflare Tunnel Disruptions Need Log-Derived Monitoring
An external endpoint can flap because the tunnel reconnects even when the local service and LAN probe remain healthy.
Socket Counts Are Capacity Clues, Not Health by Themselves
A rising established-connection count can represent normal load, a leak, slow clients or a downstream dependency holding sockets open.
Monitoring Redirects Can Create False Green Probes
A health probe that automatically follows redirects may end on an authentication page and report successful HTTP even though the original service route is wrong.
A TCP Database Probe Is Not a Database Health Check
A listening PostgreSQL or Redis port proves that something accepted a TCP connection, not that queries, authentication or storage are working correctly.
PostgreSQL Connection Utilization Needs a Denominator
A count of sixty active PostgreSQL connections is meaningless until it is compared with the configured maximum for that server.
One Deadlock Is Worth Recording
A PostgreSQL deadlock can resolve automatically by aborting one transaction, leaving the service apparently healthy after the incident.
Commit and Rollback Rates Reveal Database Behavior
Database traffic volume looked normal even when applications were rolling back more transactions than usual.
Database Size Growth Is a Capacity Trend, Not an Emergency Metric
A database normally grows, so alerting on size alone would create noise while ignoring the important question of growth rate and disk headroom.
Redis Evictions Mean Memory Policy Is Affecting Data
Redis can remain responsive while silently evicting keys because the configured memory limit has been reached.
Redis Memory Needs Role Context
A Redis instance serving disposable cache and one serving authentication sessions can show the same memory growth with very different operational risk.
Mongo Connections and Query Rate Tell Different Stories
A MongoDB service can accumulate client connections without a matching rise in useful query work, which can point to pooling or application lifecycle problems.
Database Container Metrics Need Engine Metrics Beside Them
High database container CPU or memory can be an important symptom, but it cannot identify whether the engine is busy with useful work, blocked transactions or internal maintenance.
Deep Database Collection Did Not Need Copied Passwords
Copying every application database password into the observability stack would have expanded the secret blast radius just to collect metrics.
Journal Cursor Persistence Prevented Duplicate Log Ingestion
Restarting a log collector can replay old journal entries or skip new ones if it does not persist its position correctly.
Monitor the Log Pipeline for Dropped Entries
A logging stack can look healthy at the process level while silently discarding entries because of relabeling, backpressure or write failures.
SSH Failure Alerts Need Warning and Critical Bursts
A few failed SSH logins are normal on an administered server, while a rapid burst can indicate brute-force activity or a broken automation credential.
Sudo Authentication Failures Belong in the Security Timeline
Repeated sudo failures may be operator error, expired credentials or suspicious privilege-escalation attempts, and they often happen outside application logs.
Kernel OOM Logs Are the Ground Truth for Host Memory Kills
A process disappearing can look like an application crash unless the kernel journal is checked for OOM-killer activity.
Docker Daemon Error Bursts Need a Separate Alert
Individual container logs can all look normal while the Docker daemon is failing image, network, storage or runtime operations underneath them.
Authelia Authentication Failures Need Context, Not Panic
A two-factor login system naturally records failed credentials, expired sessions and rejected access, so any single failure is not automatically an attack.
Critical Container Log Spikes Are a Triage Signal
A container can remain healthy by its HTTP probe while its logs suddenly fill with exceptions, retries or failed dependency calls.
Cloudflare and Authelia Logs Explain Edge Failures Differently
A public request can fail before reaching Authelia, inside the authentication flow, or after authentication while the upstream application is unavailable.
Do Not Turn Source IP into a High-Cardinality Metric Label
Security logs contain useful source addresses, but promoting every IP to a Prometheus or Loki index label would create unbounded cardinality on an Internet-facing service.
Drachtio Connectivity Is a Control-Plane Health Signal
The SIP proxy can be running as a process while its control connection to drachtio is down, leaving signaling logic unable to operate correctly.
FreeSWITCH Health Is a Fleet Count, Not One Boolean
A multi-worker voice stack can keep serving calls after one FreeSWITCH node fails, so one global up/down flag hides degraded capacity.
Worker Heartbeat Age Catches Silent FreeSWITCH Failure
A FreeSWITCH process may remain alive while its worker integration stops reporting useful state to the signaling layer.
No Healthy SIP Worker Is a User-Facing Failure Counter
The most important load-balancer failure is not that a backend probe failed; it is that an incoming SIP request could not be assigned to any healthy worker.
SIP Proxy Failure Rate Needs a Window
A single SIP proxy error may be harmless noise, but repeated failures over a short interval can indicate backend, routing or dependency trouble.
OpenSIPS Metrics Availability Is Separate from OpenSIPS Availability
OpenSIPS can continue processing calls while the MI metrics collector fails, leaving the service healthy but observability blind.
OpenSIPS 5xx Spikes Show Signaling Quality
Registration counts and dialog counts can look stable while transaction failures increase for a subset of calls.
RTPEngine Failure Reasons Are Better Than One Failure Total
Media sessions can close for rejection, timeout, silent timeout, final timeout or offer timeout, and those reasons point to different failure paths.
FreeSWITCH Session Counts Need Per-Node Breakdown
A healthy total session count can hide a load-balancing problem if one FreeSWITCH node carries nearly all calls while another remains idle.
Registered Endpoints Are Capacity and Behavior Signals
A sudden drop in registered SIP endpoints can indicate network reachability, credential, expiry or registrar problems even when call processing components are up.
Backup Age Is a Freshness Signal, Not a Recovery Guarantee
A recent backup timestamp can look reassuring even when the archive is incomplete, corrupt or impossible to restore.
Checksum Monitoring Detects Silent Backup Corruption
A backup directory can exist with the expected filenames while one archive is truncated or modified after creation.
Restore Verification Needs Its Own Age
A restore drill that passed months ago does not prove that today's schema, credentials and backup format can still be recovered.
Backup Job Exit Status Belongs in Prometheus
A systemd timer can fire on schedule while the backup service itself fails, times out or exits before producing a valid set.
Backup Size History Can Reveal Missing Data
A backup that suddenly becomes much smaller may have completed successfully while silently omitting a database, artifact directory or other expected state.
Backup Journals Make Failed Automation Debuggable
A red backup alert identifies the outcome but usually does not explain which command, mount or permission caused the failure.
Disk Headroom Is Part of Backup Reliability
A backup process can be perfectly configured and still fail when the destination filesystem no longer has enough capacity for the next archive.
Volume Reference Counts Help Detect Orphaned Docker State
Old Docker volumes can consume storage after services are removed, and their names alone do not always reveal whether anything still depends on them.
A Restore Alert Should Distinguish Failed from Stale
No recent restore verification and a recent restore verification that actively failed are both bad, but they communicate different operational urgency.
Backup Monitoring Needs UNKNOWN as a Real State
Least-privilege monitoring sometimes cannot read protected backup evidence, and treating that access failure as healthy would be dangerous.
Firing, Pending and Rule Count Tell Different Alerting Stories
A dashboard that only shows firing alerts hides whether conditions are approaching thresholds or whether the expected rule set is even loaded.
The `for` Clause Is an Anti-Flap Tool, Not Decoration
CPU, packet loss and endpoint probes can cross thresholds for a few seconds during harmless transitions or deployment activity.
Monitor Notification Success, Failure and Resolution Separately
An alert can fire correctly and still never reach the operator if Alertmanager or the external delivery path fails.
Self-Hosted ntfy Removed an External Delivery Dependency
The alert sink originally targeted a public ntfy service, which made production notification depend on infrastructure outside the hserver control boundary.
Notification Authentication Belongs Outside Git
Moving alert delivery to authenticated self-hosted ntfy introduced a publisher token that the alert-sink needs at runtime.
Alert Delivery Failure Needs an Alert of Its Own
A monitoring system that detects failures but cannot notify anyone is partially failed even if every Prometheus target remains green.
Severity Mapping Should Reflect Operator Action
If every alert is critical, operators lose the distinction between conditions that require immediate intervention and those that need scheduled review.
Notification Endpoint Health Needs Direct Monitoring
Delivery counters only change when an alert is sent, so a quiet system could leave a dead notification service unnoticed for hours.
Pending Alerts Are Useful During Change Windows
A deployment may push a metric over threshold without immediately firing because the configured `for` period has not elapsed.
Zero Firing Alerts Does Not Mean Monitoring Is Healthy
A broken rule evaluator or missing target can produce a beautifully quiet alert dashboard while the system is blind.
Prometheus Series Count Is an Observability Capacity Metric
The hserver stack carried roughly twenty-seven thousand active Prometheus series, enough that label growth and exporter changes could materially change memory and storage cost.
Scrape Interval Should Match Signal Speed and Collection Cost
CPU can change meaningfully in seconds while Docker image storage or Grafana process memory does not need the same fifteen-second collection cadence.
Metric Relabeling Can Be a Memory Optimization
Grafana, Loki and Alloy expose many internal metrics that are useful for development but unnecessary on a small production Prometheus.
The Monitoring Stack Needs a Memory Budget Too
Observability was becoming one of the larger workloads on a small production server, which is dangerous when monitoring competes with the services it protects.
Retention Needs Both Time and Size Limits
Keeping thirty days of Prometheus data is useful until series growth causes the TSDB to consume more disk than the host can safely spare.
Dead Synthetic Targets Should Be Removed, Not Hidden
Placeholder or obsolete probe targets can stay in configuration after architecture changes and permanently pollute availability dashboards.
Dashboards Should Follow Troubleshooting Workflows
A wall of attractive graphs is slow during an incident if related signals are scattered by exporter rather than by the question an operator is trying to answer.
A Mobile NOC Dashboard Should Show Decisions, Not Every Metric
The phone-sized operations view cannot carry hundreds of panels without turning urgent information into scrolling noise.
Observability Needs a Source of Truth and an Acceptance Artifact
A monitoring stack can drift through dashboard edits, local files and runtime tuning until nobody knows whether Git can reproduce what is currently trusted in production.