A Container Entrypoint Can Break a Perfectly Valid Validation Command
CI failed because the image entrypoint changed how arguments were interpreted, not because the OpenBao configuration was invalid.
TOPIC
Production Engineering
Lessons, explainers, experiments, and implementation notes.
Compose Rendering Is a Cheap Test That Catches Expensive Deployment Mistakes
A configuration should render successfully from a clean checkout before it is trusted on a production host.
Pinned Image Digests Turn a Container Tag into a Reproducible Dependency
A mutable tag tells you what to ask for; a digest tells you what bytes you actually accepted.
A One-Line COPY Permission Bug Can Make an Image Non-Reproducible
A configuration file copied with the wrong mode can behave differently depending on the build context and base image defaults.
A Clean Checkout Is the Real Test of Infrastructure Reproducibility
Live files that are absent from Git are technical debt even when the service is currently healthy.
Running Is Not the Same as Healthy
Container process state only proves that PID 1 exists; readiness has to test the behavior the dependency actually needs.
Mutable Tags Make Rollback Ambiguous
You cannot reliably roll back to yesterday's image if the tag you used yesterday points somewhere else today.
Entrypoint Assumptions Are Hidden Dependencies
A container command is only reproducible when you know whether the image entrypoint wraps, replaces or transforms it.
Build, Release and Run Fail Differently
Separating image construction, configuration injection and runtime startup makes failures easier to localize.
Git safe.directory Can Turn Provenance into UNKNOWN
A production checkout owned by another identity can be readable on disk while Git refuses to trust it.
UNKNOWN Is Better Than a False FAIL When Metadata Is Restricted
A posture check should distinguish evidence it cannot read from evidence that proves the system is wrong.
A Local Remote-Tracking Ref Can Be Stale While Git Says You Match Upstream
Comparing HEAD with origin/main proves consistency with the last fetched view, not with the current remote repository.
A Clean Git Tree Does Not Prove You Deployed the Latest Code
Cleanliness answers whether local tracked files changed; freshness answers whether the revision is the one you intended to run.
Deployment Records Without Commit SHAs Are Operational Debt
When a deployment fails later, the first forensic question is which reviewed source revision actually produced it.
Git, Runtime State and Secrets Need Different Jobs
Operational confusion falls when desired configuration, mutable data and confidential values stop competing to be one source of truth.
A Source-Ownership Manifest Solved the 'Who Owns This Container?' Problem
Seeing a service in Portainer does not tell you which repository, runbook or backup path can recreate it.
External Runtimes Still Need an Owner Even If They Live in Another Repository
Repository boundaries should not become operational blind spots on a shared production host.
Reconciling Concurrent Git Changes Without Force Push
Production work continued in multiple streams, so safe synchronization had to preserve both histories rather than overwrite whichever side moved first.
A Typo in primary_key=True Broke the Model Before Any Business Logic Ran
Schema declarations are executable code; a one-word typo can prevent the application model from initializing correctly.
Migration Tests Catch a Different Class of Failure Than Unit Tests
An application can pass isolated logic tests while its database schema still fails to create, upgrade or enforce the intended constraint.
Publisher Retries Need Idempotency or They Create Duplicate Posts
When a network response is lost, retrying a create request can duplicate the side effect even if the first request succeeded remotely.
Find Existing Before Create Is a Practical Reconciliation Pattern
When the remote API lacks your preferred idempotency primitive, deterministic discovery can recover an uncertain previous attempt.
Delivery State Needs Its Own Table
Content intent and remote-delivery progress are different state machines and should not be collapsed into one post status.
Background Dispatch Needs Reconciliation After Restart
A scheduler can enqueue work before crashing; recovery logic must discover incomplete deliveries after the process returns.
Remote Terminal States Should Stop Re-Publishing Logic
If the provider already reports scheduled, publishing or published, the reconciler should observe rather than repeat the side effect.
Queue Retry Policy Is Part of Data Integrity
Retry counts and backoff are not just performance settings when the job performs external side effects.
Content Is Not the Same Thing as Delivery State
The authoritative article text can remain stable while each destination has its own mutable publishing lifecycle.
Audit Tables Must Be Boring and Correct
An audit trail is only useful if its schema is simpler and more dependable than the systems it records.
Canary Checks Should Protect Unrelated Critical Services Too
A successful target deployment is not a full success if the change quietly damages another workload on the same host.
HTTP Status Semantics Are Part of Production Acceptance
A service can be reachable and still be routed through the wrong authentication or application layer.
Production Acceptance Requires Live Evidence, Not Code Review Alone
A reviewed repository can still be disconnected from the state actually running on the host.
A Clean Checkout Must Contain Every Local Build and Config Reference
Infrastructure source is incomplete if Compose points at files that only exist on the current server.
Back Up and Define Rollback Before You Touch Production
The safest time to decide how to recover is before the change has modified the evidence you depend on.
A Post-Deploy Inventory Snapshot Is Forensic Evidence
Recording containers, ports and revisions after deployment gives future incident response a known-good comparison point.
The Best Postmortem Output Is a New Invariant
Fixing one incident is useful; changing the system so the same class of failure becomes detectable or impossible is more valuable.