OTA Field Notes · advanced

Desired and Observed State Should Never Share a Field

Operator intent and device-reported reality can disagree legitimately during rollout, reboot, outage or rollback.

Current. Published as a first-person engineering field note from verified hands-on domains; no client-confidential details.

Desired and Observed State Should Never Share a Field

Operator intent and device-reported reality can disagree legitimately during rollout, reboot, outage or rollback.

I keep this as a field note because the failure mode is easy to misclassify: assignment overwrites the same field later used for device-reported running state. The useful move is to identify the boundary first, then change only the layer that owns it.

What I model

The system is easier to debug when intent, observation and transport are not collapsed into one state. For this case, my rule is simple: Model intent and observation separately, then reason about convergence.

Implementation pattern

Keep desired_release_id and running_release_id independent and derive status from their relationship.

I prefer a small explicit contract over a clever implicit one. That gives logs, tests and dashboards something concrete to verify and keeps unrelated layers from compensating for each other.

What I verify

  • assignment does not alter observed state
  • heartbeat cannot erase intent
  • UI shows pending convergence

Failure handling

When one of those checks fails, I preserve the failing evidence before restarting or changing configuration. The first broken contract determines the next investigation. That keeps troubleshooting causal instead of turning it into a sequence of guesses.

What I keep

Model intent and observation separately, then reason about convergence. The specific tools can change, but that ownership boundary remains useful across firmware, networks and infrastructure.

Share this article
Facebook WhatsApp X LinkedIn Medium
Quick navigationEsc