Skip to content

Reliability and rollback guidance

Reliability is not the absence of failure — it is a short mean time to recovery. The single biggest lever is a rollback path you can trigger in minutes, not one you improvise at 3am.

Revert first, diagnose second. When production breaks after a change, roll back to the last known-good version before you try to fix forward. Every service, every deploy, must satisfy four conditions:

  • One-command revert. kubectl rollout undo, ASG previous launch template, or a pipeline “redeploy previous artifact” button. If revert is a runbook longer than three steps, it will not happen under pressure.
  • Immutable artifacts. The previous version is a specific image tag or build number, not “whatever latest pointed to yesterday.” Promotion moves the same artifact across environments; a revert redeploys the previous one.
  • Expand/contract migrations. Schema changes are additive; the old code path keeps working against the new schema. The destructive step ships in a later release, after revert is no longer needed.
  • Feature-flagged behaviour changes. Turn off first, redeploy second. A flag flip is a config change and reverts in seconds; a redeploy takes minutes.

The engineer who broke production wants to fix forward — it feels responsible, and the fix is in their head. It is almost always slower. Fixing forward means diagnosing under pressure, writing code under pressure, and shipping code under pressure, each with its own failure mode. Reverting swaps three risky steps for one safe one.

The tradeoff: reverting can discard a legitimate change that was 95% right, and someone has to re-land it later. That cost is worth paying — customers get a working service now, and the fix ships properly tomorrow with a proper review.

Reliability follows from small changes with a fast revert. Big changes with heroic recovery look impressive; small changes with boring rollbacks keep the SLO green.

  • Fixing forward “because it’ll be quicker.” Occasionally true, mostly not. Set a timer: if the fix isn’t shipping within 15 minutes, revert.
  • Rollbacks that depend on new state. A revert that cannot handle rows the new version wrote is not a revert. Expand/contract keeps this safe.
  • Feature flags without a kill switch. A flag you cannot flip in production without a deploy is not a flag; it is a compile-time constant with extra steps.
  • “Redeploy the previous commit” as the plan. The previous commit may not build the same artifact today — dependencies drift, base images update. Pin to an artifact, not a commit.
  • Untested rollbacks. A revert path first exercised during an incident is a coin toss. Practise it in staging on a real schedule.
  1. For each service, write down the exact command or button that reverts it. If nobody knows, that is the first bug to fix.
  2. Confirm the previous N artifacts are retained and deployable. Set a retention policy — a week is usually enough.
  3. Audit pending migrations. Split anything destructive into expand/contract phases.
  4. Add a “revert drill” to the on-call rotation — one practised revert per quarter, in staging, timed.
  5. When the next incident hits, revert before you diagnose. Write down how long it took. That number is your reliability posture.