Skip to content

Deployment safety checks

Deploys go wrong. The question is whether the pipeline notices before your users do. Safety checks are the automated gates that stop a broken artifact between merge and full rollout — cheap to add, harder to design well.

Every production deploy passes through five gates, in order. A failure at any gate halts the rollout and — where possible — reverts automatically.

  • Green pipeline. Build, unit tests, integration tests, lint, and code review all pass on the merged commit. Deploys never start from a red build.
  • Migration pre-flight. Schema changes run in a dry-run mode against a staging clone. Additive-only by default (add column, backfill, dual-write); destructive migrations ship in a separate release from the code that stops using the old shape.
  • Progressive rollout. Ship to one instance, wait, then 10%, then 50%, then 100%. Each step waits long enough to observe error rate and latency — a minute is not long enough; five to ten usually is.
  • Automated health check. The pipeline hits a real endpoint after each step and compares error rate, p95 latency, and a business-signal metric (checkout success, login success — pick one) against the previous version. Regression trips a rollback.
  • Feature flag for anything risky. Ship the code dark. Turn it on behind a flag after the deploy is stable. Rollback of a flag is a config change, not a redeploy.

Deploys are the highest-frequency change in production, so they’re where you get the most benefit from making failure cheap. Progressive rollout plus a health check turns “we shipped a bug to everyone” into “we shipped a bug to 10% for six minutes.” Feature flags cut the same problem again for behaviour changes — the artifact is fine, the flag is off.

The tradeoff is deploy latency: a five-gate pipeline takes 15–30 minutes instead of five. That cost is paid on every deploy; the saving lands on the bad ones. If you deploy weekly, the ratio hurts. If you deploy daily, it pays for itself the first time it catches something.

  • Health checks that only check “is the process up.” A returning-200 liveness probe is not a safety check. Compare a real business metric against the prior version.
  • Rollout steps too short to observe. A 30-second wait between 10% and 100% means the health check runs on cached success. Set the wait to your slowest useful signal.
  • Destructive migrations shipped with the code that needs them. If the migration fails, the deploy is stuck between two schemas. Split into two releases: expand, then contract.
  • Feature flags that never get cleaned up. A flag left on for a year is dead code plus a config surface. Set an expiry when you create it.
  • Manual gates in the pipeline. “Wait for approval” boxes get rubber-stamped. If a check matters, automate it; if it doesn’t, delete it.
  1. Write down what your pipeline does today between merge and 100% rollout. If you can’t, that’s the first bug.
  2. Add a progressive rollout stage if one doesn’t exist. Start with two steps (10%, 100%) before tuning further.
  3. Pick one business-signal metric per service. Wire it into the health check so a regression halts the rollout.
  4. Audit open migrations. Convert anything destructive to expand/contract.
  5. Introduce a feature flag system before you need one — retrofitting under incident pressure is where flag sprawl comes from.