Deployment safety checks
Deploys go wrong. The question is whether the pipeline notices before your users do. Safety checks are the automated gates that stop a broken artifact between merge and full rollout — cheap to add, harder to design well.
Recommended default
Section titled “Recommended default”Every production deploy passes through five gates, in order. A failure at any gate halts the rollout and — where possible — reverts automatically.
- Green pipeline. Build, unit tests, integration tests, lint, and code review all pass on the merged commit. Deploys never start from a red build.
- Migration pre-flight. Schema changes run in a dry-run mode against a staging clone. Additive-only by default (add column, backfill, dual-write); destructive migrations ship in a separate release from the code that stops using the old shape.
- Progressive rollout. Ship to one instance, wait, then 10%, then 50%, then 100%. Each step waits long enough to observe error rate and latency — a minute is not long enough; five to ten usually is.
- Automated health check. The pipeline hits a real endpoint after each step and compares error rate, p95 latency, and a business-signal metric (checkout success, login success — pick one) against the previous version. Regression trips a rollback.
- Feature flag for anything risky. Ship the code dark. Turn it on behind a flag after the deploy is stable. Rollback of a flag is a config change, not a redeploy.
Why this is the default
Section titled “Why this is the default”Deploys are the highest-frequency change in production, so they’re where you get the most benefit from making failure cheap. Progressive rollout plus a health check turns “we shipped a bug to everyone” into “we shipped a bug to 10% for six minutes.” Feature flags cut the same problem again for behaviour changes — the artifact is fine, the flag is off.
The tradeoff is deploy latency: a five-gate pipeline takes 15–30 minutes instead of five. That cost is paid on every deploy; the saving lands on the bad ones. If you deploy weekly, the ratio hurts. If you deploy daily, it pays for itself the first time it catches something.
Common mistakes or tradeoffs
Section titled “Common mistakes or tradeoffs”- Health checks that only check “is the process up.” A returning-200 liveness probe is not a safety check. Compare a real business metric against the prior version.
- Rollout steps too short to observe. A 30-second wait between 10% and 100% means the health check runs on cached success. Set the wait to your slowest useful signal.
- Destructive migrations shipped with the code that needs them. If the migration fails, the deploy is stuck between two schemas. Split into two releases: expand, then contract.
- Feature flags that never get cleaned up. A flag left on for a year is dead code plus a config surface. Set an expiry when you create it.
- Manual gates in the pipeline. “Wait for approval” boxes get rubber-stamped. If a check matters, automate it; if it doesn’t, delete it.
Step by step
Section titled “Step by step”- Write down what your pipeline does today between merge and 100% rollout. If you can’t, that’s the first bug.
- Add a progressive rollout stage if one doesn’t exist. Start with two steps (10%, 100%) before tuning further.
- Pick one business-signal metric per service. Wire it into the health check so a regression halts the rollout.
- Audit open migrations. Convert anything destructive to expand/contract.
- Introduce a feature flag system before you need one — retrofitting under incident pressure is where flag sprawl comes from.