Runbook: Deploy & rollback

Stable

A runbook, not an essay. Follow it top to bottom. If a step fails, stop and go to Rollback.

Preconditions

The change is merged to main and CI is green.
You can reach the deploy dashboard and the logs.
You know the current running version (write it down — you’ll need it to roll back).

Verify the commit SHA on main is the one you intend to ship.

Start the deploy from the pipeline. Do not deploy from a local machine.

Watch readiness checks come green. Healthy means: pods/instances ready, error rate flat, latency within normal band.

Hit the critical path (for CRVS: search a known record; for telerad: load a study). Confirm requestId correlation appears in logs.

Post the version and SHA in the ops channel. You’re done.

If error rate spikes, the smoke test fails, or you’re unsure — roll back first, diagnose after. Rolling back is always safe.

Trigger a deploy pinned to the last known-good SHA you recorded in preconditions.

Confirm error rate and latency return to normal and the smoke test passes.

Hold further deploys until the cause is understood.

Open an incident note: what shipped, what broke, how you knew, what fixed it.