Build the system that takes CI's validated artifact the last mile into production: promote it through environments, shift real user traffic onto it a little at a time, watch health signals for evidence it is safe, and reverse everything in seconds the moment it is not.
The boundary between the two systems is a single object: the artifact - a container image or binary that CI built, tested, checksummed, and versioned. CI's final statement is "these exact bytes passed everything." This system's first rule is to never break that statement: we deploy those bytes, unmodified, to every environment. That rule has a name - build once, deploy many - and violating it is a classic production bug: if each environment rebuilds from source, then staging tested one build and production received a different one, and the promotion pipeline was theater. Anything that must differ between environments (database URLs, API keys, tuning) is injected as configuration at deploy time, never baked into the artifact. It also sets this system's character. CI answers "is this code good?" with tests in a lab. CD answers a question no lab can: "does this version survive real production traffic?" The only way to answer it is to run the experiment on production - and so this entire design is about running that experiment with the smallest possible blast radius and an instant undo.
An SLI (service level indicator) is a measured signal about how a service is behaving: the fraction of requests that fail, the p99 latency, or something business-shaped like checkout completion rate. An SLO (service level objective) is the promise drawn on top of it: "99.9% of requests succeed," "p99 stays under 400ms." In most systems SLOs are dashboards for humans. In this system they become executable: the canary's health gate is an SLO comparison evaluated by software every 30 seconds, and the software - not a person - decides whether the rollout proceeds or reverses. The crucial subtlety is WHAT to compare against. An absolute threshold ("error rate must stay under 1%") breaks in practice, because error rates move with time of day, traffic mix, and upstream weather. The honest comparison is canary versus baseline: the new version and the old version are serving the same traffic at the same minute, so any difference between their SLIs is attributable to the version - the deployment is a controlled experiment, with the old version as the control group. "Canary error rate may not exceed baseline's by more than 0.5 points" survives contact with reality; "under 1% always" does not. This is why the design insists the baseline keeps running and serving during the whole rollout: it is not just the rollback target - it is the control group that makes the health verdicts meaningful.
Every other design in this set scales along some volume axis - requests, messages, bytes, CPU-seconds. This one does not: a few thousand concurrent rollouts is nothing, and PostgreSQL plus a small worker fleet runs the whole control plane. What grows instead is consequence: each deployment is a change to a live system with real users, and the design's job is to control how much damage a bad version can do before it is detected and reversed. Read every mechanism in this design through that lens and the system becomes coherent. Canary percentages keep the experiment small (5% of users meet the new version before 100% do). Bake times and SLO gates keep it measured (exposure grows only after evidence). The always-running baseline keeps it reversible (rollback is a traffic shift, not a scramble). Feature flags keep behavior decoupled from binaries (a bad feature dies by toggle, not by redeploy). Approval gates and windows keep humans in the loop exactly where judgment beats automation. Small, measured, reversible, decoupled - four properties, one goal. Two structural notes fall out. First, the one heavy integration is the metrics platform (~800 queries/sec) - this system is a decision-maker wrapped around someone else's measurements, and its verdict quality is capped by the quality of those signals. Second, the reliability inversion from the requirements: because deployments are how incidents get fixed, the deploy system must not share fate with what it deploys - it runs on its own isolated infrastructure, and its failure modes must always leave traffic in a safe, static state. The doctor cannot be one of the patients.
[CI (Design a CI System)] --validated artifact v2.4.1--> [Artifact Registry]
(immutable, checksummed,
provenance recorded)
|
promotion policy: dev auto -> staging auto
-> prod needs approval + window
v
[Approval Gate] --approved--> [Deployment Orchestrator (state machine)]
|
v
[Strategy Engine: canary]
|
step 1: canary instances get v2.4.1 v
[Traffic Controller] ---- 5% ----> [canary: v2.4.1]
---- 95% ---> [baseline: v2.3.8] (keeps running:
control group AND
rollback target)
|
every 30s during the bake v
[Health Evaluator] <---- SLI queries ---- [Metrics Platform (external)]
compare canary vs baseline: error rate, p99, custom SLIs
| |
healthy for 10 min breach, 2 windows in a row
v v
next step: 25% -> 50% -> 100% [Rollback Engine]
then drain baseline shift 100% -> baseline (seconds)
| mark failed + store evidence
v |
[Audit Log: who, what, where, every verdict, full timeline]Three standard strategies, one honest comparison - and a house default, because "it depends" with no default is a non-answer. Rolling update: replace instances a few at a time until done. Cheapest (no extra capacity), but there is no traffic control and no measurement gate - by the time you notice a problem, most of the fleet may be running it, and "rollback" means deploying the old version again, which takes as long as deploying did. Right for dev and staging, where the blast radius is developers. Blue-green: run a complete duplicate environment (green) with the new version, test it, then switch ALL traffic at once; rollback is switching back - instant and total. The costs: 100% duplicate capacity, and exposure is all-or-nothing - the switch is instant, but so is the blast radius, and no gradual evidence was gathered on the way. Canary: run the new version on a small slice, shift traffic in measured steps, gate each step on health versus baseline. Costs ~5% extra capacity instead of 100%, gathers real-traffic evidence BEFORE committing everyone, and rollback stays a seconds-fast traffic shift the whole way (the baseline never stopped running). The price is time (~40 minutes, not one switch) and a real dependency on good metrics. The defense of canary as default: it is the only strategy that measures before it commits, and this whole system's thesis is blast-radius control. Blue-green earns its double cost where instant, total cutover matters more than gradual evidence (a payment-provider migration you cannot run half-and-half). And one warning that catches people: blue-green does NOT dodge the database problem - both colors share one database, so the schema-compatibility discipline (see the migrations flow) applies to every strategy equally.
It would be cheaper to keep only outcomes - "deployed" or "rolled back." Storing every intermediate verdict is a deliberate choice with three payoffs. First, the 2am question. When software rolls back a deployment automatically, the on-call engineer's first question is "why?" - and the answer must be evidence, not a shrug: "at 14:32, canary error rate was 2.1% against a baseline of 0.3%, breaching the 0.5-point margin for two consecutive windows." Automated decisions that affect production must be able to show their work, or engineers learn to distrust and disable the automation - and a disabled safety system is worse than none, because everyone believes it is still there. Second, tuning. Thresholds and bake times are guesses until history grades them: verdicts from hundreds of rollouts reveal which gates flap on noise (too tight), which breaches were caught at 5% that would have been outages at 100% (the system paying for itself), and which slipped through (bake too short - lengthen it). Third, the audit story. The deployment record - who triggered it, who approved it, every verdict, every transition, the outcome - is the flight recorder for the riskiest routine operation in software. Append-only is non-negotiable for the same reason flight recorders do not have erase buttons. The storage cost of all this honesty, per the estimates: approximately nothing. There is no tradeoff here to agonize over - just a decision to keep the evidence.
Database Schemas (PostgreSQL - the control plane is deliberately small)
- artifacts
artifact_id UUID PRIMARY KEY,
service_id UUID NOT NULL,
version VARCHAR(100) NOT NULL, -- "v2.4.1"
checksum CHAR(71) NOT NULL, -- "sha256:..." verified on receipt
provenance JSONB NOT NULL, -- { repo, commit_sha, ci_build_id }
created_at TIMESTAMP DEFAULT NOW(),
UNIQUE (service_id, version) -- tags never move: immutable bytes
- deployments (one rollout = one state-machine row)
deployment_id UUID PRIMARY KEY,
service_id UUID NOT NULL,
environment env_enum NOT NULL, -- dev | staging | production
artifact_id UUID NOT NULL,
strategy strategy_enum DEFAULT 'canary',
strategy_config JSONB,
-- { steps: [5,25,50,100], bake_minutes: 10,
-- gates: { error_rate_margin: 0.005, latency_p99_ratio: 1.2,
-- min_requests_per_window: 500 } }
status deploy_status_enum, -- pending_approval | rolling_out |
-- paused | succeeded | rolled_back
current_step INT DEFAULT 0,
initiated_by UUID NOT NULL,
created_at TIMESTAMP, started_at TIMESTAMP, finished_at TIMESTAMP,
-- one rollout at a time per service+environment:
UNIQUE (service_id, environment)
WHERE status IN ('pending_approval','rolling_out','paused')
- health_verdicts (append-only: the evidence)
verdict_id BIGSERIAL PRIMARY KEY,
deployment_id UUID NOT NULL,
step INT, evaluated_at TIMESTAMP DEFAULT NOW(),
canary_slis JSONB, -- { error_rate: 0.021, p99_ms: 512, requests: 8140 }
baseline_slis JSONB, -- { error_rate: 0.003, p99_ms: 240, requests: 154300 }
result verdict_enum, -- healthy | breach | insufficient_data
detail TEXT -- "error_rate 2.1% exceeds baseline 0.3% + 0.5pt margin"
- approvals
approval_id UUID PRIMARY KEY,
deployment_id UUID NOT NULL,
approver_id UUID NOT NULL, -- policy may require != initiated_by
approved_at TIMESTAMP DEFAULT NOW(),
conditions JSONB
-- structurally absent: approvals for rollbacks. Rollback never waits.
- audit_events (append-only; an editable audit log is not an audit log)
event_id BIGSERIAL PRIMARY KEY,
deployment_id UUID, actor VARCHAR(100), -- human id or "health-evaluator"
action VARCHAR(100), -- approved | step_promoted |
-- rollback_triggered | ...
detail JSONB, at TIMESTAMP DEFAULT NOW()
- configs (versioned per service + environment)
service_id UUID, environment env_enum, version INT,
values JSONB, -- injected at deploy time
secret_refs JSONB, -- pointers into the secrets manager, never values
PRIMARY KEY (service_id, environment, version){
"service_id": "svc_payment",
"environment": "production",
"artifact_version": "v2.4.1",
"strategy_config": {
"steps": [5, 25, 50, 100],
"bake_minutes": 10,
"gates": {
"error_rate_margin": 0.005,
"latency_p99_ratio": 1.2,
"min_requests_per_window": 500,
"custom_slis": ["checkout_completion_rate"]
}
}
}{
"deployment_id": "dep_8f3e2d1c",
"status": "pending_approval",
"approval_url": "https://cd.example.com/approvals/dep_8f3e2d1c",
"window": "weekdays 09:00-16:00 UTC",
"estimated_duration_minutes": 45
}
// Production policy requires a human approval inside the window. The
// gates compare canary AGAINST BASELINE (margins and ratios), not
// against absolute numbers - the old version serving the same traffic
// is the control group.{
"deployment_id": "dep_8f3e2d1c",
"service": "payment-service",
"environment": "production",
"status": "rolling_out",
"traffic": {
"canary_version": "v2.4.1", "canary_percent": 25,
"baseline_version": "v2.3.8", "baseline_percent": 75
},
"current_step": { "step": 2, "of": 4, "baking_minutes_left": 4 },
"latest_verdict": {
"result": "healthy",
"canary": { "error_rate": 0.004, "p99_ms": 251, "requests": 8140 },
"baseline": { "error_rate": 0.003, "p99_ms": 240, "requests": 24390 },
"evaluated_at": "2026-07-06T14:31:30Z"
},
"timeline": [
{ "at": "14:05", "event": "approved by oncall-lead" },
{ "at": "14:06", "event": "step 1: 5% traffic to v2.4.1" },
{ "at": "14:16", "event": "step 1 healthy: promoted to 25%" }
]
}
// Side-by-side canary/baseline SLIs from the same minute: the verdict
// is a comparison, not a threshold. Exposure grew to 25% only after
// ten minutes of evidence at 5%.{
"reason": "checkout_completion_rate dropping on canary"
}{
"deployment_id": "dep_8f3e2d1c",
"status": "rolled_back",
"traffic_restored_in_seconds": 8,
"restored_version": "v2.3.8",
"evidence": {
"trigger": "manual",
"last_verdicts": [
{ "result": "breach",
"detail": "checkout_completion_rate 91.2% vs baseline 96.8%" }
]
}
}
// Eight seconds, because rollback is a traffic-weight change - the
// baseline never stopped running. No approval was requested: rolling
// forward can wait on humans; rolling back never does. The automated
// path (two consecutive breach verdicts) produces this same response
// with trigger: "health-evaluator".{
"verdict_id": 918224,
"deployment_id": "dep_previous",
"step": 1,
"evaluated_at": "2026-07-02T09:14:00Z",
"canary": { "error_rate": 0.021, "p99_ms": 512, "requests": 6120 },
"baseline": { "error_rate": 0.003, "p99_ms": 244, "requests": 118240 },
"result": "breach",
"detail": "error_rate 2.1% exceeds baseline 0.3% + 0.5pt margin (window 2 of 2) -> rollback"
}
// Why store every one: this row IS the answer to the 2am question
// "why did it roll back?" - and hundreds of these across the fleet
// are how bake times and margins get tuned from evidence instead of
// vibes. It also documents a catch at 5% that would have been an
// outage at 100%: the system paying for itself.// 409 - one rollout at a time per service+environment
{
"error": "deployment_in_progress",
"message": "payment-service is mid-rollout in production (dep_8f3e2d1c). Pause or complete it first."
}
// 403 - deployment window closed (approval exists, window does not)
{
"error": "outside_window",
"message": "Production deploys allowed weekdays 09:00-16:00 UTC",
"next_window_opens": "2026-07-07T09:00:00Z"
}
// Note what has NO error case: rollback. It is valid in every state,
// in every window, for every caller with access to the service.Bake time: the deliberate waiting period at each canary step where nothing is promoted and everything is measured. Its length is a tradeoff between detection power and deploy speed: ten minutes catches error spikes and latency regressions, but not a slow memory leak or a code path that runs nightly - which is why bake times are tuned from verdict history, and why feature flags exist as the layer for what canary cannot see. Baseline comparison: judging the canary against the old version serving the same traffic at the same moment, rather than against an absolute threshold. It turns the rollout into a controlled experiment - the baseline is the control group - and automatically cancels out confounders like time-of-day, traffic mix, and upstream slowness that break absolute thresholds. The corollary: keep the baseline running and receiving traffic throughout the entire rollout. Expand-migrate-contract: the three-deploy discipline for schema changes. Expand (add the new shape; old code ignores it), migrate (new code writes both, reads new; backfill), contract (remove the old shape - in a later deploy, only after nothing that needs it will ever run again). It exists because traffic rolls back in seconds and schemas do not roll back at all, and because canary runs two versions against one database simultaneously, making N/N-1 compatibility a structural requirement of the strategy, not a nicety.