b8c.io All articles
Product & Delivery

The Rollback Plan That Forgot the Data

b8c.io

Application deploys get the ceremony. There is a release checklist, a canary, a Slack channel, and someone watching the error rate. Database migrations often get a ticket and a shrug. That split is backwards. Code can usually be rolled back. Data changes are the ones that leave a mark.

A schema migration can drop a column, rewrite a table, or change the meaning of a field that three services quietly depend on. By the time the application deploy looks healthy, the data may already be in a state that no previous version of the code can read. That is not a deploy risk. It is a product risk wearing a deploy risk costume.

Code Rolls Back. Data Usually Doesn't.

A rollback plan that says redeploy the previous container image is only half a plan. If the migration added a non-null column with a default that the old code never writes, the old code may still run. If the migration renamed a column, it will not. If the migration backfilled values based on a business rule that changed, rolling back the code does not undo the values.

The uncomfortable question is not whether the migration is reversible in theory. It is whether the data is reversible in practice. Can you reconstruct the previous state? Do you have a backup that is recent enough and a restore path that has been tested? Will customers see missing or duplicated records during the rollback window? Those questions belong in the release review, not in the incident channel after the fact.

Expand, Backfill, Contract—Then Breathe

The expand-contract pattern is not new, but it is still the most reliable way to make schema changes boring. First, expand: add the new column or table without breaking old readers. Deploy code that can handle both shapes. Then backfill in small, observable batches. Then switch reads and writes. Finally, contract: remove the old shape only after you are confident nothing depends on it.

The pattern works because it separates the schema change from the behavior change. It also gives you natural pause points. If the backfill starts producing errors, you stop. If the new read path shows unexpected nulls, you stop. You are not trying to roll back a single giant migration. You are turning one irreversible step into a series of reversible ones.

Migrations Need Their Own Dashboard

Most teams have dashboards for CPU, latency, and error rate. Few have dashboards for migration progress. That is a blind spot. A migration is a workload. It has throughput, lag, failures, and retries. It can lock tables, saturate replicas, and pile up dead tuples. If you cannot see those signals, you are deploying on faith.

Track batch progress, rows affected, replication lag, lock waits, and error counts. Put the migration job in the same observability system as the application. If the migration is long-running, make it resumable and idempotent. The goal is not to watch a progress bar for fun. It is to know whether the migration is healthy before a customer tells you it is not.

Rollback Plans Need a Data Chapter

A credible rollback plan answers three data questions. What state will the data be in after the deploy? Can the previous application version tolerate that state? If not, what is the restore or repair procedure, and how long does it take?

For customer-visible changes, add a fourth: what will customers see during the rollback? A repair script that takes forty minutes is not a rollback plan. It is an outage with a spreadsheet. If the answer is we will restore from backup, test the restore. A backup you have never restored is a hope, not a control.

Make the Irreversible Stuff Boring

The best way to reduce deploy risk is not to add more approvals. It is to shrink the irreversible surface. Use expand-contract. Keep migrations small. Separate schema changes from data changes. Make backfills resumable. Put feature flags around behavior changes so you can turn off the new path without touching the schema. Test the rollback in staging with production-like data, not three rows in a seed file.

And treat migrations as part of the deploy, not a pre-step someone runs while the release train is boarding. If the migration is risky, the deploy is risky. If the migration is routine, prove it with observability and a tested data recovery path.

Shipping faster only works when the boring parts are actually boring. Your rollback plan should cover the code, the configuration, and the data. Otherwise it is a story you tell yourself while the backfill runs.

All Articles

Related Articles

Blameless Doesn't Mean Toothless: What Incident Reviews Look Like When They Actually Work

Blameless Doesn't Mean Toothless: What Incident Reviews Look Like When They Actually Work

API Docs That Lie: The Documentation Drift Problem Quietly Wrecking Developer Trust

API Docs That Lie: The Documentation Drift Problem Quietly Wrecking Developer Trust

Twelve Repos, Four Languages, One Bug: The Hidden Velocity Tax of Distributed Systems Debugging

Twelve Repos, Four Languages, One Bug: The Hidden Velocity Tax of Distributed Systems Debugging