Enabling Evolutionary Database Improvement: database branching with Lakebase, continued

This collection revisits the methodolgy of Evolutionary Database Design, twenty years later. A key constraint to database-changes-as-code has all the time been round shared database assets. With copy-on-write branching in Databricks Lakebase, a one-second, zero-storage-at-creation department of a terabyte-scale manufacturing database is now an O(1) operation, and the constraint that stored Apply #4 (all people will get their very own database occasion) aspirational has lifted.  On this collection, the authors describes what modifications when the constraint lifts: not the methodology, that holds, however the practices that emerge for the primary time, the team-scale governance that turns into automated, the position evolution for the DBA, and the brand new functionality that brokers share with their human counterparts.

Jen is the developer character from Evolutionary Database Design. In that essay she applied a database refactoring, splitting an inventory_code area into location_code, batch_number, and serial_number, as a routine consumer story, illustrating that DBAs and builders can collaborate, schemas can evolve in small increments, and migrations carry the change ahead safely.

The collection picks up with Jen twenty years later. The methodology she follows is similar one she adopted in 2003. What’s new is the technical functionality beneath her workflow, enabled by the lakebase structure: copy-on-write database branching, which makes the practices she has been studying about operationally actual at manufacturing scale. Throughout the three elements of this collection she is similar Jen at three scopes, her day (Half 1), her new playbook (Half 2), and her crew (Half 3).

Half 1 walked Jen by way of one characteristic. The practices she adopted had been described within the 2003 Evolutionary Database Design essay, expanded within the 2006 Refactoring Databases e book , and introduced into the CI/CD pipeline within the 2010 Steady Supply e book (Chapter 12).

Unique Seven Practices

The 2003 essay named seven practices. 5 of the seven had limitations of their utility till 2026.

1. DBAs collaborate carefully with builders.
2. All database artifacts are model managed with utility code.
3. All database modifications are migrations.
4. All people will get their very own database occasion.
5. Builders repeatedly combine database modifications.
6. All database modifications are database refactorings.
7. Builders can replace their databases on demand.

Limitations of their utility

• Apply #1 (DBA collaboration). Each schema change had production-scale penalties if it obtained unfastened, so DBA assessment remained synchronous and gating. Collaboration was constrained by the DBA’s calendar.
• Apply #4 (All people will get their very own database occasion). Licensing prices, infrastructure prices, DBA time. Aspirational on most groups. Most groups fell again to shared improvement databases and accepted the competition.
• Apply #5 (Steady integration of database modifications). The 2010 Steady Supply wave introduced migrations into the pipeline, however the pipeline ran migrations in opposition to shared goal databases. Per-pipeline isolation was lacking.
• Apply #6 (All database modifications are refactorings). Making use of every refactoring required apply areas (take a look at databases) that almost all groups didn’t have at PR granularity.
• Apply #7 (Builders replace on demand). Builders might run migrations in opposition to shared environments on demand, however couldn’t safely experiment, as a result of their experiments would have an effect on others.

What Modified

The expertise launched by Databricks Lakebase removes the roadblocks to the implementation of the above practices. Databricks Lakebase is a managed Postgres database that makes use of the identical object storage layer (the info lake) that the remainder of the Databricks lakehouse runs on. The database’s information lives in shared, sturdy storage – successfully S3 buckets; the Postgres engine runs as a separate compute layer above it. Compute and storage scale independently. The engine can scale up below load, down when site visitors drops, and to zero when idle. Lakebase is built-in with the unity catalog permitting for unified governance throughout a number of environments.

Copy-on-write database branching is what the decoupled structure makes sensible at scale. A department creates a brand new pointer into the identical shared storage with a divergence marker. Till the department writes, it shares all pages with its father or mother. When the department writes, solely the modified pages diverge; the father or mother stays untouched. Branching is a metadata operation, no information copy required, finishing in roughly one second no matter father or mother dimension. The branches preserve information within the modified pages solely.

When the technical price of a department is decoupled from the scale of the info inside it, the constraint behind Apply #4 (all people will get their very own database occasion) is unblocked. Per-developer, per-PR, per-experiment branches grow to be routine. The compensating layer above can come out. Mocks come out of the take a look at loop, changed by actual Postgres on a per-test department. Shared staging stops being the one place to check schema modifications. In-memory database substitutes (H2, SQLite) come out of the unit-test layer. The devops tax to create docker primarily based containers to run native databases is just not obligatory, DBA ticket queues for provisioning shrink as a result of branches are self-service.

The expertise is what allows methodology optimizations and completes the purpose of the practices behind the unique 2003 publish.

Rising practices for 2026

Lakebase copy-on-write database branching lifts the earlier limitations on the unique practices and allows 4 extra ones for elaboration.

1. DBAs collaborate carefully with builders. With the schema diff posted on each PR, the DBA opinions async, like every other code reviewer. Because the provisioning tax is negligible the DBA’s now have the bandwidth to assessment and perhaps even work with the builders to create the options within the first place as a substitute of reviewing publish implementation.
2. All database artifacts are model managed with utility code. The schema diff, the database migrations and migration take a look at outcomes now be a part of the artifact set.
3. All database modifications are migrations. Plus a brand new authorship rule: idempotency. Permitting all merged migrations to be deployed to downstream environments like QA, staging and Manufacturing robotically.
4. All people will get their very own database occasion. Operational at per-developer, per-PR, and per-experiment granularity. Permitting builders to experiment on a number of options and never simply accept the primary resolution that works. This freedom will end in optimum options being developed for manufacturing scale.
5. Builders repeatedly combine database modifications. Operational at PR granularity. Each PR runs by way of CI by itself department.
6. All database modifications are database refactorings. The 2006 catalog nonetheless applies; branches offer you an inexpensive rehearsal house to attempt the refactorings on numerous sized environments, migrations that labored in improvement databases nearly all the time didn’t carry out in manufacturing, now you’ll be able to attempt migrations in bigger dimension databases to make sure they’re performant.
7. Builders can replace their databases on demand. “On demand” now means one second, remoted, in opposition to production-shaped information.
8. Damaging testing as a default choice. Blast radius is zero; reset prices one second. Don’t have to fret about, “Will my take a look at pollute information for others?”, since now after each take a look at run I can get a brand new department.
9. A/B variant prototyping on the database degree. Construct two designs on parallel branches; work with the bigger crew and the DBA’s to debate the answer and do a present and inform, then preserve the winner.
10. With the built-in unity catalog, governance is designed as soon as, inherited by all of the branches. Insurance policies comply with every department robotically, we’ll talk about this intimately in Half 3.
11. Agent-as-practitioner with the identical branching functionality. Brokers get branches, not manufacturing, we’ll talk about this intimately in Half 3.

How the workflow runs in CI

The 2 practices that get improved probably the most due to the potential offered by Lakebase are #4 (all people will get their very own database occasion) and #5 (builders repeatedly combine database modifications). Not simply everybody will get a database, each PR or each department or a number of databases per department may be had with negligible price and time. The mechanism may be automated utilizing two GitHub Actions workflow templates that may be scaffolded into each mission.

Per-PR department creation. pr.yml triggers on pull_request: [opened, synchronize] and creates ci-pr- forked from the PR’s base department:

The identical job applies migrations in opposition to the CI department and runs the appliance’s take a look at suite in opposition to actual Postgres. Full instance of the workflow may be discovered right here: pr.yml

Schema diff posted on the PR. The identical pr.yml job dumps the schema of each branches, codecs the diff, and posts it as a PR remark. That is what lets the DBA assessment async like every code reviewer (Apply #1, re-cast):

The DBA, the code reviewer, the crew and any agent writer of the migration see the identical artifact on the PR.

Department cleanup on merge. merge.yml destroys  the CI Pull Request Department ci-pr- and the linked characteristic department’s Lakebase department the second the PR merges:

Full instance of the merge workflow may be discovered right here: merge.yml With so many branches floating round it might be price having a weekly cleanup script that removes orphaned and unused branches, an instance workflow may be discovered right here: cleanup-orphans.yml. Lastly, if the merge is right into a ‘tiered’ department, e.g. used for staging or predominant/manufacturing (this idea can be additional elaborated partially 3), the workflow could also be orchestrated to chop a contemporary department from the tier to check migrations in opposition to it earlier than making use of the ultimate migration to any setting the place customers are discovered dwell and always including information.

Collectively these workflows implement each PR will get its personal database and branches are ephemeral as properties of the pipeline, not developer disciplines.

The brand new playbook for Evolutionary Database Improvement

Apply #1: DBAs collaborate carefully with builders

Rule. The DBA collaborates with builders all through the characteristic, not simply at gate-review time. The collaboration is asynchronous, inline with the PR, the best way different code reviewers take part.

Why is that this a sturdy behavior now? The schema diff and migration take a look at outcomes land on each PR robotically (see How the workflow runs in CI above). The DBA opinions a concrete artifact on their very own schedule. The migration has already run in opposition to a real-data CI department, so the DBA doesn’t have to mentally simulate the change.

Mechanics:

• The DBA is a CODEOWNER on schema-affecting paths (migrations/, db/, schema take a look at directories).
• The DBA opinions on the PR like every other reviewer, async.
• Overview focus shifts from will this break the database to is that this the suitable design? Has it been applied the suitable approach? Subjects: information integrity guidelines, indexing technique, design cohesion, future extensibility, long-term maintainability.

Anti-pattern. Not together with the DBA within the PR movement with there being respectable artifacts to assessment.

The DBA position additionally positive aspects new obligations in coverage administration and governance at crew scale. Half 3 covers these.

Apply #2: All database artifacts are model managed with utility code

Rule. Each SQL file, migration script, and schema take a look at lives in the identical repository as the appliance code. The schema diff and migration take a look at outcomes be a part of the artifact set as PR-time outputs.

Why is that this a sturdy behavior now? Branching provides two new artifacts to the set: the per-PR schema diff and the per-PR schema migration take a look at outcomes. Each are generated by CI from the precise department state. Each land within the PR as concrete proof about what was modified and the way the change migration script carried out.

Mechanics:

• Migration recordsdata in a versioned listing (migrations/,  db/migration/, alembic/variations/,  framework-dependent).
• Schema-affecting assessments within the take a look at tree alongside utility assessments.
• Schema diff generated per PR by CI, posted as a PR remark.
• Migration take a look at leads to the CI run abstract and PR remark.

Anti-pattern. Producing the schema diff exterior the PR movement (a separate dashboard the reviewer has to open). The artifact has to dwell the place the assessment occurs. Because the schema modifications are tied to the code modifications and breaking this dependency creates downstream issues for deployment.

Apply #3: All database modifications are migrations

Rule. No handbook ALTER TABLE in opposition to any setting. Each schema change is a versioned, checked-in migration script. Migrations are idempotent.

Why is that this a sturdy behavior now? The migration-as-artifact rule is unchanged from 2003. What’s new is the authorship self-discipline of idempotency. The identical migration runs in opposition to many branches over the lifetime of a transition, so it has to behave the identical approach each time. A migration that fails on re-apply is a bug.

Mechanics:

• Use migration frameworks like flyway, liquibase, Knex, Alembic or others, these frameworks preserve monitor of which migrations have been run and which haven’t been, this permits the crew to use a command like flyway migrate which simply applies the modifications that haven’t been utilized (by holding monitor of the modifications in a metadata desk)
• It’s higher to separate irreversible work throughout migrations. For instance a migration that provides new columns and drops the supply column in a single shot makes rollback tougher than they must be, so utilizing the develop first and contract later technique, offers many extra choices after a deploy cycle confirms no dwell readers reference it.
• The 2006 database refactoring catalog names which refactorings are reversible and which aren’t. Use it.

Anti-pattern. A migration that depends upon the schema being in a selected intermediate state due to native modifications made within the department. The migration should apply appropriately in opposition to any father or mother department that features prior migrations.

Apply #4: All people will get their very own database occasion

Rule. Each developer, each PR, each experiment, each take a look at run will get its personal Lakebase department.

Why is that this a sturdy behavior now? The additional effort to create docker containers, to put in native database servers, purchase licenses, hydrate empty databases with present schema and take a look at information is now not required. Only a easy create-branch Lakebase command branches a 1TB database in the identical one second as a 1MB database. No information is copied at creation; solely modified pages devour storage. Per-developer, per-PR, and per-experiment cases are routine.

Mechanics:

• Per-developer branches: created on demand by way of databricks postgres create-branch or the SCM extension’s branch-create movement.
• Per-PR branches: created robotically by CI on PR open (pr.yml), destroyed on merge or shut (merge.yml). See How the workflow runs in CI above for the PR and merge snippets.
• Per-experiment branches: forked off staging or manufacturing for design exploration; discarded after the experiment.

Anti-pattern. Sharing a improvement database throughout the crew “for comfort.” The contention-driven serialization Half 1 named comes again the second the branches collapse.

The place Jen’s instance extends. Her per-developer department was forked off staging at characteristic begin. The CI department was forked off staging on PR open. Her A/B exploration branches (Apply #9) had been forked off staging in parallel. 4 branches throughout one characteristic, all in seconds, all remoted.

Apply #5: Builders repeatedly combine database modifications

Rule. Each PR runs by way of CI in opposition to a contemporary Lakebase department, with migrations utilized and assessments run in opposition to actual Postgres.

Why is that this a sturdy behavior now? The CI pipeline has had migration self-discipline since 2010. What’s new is per-pipeline isolation: every PR will get its personal department, so integration runs in opposition to real-shaped information with out competition.

Mechanics:

• CI creates a department on PR open. See How the workflow runs in CI above for the PR snippet.
• Migrations utilized in opposition to the department by way of lakebase-migrate apply.
• Software assessments run in opposition to the migrated department by way of the ORM, no mocks.
• Schema diff posted on the PR.
• Department destroyed on merge or shut.

Anti-pattern. Operating PR validation in opposition to shared staging. The serialization comes again; the per-PR isolation property is misplaced.

Apply #6: All database modifications are database refactorings

Rule. Schema modifications comply with named refactoring patterns: Break up Column, Rename Column, Transfer Column, Exchange Kind, and so forth. Every has specific transition mechanics (preserve previous plus new in parallel, populate from previous, swap readers, drop previous).

Why is that this a sturdy behavior now? The 2006 catalog at databaserefactoring.com names 70+ refactorings with labored examples. What’s new in 2026 is an inexpensive place to rehearse the transition mechanics: a developer department absorbs the rehearsal; the CI department verifies; manufacturing sees solely the verified consequence.

Mechanics:

• Establish the refactoring by title. Look it up within the catalog.
• Apply the named transition mechanics on a per-developer department. Validate in opposition to production-shaped information.
• Open the PR. CI runs the migration by itself department and posts the diff.
• Merge as soon as the diff and the take a look at outcomes land approval.

Anti-pattern. A one-off schema change that doesn’t map to a named refactoring. The 70+ catalog covers the frequent circumstances; if you end up exterior it, you might be doubtless combining a number of refactorings into one migration and will cut up.

The place Jen’s instance extends. Her V87 migration is the Break up Column refactoring: splitting inventory_code into location_code, batch_number, and serial_number. The catalog web page at databaserefactoring.com/SplitColumns.html names the transition mechanics. Her department was the rehearsal house; the PR’s CI run was the verification.

Apply #7: Builders can replace their databases on demand

Rule. A developer can refresh their department’s database state on demand: reset to the father or mother’s present state, fork a contemporary department off manufacturing, discard an experimental department, share a department with a teammate. All in seconds.

Why is that this a sturdy behavior now? “On demand” in 2026 means one second, remoted, in opposition to production-shaped information. None of those operations seek the advice of ops calendars or DBA queues.

Mechanics:

• Reset: delete and recreate the department from its father or mother.
• Fork off manufacturing: databricks postgres create-branch --source manufacturing.
• Discard: databricks postgres delete-branch.
• Share: hand the department endpoint to a teammate for a pairing session.

Anti-pattern. Treating any department as sturdy past its objective. The migration is the sturdy artifact; the department is the workspace.

Apply #8: Damaging testing as a default choice

Rule. When the blast radius of a damaging motion is zero, damaging testing turns into a every day choice slightly than a quarterly train.

Why is that this a sturdy behavior now? A department resets in a single second. Something you do to a department may be undone by making a contemporary one off the identical father or mother. Damaging assessments cease needing ops calendars and approval gates.

Issues that now match inside a traditional characteristic cycle:

• “What occurs if my migration fails midway by way of the UPDATE assertion?” Run it. Kill the method at 50%. Confirm the rollback works. Reset.
• “What occurs if a backup-restore is mid-way by way of when a failover triggers?” Simulate the partial state. Confirm the appliance’s conduct. Reset.
• “What’s the precise time-to-recover for our DR runbook?” Run the runbook. Measure. Reset.
• “Does this migration run efficiently in opposition to manufacturing form information or dimension” confirm it earlier than c

Cultural impact. When reset prices nothing, groups cease treating the take a look at database as a treasured useful resource. Checks may be aggressive. Cleanup may be skipped, as a result of the subsequent department begins contemporary.

The place Jen’s instance extends. Earlier than opening her PR, she took the production-shaped information on her department and intentionally corrupted round one p.c of the inventory_code values to appear like edge circumstances: lacking digits, embedded areas, trailing whitespace. The sorts of artifacts that historic information accumulates. She ran her migration. Two rows failed her substring math. She fastened the script and re-ran. The department absorbed the damaging take a look at. Manufacturing by no means noticed it.

Apply #9: A/B variant prototyping on the database degree

Rule. When two designs are in competition, construct them on parallel branches, evaluate in opposition to production-shaped information, and preserve the higher resolution.

Why is that this a sturdy behavior now? Per-branch price is close to zero. Exploring two schema designs now not requires selecting one prematurely, and it now not requires a provisioning course of most groups is not going to undertake for an exploratory query.

Mechanics:

• Create two branches off the identical father or mother.
• Apply Design A’s migration to at least one department, Design B’s migration to the opposite.
• Run the appliance in opposition to every. Measure what issues: question latency on the frequent learn path, migration time at manufacturing quantity, index footprint, lock competition below simulated load.
• Choose the higher resolution. Discard the sub optimum resolution. Doc the choice within the PR description so the subsequent one who has to increase the schema is aware of what was thought of and why this design was picked.

Anti-pattern. Operating A/B prototypes with out writing down the choice and the reasoning. The branches are gone in a second; the design determination must be everlasting.

The place Jen’s instance extends. She thought of two designs for the situation/batch/serial characteristic: three new columns on the prevailing stock desk, or a separate inventory_attributes lookup desk keyed by inventory_id, anticipating that extra attributes could be added later. She constructed each on parallel branches off staging. She ran the appliance’s learn path in opposition to every, measured question efficiency in opposition to production-shaped information, and checked out how every migration would scale to manufacturing volumes. The lookup-table model carried out worse on the frequent learn path as a result of each stock show required a be a part of. She shipped the columns model, threw away the lookup-table department, and left a observe within the PR description: Thought of lookup-table model; rejected as a result of the frequent learn path turns into a be a part of. Revisit if greater than 5 attributes accumulate.

What Jen’s New Playbook Exhibits

We’ve named the seven practices from 2003 with the constraints that stored 5 of them aspirational, and re-cast them for 2026 as soon as branching landed, and added 4 new practices that branching allows. Eleven practices whole within the new playbook for Evolutionary Database Improvement, 9 of that are defined above.

In Half 1 – Jen’s story: one characteristic, one database we noticed Jen work by way of one characteristic: she paired a code department with a Lakebase department, ran an actual migration in opposition to production-shaped information in seconds, examined with out mocks, opened a PR with the schema diff posted inline, and merged with the migration utilized and the ephemeral branches cleaned up. Database change grew to become a part of regular improvement.

In Half 3 – Jen’s Crew at Scale, we take a look at the playbook at fifty builders, the DBA’s developed obligations in coverage administration and governance, and the brokers creating branches alongside Jen. Practices #10 and #11 get their full remedy there.

The Companion: Plugin Walkthrough covers the Lakebase SCM Extension for VS Code and Cursor.

A Lakebase App Dev Equipment for brokers, with a companion e-book for human practitioners, can be launched as a follow-on.