How to Stop Software Changes from Breaking Your Data Pipelines
Data contracts, compatible changes and automated tests, so that renaming a column stops being an incident
Quick Answer
Data pipelines break when software changes because they usually depend on the internal structure of an application's transactional database: its tables, its columns and its values. The team that maintains the application changes it to improve the product, without knowing that an analytics pipeline or a dashboard depends on it.
The solution combines four practices: publishing a stable interface instead of reading internal tables, agreeing a data contract between whoever produces the data and whoever consumes it, checking that contract automatically in CI before any change is deployed, and making necessary changes in compatible phases: expand, migrate and contract.
That way, a breaking change is caught before it reaches production, while it is still cheap to fix, rather than on a Monday morning in an empty dashboard.
A dashboard does not break when someone renames a column. It breaks because nobody knew that dashboard depended on it.
It is a familiar scene. On Friday, the product team ships an improvement to the ordering application and, along the way, renames the customer_id column to client_id to align it with the rest of the code. The release goes well. On Monday, the sales dashboard shows half the orders, the nightly pipeline has loaded nulls into the customer key and the data team spends the morning working out what changed.
Nobody did anything wrong. The product team had no way of knowing that someone read that column, and the data team had no way of learning about the change before it hit them.
The problem is not the change. It is that the change travels without warning from the team that makes it to the team that suffers it.
Why Application Changes Break Analytics
Most data platforms start by copying tables from transactional databases, either with periodic loads or with change data capture (CDC). It is the quickest way to make data available, but it creates a dependency nobody agreed to: the application's internal schema, designed to serve the product, unintentionally becomes the interface that reports, models and dashboards depend on.
The changes that break that dependency are not only the obvious ones:
- Structural changes: renaming or dropping a column, splitting a table in two or changing a data type.
- Changes in meaning: an amount that goes from including tax to excluding it, or a date that changes time zone.
- New values: a new value in a status field, such as
PARTIALLY_SHIPPED, that no analytics filter accounts for. - Changes in granularity: a table that goes from one row per order to one row per order line.
Structural changes usually break loudly: the pipeline fails and someone notices. The worst are changes in meaning and new values, because they break silently: the pipeline completes, the figures change and nobody notices until a business lead starts distrusting a report.
What a Data Contract Is and What It Should Include
A data contract is an explicit, versioned agreement between the team that produces a piece of data and the teams that consume it. It describes what is published and what guarantees it comes with, so that any change can be checked automatically against what was agreed. A good contract covers:
- Schema and types: which fields exist, what type they are and which are mandatory.
- Meaning: units, time zones, allowed values and how to interpret each field.
- Quality guarantees: key uniqueness, no nulls where there should be none, valid ranges.
- Freshness: how often the data is updated and with what maximum delay.
- Owner and notification channel: who answers for the data and where changes are announced.
- Version and change policy: what counts as a breaking change and how far in advance it is announced.
The contract lives as code, alongside the producer's code, in a readable format such as YAML. There are open standards, such as the Open Data Contract Standard, and tools that already enforce it in specific parts of the flow: dbt model contracts prevent publishing a model whose columns or types do not match what was declared, and the schema registries of event systems reject a new schema that is not compatible with the previous one.
How to Apply Data Contracts Without Slowing Product Teams Down
The goal is not for the product team to ask permission for every change. It is for compatible changes to flow without friction and for breaking ones to be caught automatically, in time.
1. Stop Reading Internal Tables: Publish an Interface
The first step is to separate what the application uses internally from what it offers externally. Instead of copying its internal tables, the application publishes a stable data interface: a dedicated view, an output table or domain events emitted with the outbox pattern. The team can reorganise its internal schema freely as long as it maintains that interface.
2. Check the Contract in the Producer's CI
Every time the application team opens a change, its continuous integration pipeline compares the resulting schema with the contract. If the change breaks it, the deployment stops and the team itself is notified, with the list of affected consumers. It is the same principle as software testing: catch the problem where it originates, not where it hurts.
3. Change in Phases: Expand, Migrate and Contract
Breaking changes are sometimes necessary. The way to make them without interrupting anyone is in three phases:
- Expand: the new column,
client_id, is added without removing the old one, and both are populated. - Migrate: consumers switch to the new one, within an agreed and visible deadline.
- Contract: once no consumer uses the old one, it is removed.
4. Version When There Is No Other Way
If the change is so deep that the old and new cannot coexist, a new version of the data or event is published in parallel, and the previous one is kept for an announced transition period.
5. Watch for What Slips Through
No contract covers everything. A data observability layer detects schema drift, drops in volume, delays and unexpected values, and alerts the data owner before a user does. Data tests within the pipeline itself, such as dbt tests, are the first line of that monitoring.
Who Is Responsible for What
Contracts work when responsibilities are clear:
- The team that produces the data owns the contract: it maintains it, respects its change policy and gives advance notice of breaking changes.
- The platform team automates the CI check, registers contracts in the catalogue and maintains end-to-end lineage so everyone knows who consumes each dataset.
- The consuming teams declare what they depend on and test their transformations against the contract, not against whatever happens to be in the table today.
Knowing who consumes each dataset is what turns a change into an informed decision. Without that map, every product team works blind.
The Hidden Cost of Not Doing It
Without contracts, platform maintenance becomes a string of reactive incidents. The data team spends a growing share of its time repairing what others change, instead of building what the business asks for. And every incident erodes something harder to recover than lost hours: the business's trust in its figures. When a director stops trusting a dashboard, they go back to their spreadsheet, and the investment in the platform loses much of its point.
Is every release of your applications a gamble for your dashboards?
At Galde, we design data platforms where changes at the source are caught before they are deployed, with stable interfaces, contracts checked in CI and observability that alerts the right owner, so that maintenance stops being a series of fires.
How Galde Can Help Protect Your Pipelines from Change
Through data platforms, we separate data interfaces from applications' internal schemas, automate contract checks in product teams' CI and deploy the observability that catches what slips through.
Through data governance, we define who owns each dataset, which change policy applies and how each contract is registered in the catalogue, so that the rules do not depend on anyone's memory.
And through generative AI, we apply the same discipline to the data that feeds your assistants and agents, which need stable sources to keep working: it is the difference between a Data Product and a Data Project.
Conclusion
Application changes are not going to stop, and they should not: they are the sign that the product is evolving. What can be prevented is every change reaching analytics as a surprise. With a stable interface, a contract checked automatically and changes made in phases, product teams gain the freedom to improve and data teams stop fighting fires.
Frequently Asked Questions
What is a data contract?
It is an explicit, versioned agreement between whoever produces a piece of data and whoever consumes it. It defines the schema, the meaning of each field, the quality and freshness guarantees, the owner and the change policy, and it is written as code so it can be checked automatically.
Do data contracts slow the development team down?
Not if they are applied well. Compatible changes pass without friction and only those that would break other teams are stopped. Catching them in CI is far cheaper than repairing them in production.
What is the difference between a data contract and a schema?
The schema describes the shape of the data. The contract adds meaning, quality and freshness guarantees, the owner and how changes are managed. A schema can be technically identical and still break the contract if the meaning of a field changes.
Is change data capture (CDC) enough to prevent breakages?
No. CDC faithfully replicates changes, including structural ones, so it propagates breakages faster. It works well when applied to a stable interface, such as an output table, rather than to the application's internal tables.
Where should we start if we have no contracts at all?
With the three or four datasets that feed the most critical reports. Identify who produces them, publish a stable interface, write their contract and add the check to the producer's CI. That alone removes most of the incidents that hurt the most.
