Data Lineage in Multi-Cloud Environments: The Challenge of Complete Traceability

Quick Answer

Data lineage is the record of the path a data point travels: which sources it comes from, which transformations change it and which reports it ends up in. In a multi-cloud environment, each cloud and each tool has its own catalogue with partial lineage, and the full path breaks at the boundaries between them.

The solution is to capture lineage automatically at runtime: every process reports, as it runs, which data it read and which it wrote. By using an open standard such as OpenLineage and a shared convention for naming datasets across all clouds, those events are consolidated into a single end-to-end graph.

That way, traceability stops depending on manual documentation and becomes reliable for audits, impact analysis and quality control.

Lineage documented by hand describes how the pipeline was designed. Lineage captured at runtime describes what actually happened.

An auditor asks where a specific figure in a regulatory report comes from. The answer requires reconstructing a journey that starts in an operational system in AWS, goes through a Spark job, is transformed in Databricks, aggregated in Snowflake and ends in a Power BI report. Three teams have to be brought together, outdated documentation reviewed and, a week later, the answer arrives with several "we believe"s.

Each tool, on its own, knew part of the path. None knew the whole path.

In multi-cloud, lineage is not lost inside each tool. It is lost at the boundaries between them.

Why Lineage Breaks in Multi-Cloud Environments

Each platform's native catalogue, such as AWS Glue Data Catalog, Microsoft Purview, Unity Catalog or Google Dataplex, captures lineage well for what happens within its scope. The problems appear at the hand-offs:

• Each catalogue sees only its part: Unity Catalog lineage ends where the data leaves for Snowflake, and Snowflake's begins without knowing where it came from.
• Inconsistent names: the same dataset has one name in storage, another in the catalogue and another in the report, and nobody can join the segments.
• Manual documentation: hand-drawn lineage diagrams go out of date with the first change in production.
• Processes off the radar: scripts run directly against a database, or an intermediate spreadsheet, break the chain without leaving a trace.

Static Lineage Versus Runtime Lineage

There are two ways to obtain lineage. Static lineage is inferred by analysing the code or SQL of the processes: it describes what the pipeline would do. Runtime lineage is captured when the process runs: it records which data was actually read and written, in which run, with which version of the code and with what result.

Static lineage is useful for anticipating the impact of a change before deploying it. But for an audit, or to investigate a wrong figure, the question is what actually happened, and only runtime lineage answers that. The sensible approach is to combine both, with runtime lineage as the source of truth.

OpenLineage: An Open Standard for Capturing Lineage at Runtime

OpenLineage is an open standard, hosted by the LF AI & Data Foundation, that defines how to describe each run of a process: an event with the job being run, the specific run, the input and output datasets, and optional facets with the schema, column-level lineage, quality metrics or the location of the code.

Its big advantage in multi-cloud is that it belongs to no cloud. There are integrations for Apache Spark, Apache Airflow, dbt and Apache Flink that emit the events without changing the processes' code; Marquez is its reference implementation for storing them, and several of the clouds' own services, such as Dataplex lineage in Google Cloud or Amazon DataZone, already accept OpenLineage events.

A simplified event looks like this:

{
  "eventType": "COMPLETE",
  "eventTime": "2026-09-22T06:12:03Z",
  "run": { "runId": "3f1c2a90-5b7e-4d21-9c3a-8e2f1b6d7a44" },
  "job": { "namespace": "databricks-prod", "name": "sales.daily_orders" },
  "inputs": [{ "namespace": "s3://raw-data", "name": "erp/orders" }],
  "outputs": [{ "namespace": "snowflake://acme-eu", "name": "ANALYTICS.SALES.DAILY_ORDERS" }]
}

How to Design a Multi-Cloud Lineage Architecture

1. Agree How Every Dataset Is Named Across All Clouds

It is the least glamorous step and the most important. If the same data gets different names in each segment, the events cannot be joined. OpenLineage proposes naming conventions by storage type, such as the bucket path in S3 or the account, database and schema in Snowflake, which should be adopted without exceptions.

2. Instrument the Execution Engines, Not the People

Reliable lineage is not written by anyone: the engines emit it. The Spark, Airflow and dbt integrations are switched on, and for SQL run directly in data warehouses, their own access history is used, such as Snowflake's ACCESS_HISTORY view, which records which objects and columns each query read and wrote.

3. Consolidate the Events into a Shared Graph

Events from every cloud are sent to a single lineage backend, whether Marquez or the corporate catalogue acting as the common point. There, the full path is joined up, from source to report, and exposed to data owners and compliance teams.

4. Go Down to Column Level Where It Matters

Not everything needs column-level lineage. But for regulated metrics, financial indicators and personal data, knowing which source column feeds each field of a report is what turns lineage into audit evidence.

5. Connect Lineage with Quality and Contracts

Lineage delivers its full value when combined with quality results: it lets you find the root cause of a wrong figure and run impact analysis before a change, alerting whoever depends on it. It is the foundation that lets data contracts know whom to protect.

What Audits Require and How Lineage Helps

More and more regulations require proving where a figure comes from, not just asserting it. In banking, the BCBS 239 principles require traceability in the aggregation of risk data. The General Data Protection Regulation requires knowing what processing is carried out on personal data and where it flows. And the EU Artificial Intelligence Act requires documenting the origin and processing of data in high-risk systems.

In every case, lineage captured at runtime, retained and queryable answers in minutes what a manual reconstruction answers in weeks, with verifiable evidence instead of an estimate. For detail on automating metadata with Unity Catalog and AWS services, see our governance architecture for multi-cloud data pipelines.

How Galde Can Help Guarantee the Traceability of Your Data

Through data governance, we define what needs column-level traceability, how datasets are named across all clouds and what evidence your audits need, with criteria we already apply when choosing data governance tools.

Through data platforms, we instrument the execution engines in AWS, Azure and Databricks with OpenLineage, consolidate the events into a shared graph and connect them with quality tests.

And through generative AI, we extend traceability to the data that feeds your models and assistants, an increasingly common requirement for AI systems.

Conclusion

In multi-cloud, complete traceability is not achieved by choosing the best catalogue, but by keeping lineage from depending on any single catalogue. Capturing every run with an open standard, naming data the same way across all clouds and consolidating everything into a shared graph turns lineage into what audits and quality teams need: the verifiable history of every data point.