How to stop governance from depending on someone filling in a field by hand
Metadata automation in data pipelines means capturing lineage, sensitivity classification, quality, and ownership as part of the pipeline’s own execution, instead of as manual documentation added after the fact.
In multi-cloud environments, the added challenge is that platforms like Databricks Unity Catalog and AWS Glue Data Catalog / Lake Formation don’t share governance natively: federation between catalogs lets you query tables from one side on the other, but it doesn’t automatically propagate permissions, tags, or policies between them.
That’s why, before automating anything, you need to explicitly decide where the source of truth for policy lives, or evaluate a neutral open catalog like Apache Polaris or Apache Gravitino when the multi-engine commitment is real and long-term.
Automating metadata isn’t about adding more fields to fill in. It’s about designing the pipeline so metadata generates itself, as a side effect of running.
It’s a familiar scene in architecture review meetings: someone asks who owns a specific table, and the answer takes three days and two meetings to surface. The catalog says one thing, the pipeline does another, and the documentation in Confluence hasn’t been updated in eight months. None of the three sources is necessarily wrong; nobody is keeping them in sync, because keeping them in sync by hand isn’t a job that scales with the number of pipelines.
The pattern is a familiar one for any CTO managing more than one data platform: every team documents things their own way, with its own idea of what counts as a critical table, and with varying discipline about keeping that documentation current. The result isn’t a lack of governance on paper (there’s usually a framework, some guidelines, even a committee), but a permanent gap between what the documents say and what the pipelines actually do in production.
When the organization also operates across more than one cloud or platform; typically Databricks for transformation and ML, and native AWS, Azure, or GCP services for the rest of the stack, the problem multiplies: each platform has its own catalog, its own permissions model, and its own way of recording lineage. Without an explicit architectural decision, the result is duplicated catalogs and policies that say different things about the same data.
Automating metadata means lineage, sensitivity classification, quality metrics, and ownership are generated as part of the pipeline’s execution; read from the code itself, from the execution engine, or from declarative rules, rather than as a separate documentation task someone has to remember to do afterward.
This carries an important design implication: metadata stops being a person’s responsibility and becomes the pipeline’s responsibility. If the pipeline runs, the catalog updates. If the pipeline’s schema changes, the lineage reflects it automatically. No one has to remember anything.
Unity Catalog organizes data assets in a three-level namespace (catalog.schema.table) on top of an account-level metastore, typically one per cloud region, to which every Databricks workspace connects across AWS, Azure, or GCP, without needing to move the data. Its key differentiator versus a traditional catalog is that it unifies governance not just for tables, but also for files, machine learning models, notebooks, and, increasingly, the tools used by AI agents, with automatic column-level lineage and fine-grained access control.
On the AWS side, AWS Glue Data Catalog acts as the metadata repository for Athena, Redshift Spectrum, or EMR, and is the technical metastore that AWS Lake Formation relies on to apply permissions through LF-Tags: tags that let you define attribute-based access control instead of permission-by-permission access. The important limitation to keep in mind: both Lake Formation permissions and LF-Tags are scoped to the AWS account where they’re created, and they don’t automatically propagate to other accounts or other clouds.
This is where it’s worth clearing up a common misunderstanding in multi-cloud architecture. AWS Glue has added catalog federation with remote Iceberg catalogs, including Unity Catalog, which lets engines like Athena or Redshift query tables governed by Unity Catalog without duplicating the data. That’s genuinely useful for consumption, but it’s federation for querying, not for governance: a table becomes queryable from the other side, but it doesn’t become governed by the other side. Policies, sensitivity tags, and business lineage keep living wherever they were originally defined.
For an organization with a real, sustained commitment to more than one engine, this makes it increasingly worth evaluating open catalogs like Apache Polaris; a REST-based Iceberg catalog with fine-grained RBAC and native credential vending across AWS, Azure, and GCP; or Apache Gravitino, designed as a catalog of catalogs that federates heterogeneous metadata without forcing everything into a single proprietary platform.
The pattern that actually reduces operational overhead is capturing lineage and quality at execution time, at the processing engine level, rather than reconstructing them later by parsing logs or asking every team to document them. When lineage is generated as part of how the pipeline runs, it stays accurate even as the pipeline changes; when it’s documented by hand, it starts drifting from reality the moment a schema change goes unreported.
A catalog you have to update by hand isn’t metadata governance. It’s a to-do list shaped like a database.
Is your data team spending more time keeping catalogs in sync by hand than building new pipelines?
At Galde, we design automated metadata governance architectures on Databricks, AWS, and Unity Catalog, tailored to each organization’s actual level of multi-cloud complexity.
Through data platforms, Galde designs and implements pipeline architectures with automatic capture of lineage, quality, and sensitivity classification, on Databricks, Unity Catalog, and the native governance services of AWS, Azure, or GCP.
When organizational complexity justifies it (multiple heterogeneous sources, a need to expose data to business applications rather than just dashboards), we also assess whether a semantic layer like Palantir Foundry adds value on top of the catalog architecture.
This work is grounded in the same operational data governance approach we apply across every domain: automate governance in the tooling, rather than leaving it to each data engineer’s individual discipline.
Manual metadata governance doesn’t fail for lack of effort from the teams involved: it fails because it doesn’t scale. Every new pipeline, every derived table, and every additional cloud multiplies the effort required to keep catalogs, lineage, and classification current by hand. Automating that capture as part of pipeline execution (and explicitly deciding where the source of truth lives when more than one platform is involved) is what lets data governance grow at the same pace as the platform, instead of becoming the bottleneck that holds it back.
It can query them through catalog federation, but it can’t govern them in the sense of applying its own access policies or tags to them. Policies for those tables are still defined in Lake Formation; Unity Catalog just makes them queryable from the Databricks side.
Manual cataloging depends on someone documenting every table or pipeline, which guarantees drift the moment anything changes. Automation generates lineage, classification, and quality as part of the pipeline’s own execution, so it stays current without human intervention.
They can coexist, and in practice that’s the norm in mid-size and large organizations. What matters is explicitly deciding which catalog is the source of truth for policy for each data domain, rather than assuming federation between the two automatically solves governance.
It’s the application of rules or models that detect and tag personal or sensitive data the moment it enters the system. It matters especially in multi-cloud because, without it, each platform can end up classifying the same piece of data differently, creating inconsistencies in access control.
By explicitly assigning one catalog as the source of truth per data domain, treating any federation to other platforms as a read-only view, and considering open catalogs like Apache Polaris or Apache Gravitino when the number of engines and clouds involved makes it impractical to keep a single proprietary platform as the center of governance.
