Data Lineage and Provenance Tracking for Regulatory Compliance

Data lineage — the documented record of data's origin, transformations, movements, and usage across an organization's systems — has become a regulatory compliance requirement across multiple frameworks. BCBS 239 (Principles for Effective Risk Data Aggregation and Risk Reporting), issued by the Basel Committee in January 2013 and enforced through supervisory review of G-SIBs and D-SIBs, Principle 2 (Data Architecture and IT Infrastructure) requires banks to design data architecture that supports risk data aggregation with documented data flows from source systems to risk reports. GDPR Articles 13–14 (data subject information rights) and Article 30 (Records of Processing Activities, ROPA) require organizations to document the sources, purposes, and recipients of personal data — which is fundamentally a lineage requirement for personal data flows. The EU AI Act (Regulation 2024/1689) Article 10 requires that training data for high-risk AI systems be documented with data provenance information. SEC Rule 17a-4 and FINRA Rule 4370 require broker-dealers to maintain records that can demonstrate the integrity of reported data from source through to regulatory submission.

Data lineage architecture for regulatory compliance operates at two levels: technical lineage and business lineage. Technical lineage captures column-level data flows between systems — for example, a specific field in a regulatory capital report traces through a data warehouse transformation, through a risk aggregation engine, back to a specific source system field in the core banking system. Technical lineage is captured by: ETL/ELT tool-native lineage (Azure Data Factory, dbt, Informatica PowerCenter each generate lineage metadata); database-level query parsing (tools such as Alation, Collibra, and Apache Atlas parse SQL to extract column-level lineage from ad hoc queries); and Apache Kafka header propagation for streaming lineage across event-driven architectures. Business lineage maps technical data elements to business glossary terms, data owners, and regulatory purposes — providing the human-readable layer that satisfies GDPR ROPA requirements and BCBS 239 documentation obligations.

BCBS 239 Principle 6 (Adaptability) requires that banks can produce risk data quickly for supervisory requests, ad hoc scenarios, and stress tests — which requires lineage sufficient to trace which source data contributed to any specific risk aggregate and how it was transformed. The 2023 FSB report on BCBS 239 implementation noted that data lineage remains the most common area of supervisory finding, with many banks lacking automated lineage for critical risk metrics. Practical implementation challenges include: legacy systems (mainframes, packaged applications) that do not expose metadata APIs for lineage capture, requiring reverse-engineering through schema comparison; heterogeneous data stores where lineage tools may cover SQL databases but not unstructured data stores, NoSQL databases, or real-time data streams; and lineage coverage of manual data processes (spreadsheets, Access databases used for regulatory adjustments) that are outside automated tooling scope. GDPR enforcement actions have specifically cited missing ROPA entries for data received from third parties as a compliance gap, making data provenance tracking for externally sourced data a priority.

We implement data lineage programs for regulated organizations covering technical lineage capture through ETL metadata, dbt model documentation, and Apache Atlas/OpenLineage integration; business lineage cataloguing in Collibra or Atlan; ROPA generation from lineage metadata for GDPR compliance; and BCBS 239-aligned risk data lineage reporting for supervisory submission.

Compliance-Native Architecture Guide

Design principles and a structured checklist for building software that is compliant by default — not compliant by retrofit. Covers data architecture, access controls, audit trails, and vendor due diligence.