22 min
Lesson 3 of 4
Data Versioning and Lineage
From the course AI Data Pipeline Engineering
Track data versions and lineage across the ML lifecycle for reproducibility and debugging.
Data Versioning
DVC (Data Version Control)
# Initialize DVC
git init
dvc init
# Track large datasets with DVC
dvc add data/training_set.parquet
git add data/training_set.parquet.dvc .gitignore
git commit -m "Track training dataset v1.0"
# Configure remote storage
dvc remote add -d myremote s3://ml-data-lake/dvc-cache
# Push dataset to remote
dvc push
# Reproduce specific dataset version
git checkout v1.0
dvc pull # Downloads exact dataset from that version
Dataset Versioning with Delta Lake
# Time travel to any historical version
df_historical = spark.read.format("delta") \
.option("timestampAsOf", "2024-01-15") \
.load("s3://data-lake/gold/user_features")
# Or by version number
df_version = spark.read.format("delta") \
.option("versionAsOf", 42) \
.load("s3://data-lake/gold/user_features")
# Audit data history
history = spark.sql("""
DESCRIBE HISTORY delta.\`s3://data-lake/gold/user_features\`
""")
Data Lineage Tracking
# OpenLineage for end-to-end lineage
from openlineage.client import OpenLineageClient
from openlineage.client.run import Run, RunEvent, RunState
client = OpenLineageClient(url="http://marquez:5000")
# Track pipeline run
run = Run(run_id=str(uuid4()), facets={})
# Record input dataset
start_event = RunEvent(
eventType=RunState.START,
eventTime=datetime.utcnow().isoformat(),
run=run,
job=Job(namespace="ml-team", name="feature-engineering"),
inputs=[Dataset(namespace="s3://data-lake", name="silver/events")],
outputs=[Dataset(namespace="s3://data-lake", name="gold/user_features")]
)
client.emit(start_event)
Monitoring Data Pipelines
- Track data freshness SLAs (how old is training data?)
- Monitor data volume anomalies (unexpected drops or spikes)
- Alerting on schema changes in source data
- Automatic retraining triggers on significant data drift
Key Takeaways
- •DVC provides Git-like versioning for large datasets linked to model training runs
- •Delta Lake time-travel enables reproducing any historical dataset version
- •OpenLineage tracks end-to-end data flow from source to model
- •Pipeline monitoring with freshness SLAs and volume anomaly detection ensures reliability
0 of 4 lessons marked complete
Progress is kept in this browser only.