AI Model Wiki

    Skip to main content
    22 min
    Lesson 3 of 4

    Data Versioning and Lineage

    From the course AI Data Pipeline Engineering

    Track data versions and lineage across the ML lifecycle for reproducibility and debugging.

    Data Versioning

    DVC (Data Version Control)

    # Initialize DVC
    git init
    dvc init
    
    # Track large datasets with DVC
    dvc add data/training_set.parquet
    git add data/training_set.parquet.dvc .gitignore
    git commit -m "Track training dataset v1.0"
    
    # Configure remote storage
    dvc remote add -d myremote s3://ml-data-lake/dvc-cache
    
    # Push dataset to remote
    dvc push
    
    # Reproduce specific dataset version
    git checkout v1.0
    dvc pull  # Downloads exact dataset from that version
    

    Dataset Versioning with Delta Lake

    # Time travel to any historical version
    df_historical = spark.read.format("delta") \
        .option("timestampAsOf", "2024-01-15") \
        .load("s3://data-lake/gold/user_features")
    
    # Or by version number
    df_version = spark.read.format("delta") \
        .option("versionAsOf", 42) \
        .load("s3://data-lake/gold/user_features")
    
    # Audit data history
    history = spark.sql("""
        DESCRIBE HISTORY delta.\`s3://data-lake/gold/user_features\`
    """)
    

    Data Lineage Tracking

    # OpenLineage for end-to-end lineage
    from openlineage.client import OpenLineageClient
    from openlineage.client.run import Run, RunEvent, RunState
    
    client = OpenLineageClient(url="http://marquez:5000")
    
    # Track pipeline run
    run = Run(run_id=str(uuid4()), facets={})
    
    # Record input dataset
    start_event = RunEvent(
        eventType=RunState.START,
        eventTime=datetime.utcnow().isoformat(),
        run=run,
        job=Job(namespace="ml-team", name="feature-engineering"),
        inputs=[Dataset(namespace="s3://data-lake", name="silver/events")],
        outputs=[Dataset(namespace="s3://data-lake", name="gold/user_features")]
    )
    
    client.emit(start_event)
    

    Monitoring Data Pipelines

    • Track data freshness SLAs (how old is training data?)
    • Monitor data volume anomalies (unexpected drops or spikes)
    • Alerting on schema changes in source data
    • Automatic retraining triggers on significant data drift

    Key Takeaways

    • •DVC provides Git-like versioning for large datasets linked to model training runs
    • •Delta Lake time-travel enables reproducing any historical dataset version
    • •OpenLineage tracks end-to-end data flow from source to model
    • •Pipeline monitoring with freshness SLAs and volume anomaly detection ensures reliability
    0 of 4 lessons marked complete

    Progress is kept in this browser only.