AI Model Wiki

    Skip to main content
    22 min
    Lesson 1 of 4

    ML Data Architecture

    From the course AI Data Pipeline Engineering

    Design robust data architectures that support the full ML lifecycle from data collection to model training.

    Modern Data Architecture for ML

    Lakehouse Architecture

    Data Sources → Bronze (Raw) → Silver (Cleaned) → Gold (ML-Ready)
                  ↓               ↓                  ↓
             Delta Lake      Feature Store      Training Datasets
    

    Delta Lake Setup

    from pyspark.sql import SparkSession
    from delta import configure_spark_with_delta_pip
    
    builder = SparkSession.builder \
        .appName("ML Data Pipeline") \
        .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    
    spark = configure_spark_with_delta_pip(builder).getOrCreate()
    
    # Write Bronze layer (raw data)
    df_raw = spark.read.json("s3://data-lake/raw/events/*.json")
    df_raw.write.format("delta") \
        .mode("append") \
        .save("s3://data-lake/bronze/events")
    
    # Silver layer (cleaned data)
    df_silver = df_raw.dropDuplicates() \
        .dropna(subset=["user_id", "timestamp"]) \
        .withColumn("date", to_date("timestamp"))
    
    df_silver.write.format("delta") \
        .partitionBy("date") \
        .mode("overwrite") \
        .save("s3://data-lake/silver/events")
    
    # Gold layer (ML features)
    df_gold = df_silver.groupBy("user_id") \
        .agg(
            count("event").alias("event_count"),
            countDistinct("session_id").alias("session_count"),
            max("timestamp").alias("last_activity")
        )
    
    df_gold.write.format("delta") \
        .mode("overwrite") \
        .save("s3://data-lake/gold/user_features")
    

    Data Quality Framework

    from great_expectations.core import ExpectationSuite
    import great_expectations as ge
    
    # Define expectations for ML data quality
    suite = ExpectationSuite("ml_training_data")
    
    df_ge = ge.from_pandas(training_df)
    
    # Feature completeness
    df_ge.expect_column_values_to_not_be_null("user_id")
    df_ge.expect_column_values_to_not_be_null("target_variable")
    
    # Feature distributions (detect drift)
    df_ge.expect_column_mean_to_be_between("age", 20, 70)
    df_ge.expect_column_values_to_be_between("purchase_amount", 0, 10000)
    
    # Cardinality checks
    df_ge.expect_column_unique_value_count_to_be_between("category", 5, 20)
    
    results = df_ge.validate()
    if not results.success:
        raise DataQualityException("Training data quality checks failed")
    

    Key Takeaways

    • •Lakehouse architecture with Bronze/Silver/Gold layers organizes raw to ML-ready data
    • •Delta Lake provides ACID transactions and time-travel for ML data versioning
    • •Great Expectations automates data quality validation in ML pipelines
    • •Feature stores bridge the gap between data engineering and model training
    0 of 4 lessons marked complete

    Progress is kept in this browser only.