22 min
Lesson 1 of 4
ML Data Architecture
From the course AI Data Pipeline Engineering
Design robust data architectures that support the full ML lifecycle from data collection to model training.
Modern Data Architecture for ML
Lakehouse Architecture
Data Sources → Bronze (Raw) → Silver (Cleaned) → Gold (ML-Ready)
↓ ↓ ↓
Delta Lake Feature Store Training Datasets
Delta Lake Setup
from pyspark.sql import SparkSession
from delta import configure_spark_with_delta_pip
builder = SparkSession.builder \
.appName("ML Data Pipeline") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
spark = configure_spark_with_delta_pip(builder).getOrCreate()
# Write Bronze layer (raw data)
df_raw = spark.read.json("s3://data-lake/raw/events/*.json")
df_raw.write.format("delta") \
.mode("append") \
.save("s3://data-lake/bronze/events")
# Silver layer (cleaned data)
df_silver = df_raw.dropDuplicates() \
.dropna(subset=["user_id", "timestamp"]) \
.withColumn("date", to_date("timestamp"))
df_silver.write.format("delta") \
.partitionBy("date") \
.mode("overwrite") \
.save("s3://data-lake/silver/events")
# Gold layer (ML features)
df_gold = df_silver.groupBy("user_id") \
.agg(
count("event").alias("event_count"),
countDistinct("session_id").alias("session_count"),
max("timestamp").alias("last_activity")
)
df_gold.write.format("delta") \
.mode("overwrite") \
.save("s3://data-lake/gold/user_features")
Data Quality Framework
from great_expectations.core import ExpectationSuite
import great_expectations as ge
# Define expectations for ML data quality
suite = ExpectationSuite("ml_training_data")
df_ge = ge.from_pandas(training_df)
# Feature completeness
df_ge.expect_column_values_to_not_be_null("user_id")
df_ge.expect_column_values_to_not_be_null("target_variable")
# Feature distributions (detect drift)
df_ge.expect_column_mean_to_be_between("age", 20, 70)
df_ge.expect_column_values_to_be_between("purchase_amount", 0, 10000)
# Cardinality checks
df_ge.expect_column_unique_value_count_to_be_between("category", 5, 20)
results = df_ge.validate()
if not results.success:
raise DataQualityException("Training data quality checks failed")
Key Takeaways
- •Lakehouse architecture with Bronze/Silver/Gold layers organizes raw to ML-ready data
- •Delta Lake provides ACID transactions and time-travel for ML data versioning
- •Great Expectations automates data quality validation in ML pipelines
- •Feature stores bridge the gap between data engineering and model training
0 of 4 lessons marked complete
Progress is kept in this browser only.