Loading Academy Hub...
Master the industry-standard Databricks Lakehouse Medallion pattern: Bronze raw ingestion with append-only semantics, Silver conformed business cleansing with Delta MERGE upserts, and Gold aggregated dimensional feature stores.
[!NOTE] The Medallion Architecture describes a series of data quality refinement layers (Bronze -> Silver -> Gold) as data moves through the Lakehouse. Each layer provides increasing structure, schema enforcement, and ACID transactional guarantees.
[ Sources: Kafka, IoT, RDBMS, CDC ]
│
▼
🥉 BRONZE (Raw Ingest)
• Append-only, raw unmodified data
• Preserves history, schema evolution (`mergeSchema`)
• Fast ingestion without data transformation
│
▼ (Cleansing, Dedup, Enforce Types, MERGE)
🥈 SILVER (Cleaned & Conformed)
• Single Source of Truth enterprise domain models
• Deduplicated, validated business keys
• ACID upserts (MERGE INTO) & schema constraints
│
▼ (Aggregations, Star Schema, Feature Store)
🥇 GOLD (Curated Business & AI Marts)
• Dimensional models (Facts & Dimensions)
• Optimized with Liquid Clustering / Z-Order for sub-second BI
• Ready for Power BI, Tableau, and LLM Vector Search
| Architectural Dimension | Traditional Parquet Data Lake | Databricks Medallion Lakehouse |
|---|---|---|
| ACID Transactions | None (risk of partial read during write) | Full ACID via Delta Transaction Log |
| Upserts (MERGE) | Expensive rewrite of entire partition | Native MERGE INTO with row-level modifications |
| Schema Enforcement | Silent column corruption or nulls | Strict schema validation with explicit mergeSchema |
| Time Travel | None (requires manual date snapshots) | Native versioned time travel up to retention window |
2026–2027): Control vs Data Plane, Unity Catalog & Serverless SQL/Jobs4–7 minute VM cloud-provider startup cold starts and pay strictly per active execution second.autotermination_minutes (<= 15 min), forbidding oversized All-Purpose interactive clusters for production cron schedules, and forcing Unity Catalog (Shared or Single User Access Modes) over deprecated No Isolation Shared modes.Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
from delta.tables import DeltaTable
from pyspark.sql.functions import col, current_timestamp
# 1. Read newly arrived Bronze streaming/batch records
bronze_df = spark.read.table("lakehouse_bronze.raw_transactions") \
.filter(col("ingested_at") >= "2026-03-01")
# 2. Deduplicate on primary key keeping latest record
cleaned_df = bronze_df.dropDuplicates(["transaction_id"])
# 3. Perform ACID Upsert into Silver conformed table
silver_table = DeltaTable.forName(spark, "lakehouse_silver.transactions")
silver_table.alias("t").merge(
cleaned_df.alias("s"),
"t.transaction_id = s.transaction_id"
).whenMatchedUpdate(set={
"amount": col("s.amount"),
"status": col("s.status"),
"updated_at": current_timestamp()
}).whenNotMatchedInsert(values={
"transaction_id": col("s.transaction_id"),
"customer_id": col("s.customer_id"),
"amount": col("s.amount"),
"status": col("s.status"),
"created_at": col("s.created_at"),
"updated_at": current_timestamp()
}).execute()
print("Silver Delta MERGE completed successfully.")org.apache.spark.sql.AnalysisException: A schema mismatch detected when writing to the Delta table. Source column 'merchant_category' not found in target table schema.