Loading Academy Hub...
Browse blueprints in the docked sidebar across PySpark AQE/Arrow, Databricks Photon/Liquid Clustering, Snowflake Micro-Partitions & Dynamic Tables, Azure Synapse, AWS Iceberg, SARGable SQL, and Power BI VertiPaq.
How Spark 3.x re-optimizes logical and physical plans at runtime using real partition statistics gathered after shuffle stages.
Fixed 200 shuffle partitions -> 185 tiny tasks + 2GB skewed straggler (42 mins)
AQE coalesced to 24 balanced 128MB tasks + split skewed partition (4.8 mins)
By default, Spark uses a rigid 200 shuffle partitions (`spark.sql.shuffle.partitions=200`). For small datasets, 200 causes tiny task overhead; for 500GB datasets, 200 causes catastrophic 2.5GB task OOMs! AQE dynamically inspects shuffle output and merges tiny contiguous partitions into optimal 128MB buckets.
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "134217728") # 128 MB targetWhen one key has 10M rows and other keys have 1,000 rows, AQE automatically detects the skewed partition, splits it into multiple smaller sub-partitions, and duplicates the matching row on the other side of the join.
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor", "5")
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", "268435456") # 256 MBSkip AQE skew-splitting only on tiny sub-5GB jobs where metadata planning overhead exceeds shuffle cost, or ensure Spark 3.2+ is active before relying on skewed join splitting in outer joins.