Data Engineering Production Error & Troubleshooting Encyclopedia

Select any incident or category in the left panel to inspect full stack traces, telemetry log locations, verified production fixes, alerting SLOs, and interview takeaways.

Apache Spark / Databricks / EMRCritical Severity

Target MTTR: 15–30 min

Executor OOM: Container killed by YARN / K8s for exceeding memory limits (Exit Code 137)

Production Stack Trace Signature

ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 16.4 GB of 16.0 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. Exit code 137.

Root Cause Breakdown

Off-heap memory exhaustion or heavy Python worker memory usage. PySpark passes data via Py4J/Arrow IPC pipes to Python daemon processes. High cardinality UDFs, wide row serialization, or excessive off-heap buffers exceed `spark.executor.memoryOverhead` (defaults to only 10% of executor memory).

Where to Inspect Telemetry

Spark UI -> Executors Tab -> Failed Tasks / YARN Resource Manager NodeManager logs / stderr

Step-by-Step Verified Production Fix

1. Increase Memory Overhead in spark-defaults.conf or spark-submit:
   --conf spark.executor.memoryOverhead=4096m (or 25% of executor memory for heavy Python ETL)
2. If using Python UDFs, replace with Native PySpark SQL functions or Vectorized Pandas UDFs (@pandas_udf).
3. Check for high-cardinality groupBy or skewed shuffle partitions; increase `spark.sql.shuffle.partitions` from default 200 to 1200+.
4. Ensure JVM Garbage Collection is tuned: `spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35`.

Proactive Alerting & Prevention SLO

Datadog / Prometheus Alert: `spark.executor.jvm.heap.usage > 85%` for > 3 minutes; CloudWatch `ContainerPendingRatio > 0.2`.

Loading Academy Hub...

ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 16.4 GB of 16.0 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. Exit code 137.

1. Increase Memory Overhead in spark-defaults.conf or spark-submit: --conf spark.executor.memoryOverhead=4096m (or 25% of executor memory for heavy Python ETL) 2. If using Python UDFs, replace with Native PySpark SQL functions or Vectorized Pandas UDFs (@pandas_udf). 3. Check for high-cardinality groupBy or skewed shuffle partitions; increase `spark.sql.shuffle.partitions` from default 200 to 1200+. 4. Ensure JVM Garbage Collection is tuned: `spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35`.