E-Commerce & Payments
E-Commerce Flash Sale & Payment Gateway Reconciliation1. AWS S3 & EMR/Glue Source Tables (3):
payment_events.csv
CSV
Gateway Payment Events (100K+ batch feed)
100,000 rows✓ Active
orders_raw.json
JSON
Flash Sale Checkout Orders (JSON)
85,000 rowsInspect
merchant_dim.csv
CSV
Merchant Category Lookup (Broadcast Dimension)
450 rowsInspect
2. Inspect AWS S3 & EMR/Glue Architecture Components:
1. S3 Bronze Landing & Lambda Guardrail
Kinesis Firehose + AWS Lambda + S3 Bronze
2. Silver Reconciled Orders (Glue / EMR Delta)
Amazon EMR Spark 3.5 + Delta Lake on S3
3. Gold Gateway Settlement & Redshift Mart
S3 Gold Delta + Redshift Spectrum + QuickSight
4. IAM Roles, STS AssumeRole & Lake Formation
AWS IAM + STS + Secrets Manager + Lake Formation
5. S3 Prefix Salting, AQE & Skew Handling
Spark AQE + S3 Coalesce + Broadcast Joins
6. MWAA (Airflow) & Step Functions Orchestration
Amazon MWAA + Glue/EMR Operators + CloudWatch
2. Silver Reconciled Orders (Glue / EMR Delta)
Amazon EMR Spark 3.5 + Delta Lake on S3 • s3://enterprise-prod-lakehouse/
ACID MERGE + Dedup
Deduplicates Kinesis/gateway retries on txn_id, filters failed settlements, and joins checkout orders & broadcast merchant fees.
Compute Engine
EMR Serverless / Glue 4.0
Deduplication
Window.partitionBy("txn_id")
Join Strategy
F.broadcast(merchant_dim)
Commit Protocol
Delta ACID MERGE INTO
Cloud-Specific PySpark Snippet:Runs Live in Pipeline Runner
# Silver Deduplication & Broadcast Enrichment
w = Window.partitionBy("txn_id").orderBy(F.col("event_ts").desc())
silver_orders = (
bronze_payments.filter(F.col("status") == "SUCCESS")
.withColumn("_rn", F.row_number().over(w))
.filter(F.col("_rn") == 1).drop("_rn")
.join(F.broadcast(merchant_dim), on="merchant_id", how="left")
)Active Cloud: Amazon Web Services (AWS)
Source Data Preview: Gateway Payment Events (100K+ batch feed)7 columns
| txn_id (string) | order_id (integer) | merchant_id (string) | gateway (string) | amount (double) | status (string) | event_ts (string) |
|---|---|---|---|---|---|---|
| TXN_901 | 5001 | M_ELECTRONICS | HDFC_PG | 14500 | SUCCESS | 2026-11-10 10:01:12 |
| TXN_902 | 5002 | M_APPAREL | ICICI_PG | 3200 | SUCCESS | 2026-11-10 10:02:05 |
| TXN_903 | 5003 | M_ELECTRONICS | SBI_PG | 28900 | SUCCESS | 2026-11-10 10:02:44 |
| TXN_904 | 5004 | M_GROCERY | AXIS_PG | 850 | FAILED | 2026-11-10 10:03:11 |
| TXN_901 | 5001 | M_ELECTRONICS | HDFC_PG | 14500 | SUCCESS | 2026-11-10 10:03:49 |
| TXN_905 | 5005 | M_APPAREL | HDFC_PG | 5400 | SUCCESS | 2026-11-10 10:04:20 |