Google BigQuery Architecture: Capacitor Storage, Slots & Partitioning vs Clustering

[!NOTE] Google BigQuery decouples compute from storage via Google's Jupiter petabit network. The compute tier consists of dynamic worker units called Slots orchestrated by Borg, while the storage tier uses Capacitor, Google's proprietary columnar format.

1. Partitioning vs Clustering in BigQuery

Staff Architecture: BigQuery Capacitor Storage, Slot Pricing & Partitioning + Clustering

• Separation of Compute (Dremel) & Storage (Colossus / Capacitor): BigQuery physically stores columns independently in compressed Capacitor format on Colossus, shuffling intermediate execution stages over Google’s Jupiter petabit network. Because shuffling occurs across decoupled worker slots, SELECT * pays a massive byte-scan & network penalty—always project explicit columns (SELECT user_id, amount).
• Partitioning vs Clustering (4,000 Partition Limit): Partition exclusively on coarse low-cardinality fields (usually DATE(event_ts)—BigQuery enforces a 4,000-partition hard limit per table), then apply Clustering (CLUSTER BY tenant_id, event_type) up to 4 columns to sort storage blocks within each day for zero extra storage fee.
• On-Demand vs Editions (Autoscale Slots): On-Demand bills $6.25/TB scanned (logical or physical bytes—always audit Active vs Time-Travel compression ratios before switching a dataset to Physical Storage Billing!). High-concurrency teams move to BigQuery Editions (Standard/Enterprise) with baseline + autoscaling Slots and require_partition_filter = true guardrails.