Loading Academy Hub...
Understand the internal BlockManager, cut DataFrame RAM usage by 80% with downcasting and categorical types.
By default, Pandas imports text columns as object (pointers to PyObject strings) and integers as 64-bit int64 / floats as float64.
A Pandas DataFrame is organized into 2D NumPy arrays grouped by data type (FloatBlock, IntBlock, ObjectBlock).
object column storing country names like "USA" creates a separate 50-byte Python string for every single row!category encodes unique string values in a single internal integer lookup dictionary, cutting memory by 85%+!int64 consumes 8 bytes per number (supports up to 9 quintillion). If your age column ranges from 0 to 100, downcasting to uint8 (1 byte) saves 87.5% memory!object string columns store a 64-bit pointer array PLUS full Python str heap objects (49+ bytes overhead per string even for "OK"!). A 500MB CSV routinely bloats to 4GB+ in Pandas RAM!int64 -> int32 / int16 and float64 -> float32 cuts numeric RAM by 50%–75%.< 50% distinct ratio like country, currency, status) drop by up to 95% RAM by storing compact integer codes.dtype_backend="pyarrow"): Stores contiguous off-NumPy Apache Arrow string buffers cleanly with native nullable types (int64[pyarrow], no silent int-to-float casting on NaN!).Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
import pandas as pd
import numpy as np
# Create sample DataFrame
n = 500000
df = pd.DataFrame({
'status': np.random.choice(['PENDING', 'COMPLETED', 'FAILED'], n),
'age': np.random.randint(18, 90, n),
'score': np.random.rand(n)
})
initial_mem = df.memory_usage(deep=True).sum() / 1024**2
print(f"Initial Memory: {initial_mem:.2f} MB")
# Optimize: Convert status to category and downcast numbers
df['status'] = df['status'].astype('category')
df['age'] = pd.to_numeric(df['age'], downcast='unsigned') # uint8 (1 byte!)
df['score'] = pd.to_numeric(df['score'], downcast='float') # float32 (4 bytes!)
optimized_mem = df.memory_usage(deep=True).sum() / 1024**2
print(f"Optimized Memory: {optimized_mem:.2f} MB")
print(f"Memory Reduced by: {((initial_mem - optimized_mem) / initial_mem) * 100:.1f}%!")
SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame