ADLS Gen2 Storage Hierarchy, RBAC, Managed Identities & SAS Tokens

[!NOTE] Azure Data Lake Storage Gen2 (ADLS Gen2) converges the massive scalability of Azure Blob Storage with an enterprise-grade Hierarchical Namespace (HNS). In flat object stores (like Amazon S3 or traditional Blob storage), folders are purely conceptual prefixes; deleting or renaming a directory with 1,000,000 files requires 1,000,000 independent copy and delete API calls. In ADLS Gen2, directories are true first-class filesystem objects, enabling atomic $O(1)$ renames and directory-level POSIX ACL inheritance.

1. Storage Architecture: Flat Blob vs Hierarchical Namespace (HNS)

2. Authentication Matrix: Zero-Trust Security in Enterprise Pipelines

Never use Storage Account Shared Access Keys in production data engineering pipelines. Shared Access Keys grant unrestricted superuser access to the entire storage account and cannot be scoped by folder or audited to individual services.

The Three Tiers of Identity & Authorization

1. Azure RBAC (Azure Resource Manager Role-Based Access Control):
   • Assign roles at Subscription, Resource Group, Storage Account, or Container level.
   • Essential roles: Storage Blob Data Contributor (read/write/delete data), Storage Blob Data Reader (read-only), Storage Blob Data Owner (manage POSIX ACLs).
   • Senior Pitfall: The general Azure "Contributor" or "Owner" role only grants management plane permissions (creating storage accounts, updating firewall); it does NOT grant data plane access to read/write blobs!
2. POSIX Access Control Lists (ACLs):
   • Granular directory- and file-level permissions (rwx: Read, Write, Execute).
   • Two categories: Access ACLs (control access to current object) and Default ACLs (inherited by newly created child items within a directory).
3. Shared Access Signatures (SAS):
   • User Delegation SAS: Secured by Microsoft Entra ID credentials rather than storage account keys. Recommended for temporary, time-bounded client access (e.g. generating a 15-minute presigned download URL).
   • Service SAS / Account SAS: Signed directly with account keys (avoid in production due to lack of Entra ID auditing).

3. Network Security: Private Endpoints & Service Endpoints

In enterprise banking and healthcare environments, storage accounts must block all public internet access:

• Private Endpoints: Injects a private IP from your Azure Virtual Network (VNET) into the storage account using Azure Private Link. All ADF, Databricks, and Synapse compute communicate over internal Azure backbone fiber without public egress.
• Trusted Microsoft Services Exception: Grants authenticated ADF and Synapse instances bypass through the storage firewall using their Managed Identity.

Staff Architecture: ADLS Gen2 Hierarchical Namespace, ABFSS Auth & Event-Driven Ingestion

• Why Hierarchical Namespace (HNS) Is Mandatory for Lakehouses: Enabling HNS upgrades Blob Storage into true POSIX-like directories on ADLS Gen2. Rename/delete of a 10,000-file staging directory executes as an O(1) atomic metadata operation rather than 10,000 slow HTTP blob copies, making Delta Lake / Synapse / Databricks commits fast and atomic.
• ABFSS (abfss://) + Managed Identity: Always use abfss:// (TLS-encrypted ABFS driver) over deprecated wasbs://, and bind Azure Managed Identity (or Databricks Unity Catalog Access Connectors) instead of Account Keys so storage credentials never appear in notebooks or Spark confs.
• Private Endpoints & Firewall Traps: In locked-down Azure VNets, enabling Private Endpoints on ADLS Gen2 without allowing Trusted Azure Services or VNet subnet rules breaks Azure Data Factory / Synapse Managed VNet Integration Runtimes with 403 AuthorizationFailure.