AI System Design, Enterprise RAG & 5-Lever FinOps Scaling (30 Q&As + Code)

(1) Semantic + Exact Caching (tenant/RBAC partitioned) returns cached answers for 30% repeat traffic at ~$0; (2) Complexity Routing sends 70%+ simple queries to mini/flash tiers (10x-15x cheaper); (3) Prompt Prefix Caching + Compression freezes static instructions at token 0 for 50–90% prefill discounts; (4) Async Batch APIs save a flat 50% on offline ETL/evals; (5) OTel FinOps Telemetry + Budgets stops looping runaway workers.

def route_tier(query: str, complexity_score: float):
    if complexity_score < 0.35:
        return "gpt-4o-mini-tier" # Fast <120ms & 15x cheaper
    if is_offline_job(query):
        return "batch-queue-50pct-off"
    return "frontier-reasoning-tier"

If Customer A asks "What is our Q4 merger valuation?" and Customer B asks a similar phrase later, a shared global vector cache serves Customer A’s confidential private memo directly to Customer B! Every semantic cache key MUST include tenant_id AND sorted caller_acl_groups hash.

cache_key_ns = f"rag_cache:{tenant_id}:{hash_sorted_groups(user_roles)}"
# Only search cached vectors strictly inside user's RBAC partition!