AI features have two budgets that always run out — money and milliseconds — and a third limit that appears at scale: provider quotas and architecture ceilings. This tutorial covers making AI applications fast and cheap, then taking them from tens of users to millions.

Part 1: Cost & Performance Optimization

Where the Money Actually Goes

For a typical LLM product, the bill breaks down roughly like this:

Cost driver	Typical share	Lever
LLM inference (input tokens)	40–60%	Prompt trimming, caching, routing
LLM inference (output tokens)	20–30%	Concise outputs, max_tokens caps
Embeddings + vector DB	5–15%	Incremental indexing, cheap models
Infrastructure	5–15%	Autoscaling, right-sizing

Input tokens usually dominate — a bloated 8k-token system prompt multiplied by a million requests costs real money. Audit before optimizing: measure cost per request by component first.

Prompt Optimization

Every token in every request is billed. Trim systematically:

Delete instructions the model never follows anyway (they're dead weight).
Move few-shot examples out of the prompt when they're stable — use fine-tuning or a cheaper model with a shorter prompt.
Put dynamic content (user data) in, keep static boilerplate minimal.

# Before: ~1,200 tokens of instructions on every call
SYSTEM_V1 = """You are a helpful assistant... [800 words of rules] ..."""

# After: compressed instructions, rules only for the relevant path
SYSTEM_V2 = """Classify the ticket into: billing | technical | account | other.
If billing and amount mentioned, extract as JSON {"amount": number}.
Be terse."""

def build_prompt(ticket: str) -> list:
    return [{"role": "system", "content": SYSTEM_V2},
            {"role": "user", "content": ticket[:2000]}]   # truncate inputs

The explanation: the rewrite cut the system prompt ~10×, and truncating oversized tickets caps worst-case input cost. Re-run your eval set after every prompt change — a cheaper prompt that fails tests is not cheaper.

Model Right-Sizing and Routing

The single biggest cost lever: don't pay for frontier intelligence on tasks that don't need it.

def route(request) -> str:
    if request.is_simple_classification or request.expected_output_tokens < 50:
        return "gpt-4o-mini"          # ~17x cheaper
    if request.requires_reasoning:
        return "gpt-4o"
    return "gpt-4o-mini"              # default cheap

Smarter version — a "cascade": try the cheap model first, escalate only when it fails.

def cascade(request):
    answer = cheap_model(request)
    if confidence(answer) > 0.85:     # via logprobs or a judge
        return answer
    return expensive_model(request)

The explanation: most requests (often 70–90%) resolve at the cheap tier. Track the escalation rate — it tells you exactly what the routing is saving.

Caching Strategies

Three levels, cheapest wins first:

Exact-match caching — identical prompts return the identical cached answer. Hash the full request (model + messages + params).
Semantic caching — embed the query; if a cached query is above a similarity threshold, return its answer. Riskier: "How do I reset my password?" and "I can't log in" may not want the same answer. Set thresholds conservatively.
Prompt caching (provider-side) — many providers bill cached input tokens at a fraction of the price. Structure prompts so static content comes first and dynamic content last — that's what makes it cacheable.

import hashlib, json

def cache_key(model, messages, **params) -> str:
    return hashlib.sha256(
        json.dumps([model, messages, params], sort_keys=True).encode()
    ).hexdigest()

def cached_llm_call(model, messages, **params):
    key = cache_key(model, messages, **params)
    if hit := cache.get(key):
        return hit
    resp = llm.chat(messages, model=model, **params)
    cache.set(key, resp, ttl=3600)
    return resp

The explanation: for FAQ-style traffic, exact-match caches routinely absorb 20–40% of requests at near-zero cost. Always include model and params in the key — a cached answer from a different model is a subtle bug.

Batching and Embedding Costs

Embed in batches (providers charge per token, but batching cuts request overhead and latency).
Re-embed only changed documents — hash document content and skip unchanged ones.
Use the cheapest embedding model that passes your retrieval evals; embedding bills run over the whole corpus.

def embed_incremental(docs, store):
    to_embed = [d for d in docs if store.hash_of(d.id) != d.content_hash]
    for batch in chunk_list(to_embed, 100):
        store.upsert(embed([d.content for d in batch]))

Reducing Retrieval Costs

RAG cost scales with retrieved context. Control it:

Rerank aggressively: retrieve 50 cheaply, feed the LLM only the top 3–5 chunks.
Compress context: strip boilerplate, footers, and repeated headers from chunks before inserting them into the prompt.
Cap chunk length in the prompt, not just at index time.

Context budget example (8k window):
  instructions      400 tokens
  retrieved chunks  2,500 tokens  (5 chunks × 500)
  history           1,000 tokens
  question + answer  2,000 tokens
  ── headroom       2,100 tokens

Latency: Where Time Goes

A RAG request's wall-clock time:

Embed query          ~50ms
Vector search        ~30ms
Rerank               ~150ms
LLM generation   2,000–10,000ms   ← dominates everything

The LLM call is almost always the bottleneck, so latency work focuses on it: fewer input tokens, fewer output tokens, faster models, and streaming so users don't wait for the full answer.

Also measure time to first token (TTFT) separately from total time — users perceive responsiveness through TTFT, not completion.

Streaming Responses

Streaming doesn't reduce total time, but it cuts perceived latency dramatically — the user starts reading while generation continues.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import httpx

app = FastAPI()

@app.post("/chat")
async def chat(body: dict):
    async def generate():
        async with httpx.AsyncClient(timeout=None) as client:
            async with client.stream("POST", LLM_URL, json=payload(body)) as r:
                async for chunk in r.aiter_text():
                    yield chunk
    return StreamingResponse(generate(), media_type="text/event-stream")

The explanation: chunks are forwarded to the browser as they arrive. For a 3,000ms generation, perceived latency drops from 3,000ms to roughly the TTFT (~300ms).

Parallelism and Prefetching

Run independent steps at the same time instead of in sequence:

import asyncio

async def answer(question: str, user_id: str):
    # retrieval, permission check, and profile fetch in parallel
    chunks, allowed, profile = await asyncio.gather(
        retrieve_async(question),
        check_permission_async(user_id),
        fetch_profile_async(user_id),
    )
    return generate(question, chunks, profile)

And prefetch: start retrieving (or even embedding) while the user is still typing, using debounce. Every millisecond of work done before the request lands is a millisecond saved.

Reducing Output Tokens and Rounds

Output tokens usually cost 3–4× more than input tokens and take longer to generate.

Set realistic max_tokens caps.
Ask for structured, terse output ("answer in under 100 words" actually works).
For agents: fewer steps = fewer full-prompt re-sends. Compact history between steps.

# Agentic loop: trim context as the loop progresses
def run_agent(goal):
    history = [system_msg, user_msg(goal)]
    for step in range(MAX_STEPS):
        resp = llm.chat(compact(history), tools=TOOLS)  # summarizes old turns
        ...

The explanation: in a 10-step agent run, the prompt is re-sent every step. Without compaction, you pay for step 1's full context ten times — this is where agent costs explode.

Smaller Models, Distillation, and Quantization

When volume is high and the task is narrow:

Fine-tune a small model on your task's data — often matches a frontier model on narrow tasks at 5–20× lower cost.
Distillation: use the big model to generate training data, train the small model on it.
Quantization / smaller checkpoints: for self-hosted models, run quantized weights to cut GPU memory and cost per token.

Migration path:  frontier model prototype
              → measure where it's overkill
              → fine-tune small model on logged (input, output) pairs
              → eval small vs frontier on the golden set
              → route traffic to small model, keep frontier as fallback

Measuring Optimizations

Never ship an optimization without before/after numbers:

def optimization_report(before: dict, after: dict) -> str:
    cost_delta = (after["cost"] - before["cost"]) / before["cost"]
    qual_delta = after["eval_score"] - before["eval_score"]
    return (f"cost: {cost_delta:+.0%}  eval: {qual_delta:+.2f}  "
            f"p95: {before['p95_ms']}→{after['p95_ms']}ms")

The rule: cost reductions are only valid if eval scores hold within tolerance (see the AI Evaluation Frameworks tutorial). A 60% cost cut that drops accuracy 15 points is not an optimization.

Best Practices

Measure cost per request by component before optimizing anything.
Route by difficulty: cheap model by default, escalate on low confidence.
Cache aggressively at the exact-match level; be careful with semantic caches.
Put static prompt content first so provider-side prompt caching kicks in.
Stream always; users judge speed by first token.
Re-run the eval set after every cost change — the tolerance check is your safety net.

Part 2: Scaling AI Applications

What "Millions of Users" Actually Means

"Millions of users" translates into engineering numbers:

1,000,000 users × 10 AI interactions/day = 10M model calls/day ≈ 115 calls/second average
Peak traffic is 2–5× average: plan for 300–600 calls/second at peak
Each AI call is seconds long and token-hungry — your "requests" are heavier than typical web traffic

Two implications: average numbers lie (always plan for peak), and AI endpoints need 10–100× more downstream capacity (provider quota, tokens, GPU time) than a CRUD endpoint.

Capacity Math for AI Workloads

Do the math before the architecture:

Peak load estimate:
  1M DAU × 10 calls/day            = 10M calls/day
  ÷ 86,400 seconds                 = 116 calls/sec average
  × 3 peak factor                  = 350 calls/sec peak

Provider constraint:
  gpt-4o-mini tier: 30,000 TPM, 500 RPM
  350 calls/sec × ~2,000 tokens     = 42,000,000 TPM needed
  → 30,000 TPM quota is 1,400x too small
  → need: higher tier, multiple orgs, caching, smaller models, or self-hosting

The explanation: provider token-per-minute limits — not your servers — are usually the first wall you hit. Run this calculation early; it dictates your whole architecture.

Stateless Services and Horizontal Scaling

Your API layer should hold no state in memory: no sessions, no caches of truth, no in-flight job tracking. Anything a request needs comes from a database, cache, or queue.

                    ┌→ API pod 1 ─┐
Load balancer ──────┼→ API pod 2 ─┼──→ Redis / Postgres / Queue
                    └→ API pod N ─┘   (all shared, external)

The explanation: with stateless pods, scaling is just adding replicas — Kubernetes or a serverless platform does it automatically. Long AI calls (30s+) favor async patterns over holding HTTP connections open, which leads to queues.

Queues and Asynchronous Workflows

For anything slower than ~2 seconds, don't make the user wait on an open connection. Enqueue, acknowledge, deliver later.

# API: return immediately with a job id
@app.post("/reports")
async def create_report(req: ReportRequest):
    job_id = await queue.enqueue("generate_report", req.dict())
    return {"job_id": job_id, "status": "queued"}, 202

# Worker fleet: scale independently of the API
@app.get("/reports/{job_id}")
async def status(job_id: str):
    return await jobs.get(job_id)        # queued | running | done | failed

The explanation: the API pods stay snappy regardless of model latency; workers scale to match queue depth. The client polls or receives a webhook when done. This pattern absorbs traffic spikes the way a dam absorbs floods.

Rate Limiting and Load Shedding

Protect the system (and your bill) with layered limits:

# Token bucket per user (Redis-backed)
def allow_request(user_id: str) -> bool:
    key = f"rl:{user_id}:{minute_bucket()}"
    count = redis.incr(key)
    redis.expire(key, 60)
    return count <= USER_LIMIT  # e.g. 20 AI calls/min

Per-user limits — fairness; one user can't starve everyone.
Per-tenant limits — enterprise customers get guaranteed capacity.
Global concurrency caps — never exceed what providers/GPUs can serve.
Load shedding — under extreme load, return 503 with Retry-After for low-priority traffic rather than letting everything time out.

Priority classes:  paying-user chat (protect)  >  batch jobs (shed first)

Multi-Region and Data Residency

At millions of users, geography becomes a feature requirement:

Latency: cross-continent round trips add 100–300ms before the model even starts.
Residency law: EU user data may need to stay in the EU (GDPR); some countries mandate local processing.

Architecture:  regional edge (API + cache)  →  regional AI gateway
               →  provider endpoint in-region where required
               →  central control plane (config, evals, billing)

The explanation: run stateless frontends in every region, keep heavy state and control logic centralized, and route model calls through a regional gateway so residency rules are enforced in one place.

Multi-Tenancy at Scale

Serving many enterprise customers on shared infrastructure:

Noisy neighbors: one customer's bulk job must not degrade another's chat latency. Use per-tenant concurrency limits and separate worker pools for batch work.
Isolation tiers: large tenants on dedicated capacity (dedicated provider orgs, dedicated vector namespaces); everyone else shared.
Per-tenant config: model choice, prompts, and guardrails as tenant-scoped configuration, not code branches.

config = tenant_config(tenant_id)
model = config.model or DEFAULT_MODEL      # enterprise plan → frontier model
limit = config.rate_limit                  # enterprise plan → higher quota

Managing Provider Rate Limits

Provider quotas are a hard ceiling. Tactics, in order:

Upgrade tiers — easiest, do it first.
Token budgeting — trim prompts so more requests fit the same TPM.
Client-side throttling — a global semaphore so you never send more than the quota allows (better than eating 429s).
Multi-provider routing — spread load across providers with an abstraction layer.
Self-hosted overflow — route overflow traffic to your own GPUs for well-understood tasks.

# Global throttler: stays under the provider quota
class ProviderThrottle:
    def __init__(self, rpm: int, tpm: int):
        self.rpm, self.tpm = rpm, tpm

    async def acquire(self, est_tokens: int):
        await wait_for_slot(rpm=self.rpm, tpm=est_tokens)  # token bucket

Autoscaling AI Workloads

API layer: scale on request rate or CPU; serverless also works well for spiky traffic.
Worker layer: scale on queue depth ("more than 100 pending jobs per worker → add workers").
Self-hosted models: scale on GPU utilization and queue time; keep a warm pool because cold-starting a model takes minutes.

# KEDA-style autoscaling on queue depth
triggers:
  - type: redis
    metadata:
      listName: jobs:generate_report
      listLength: "100"        # one worker per 100 pending jobs

Graceful Degradation

At scale, something is always degraded. Design the degradation ladder ahead of time:

Normal:        full RAG + frontier model + reranker
Degraded 1:    smaller model, fewer retrieved chunks
Degraded 2:    cached/template answers + BM25 keyword results
Degraded 3:    honest queue message ("high demand — we'll email you")
Never:         a raw 500 page or a hallucinated answer from a broken pipeline

def chat_with_fallback(query: str):
    try:
        return full_pipeline(query, timeout_s=10)
    except ProviderOverloaded:
        try:
            return cheap_model(query)
        except Exception:
            return template_answer(query)   # curated FAQ match

The explanation: every fallback keeps the product answering — just less richly. Users forgive "simpler answers during peak hours"; they don't forgive downtime.

Scaling the Data Layer

Read replicas for the vector DB and Postgres; searches are read-heavy.
Partition/shard vector indexes by tenant or time; a single 500M-vector index is an operational nightmare.
Cache embeddings of repeated queries and cache final answers for common questions.
Archive aggressively: old conversations move to cold storage; hot indexes stay small and fast.

A Scaling Roadmap

Match investment to actual load:

Stage	Users	Priority
Prototype	< 100	Move fast; single region; skip everything
Early product	< 10k	Queues for slow tasks, basic rate limits, cost tracking
Growth	< 100k	Autoscaling, caching, provider quota management, evals in CI
Scale	< 1M	Multi-provider routing, graceful degradation, per-tenant isolation
Millions	1M+	Multi-region, self-hosted overflow capacity, dedicated tenant tiers

Don't build the million-user architecture at 10k users — but do keep the seams (stateless services, queues, provider abstraction) so each step is an addition, not a rewrite.

Scaling Best Practices

Do the peak-load math on paper before buying anything.
Provider rate limits will hit before your servers do — engineer for them explicitly.
Queue anything slower than 2 seconds; return a job id, not a hanging connection.
Build the degradation ladder early; it converts outages into "slower answers."
Measure per-tenant cost and latency — noisy neighbors are invisible in averages.
Keep the provider abstraction layer; you will need multi-provider routing eventually.

AI at Scale

AI Cost Optimization, Performance Engineering, and Large-Scale Deployment

Table of Contents