Case Study 1 — Building an AI Customer Support System

A full end-to-end FDE case study: take a company drowning in support tickets and ship an AI-powered support assistant — from discovery to monitoring in production.

Table of Contents

• The Problem
• Discovery Phase
• Designing the Solution
• Building the Prototype
• API Integration
• Deployment
• Monitoring
• Full Build — Production Codebase
• Prompt Library
• Evaluation Plan
• Project Timeline and Cost
• Security and PII Checklist
• Runbook — When Things Break

The Problem

A mid-size e-commerce company receives 2,000+ support tickets per day. 70% are repetitive questions: "Where is my order?", "How do I return an item?", "My refund is late."

The support team takes 6–10 hours to reply during peak days. Customers churn. The company wants automation without losing the human touch for complex issues.

This is the classic FDE opportunity: a well-defined pain point, a measurable outcome (response time), and a scope small enough to prototype fast.

Discovery Phase

Never build before you understand. Discovery answers three questions:

1. What exactly is happening? — Sit with support agents for a day. Read 200 real tickets.
2. Where does time go? — Measure. You'll likely find agents spend 5 minutes per ticket just looking up order status in the CRM.
3. What does success look like? — Define it in numbers: "First response time under 2 minutes for the top 10 ticket categories."

Practical discovery checklist

Note: The single most valuable discovery output is the top 10 ticket templates. If 70% of volume is 10 patterns, your MVP only needs to handle 10 patterns well.

Designing the Solution

The architecture that comes out of discovery:

Customer ticket
      │
      ▼
Ticket Classifier (LLM) ──► category + urgency
      │
      ▼
Intent Router
      ├─ Order status  ──► call Order API ──► instant reply
      ├─ Returns       ──► runbook answer ──► instant reply
      ├─ Refund delay  ──► call Payment API ──► reply + flag
      └─ Complex       ──► route to human agent

Key design decisions:

• Draft, don't send. Version 1 shows AI-suggested replies to agents; agents hit send. This builds trust and gives you labeled data for free.
• APIs over guesses. The model must never invent an order status. Status comes from the Order API; the model only phrases the reply.
• Clear escalation. Legal threats, refunds above a threshold, or low-confidence classification all go to a human.

Building the Prototype

Build in one week with real (anonymized) tickets. No deployment, no UI polish — just prove the core loop works.

import json, openai

def classify_ticket(ticket_text: str) -> dict:
    """Classify a support ticket into a known category."""
    prompt = f"""Classify this support ticket into exactly one category:
order_status, return_request, refund_delay, complaint, other.

Ticket: "{ticket_text}"

Reply in JSON: {{"category": "...", "urgency": "low|medium|high"}}
"""
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
        temperature=0,
    )
    return json.loads(response.choices[0].message.content)

Simple explanation: one LLM call turns messy customer text into structured data — json.loads turns the JSON reply into a Python dict you can route on. temperature=0 keeps classification consistent, and forcing JSON output makes the result easy to parse in code.

Adding a draft reply generator

def draft_reply(ticket: dict, order_info: dict | None) -> str:
    context = f"Order details: {order_info}" if order_info else "No order lookup performed."
    prompt = f"""You are a polite support assistant for an e-commerce store.
Write a short reply (max 80 words) to this ticket.

Category: {ticket['category']}
{context}

Rules: never promise a refund date; never invent order details."""
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.3,
    )
    return response.choices[0].message.content

Simple explanation: the model gets real facts (order info from the API) as context, so it phrases an accurate reply instead of hallucinating one.

Test the prototype against 100 historical tickets and score it with the support lead. Aim for ~85%+ category accuracy before integration.

API Integration

Now wire the prototype into the systems the company already uses. This is where most FDE work actually happens.

Zendesk-style webhook integration

from flask import Flask, request, jsonify

app = Flask(__name__)
# orders_api and zendesk are thin client wrappers around the company's
# existing systems (internal Orders API + Zendesk API) — defined in integrations.py.

@app.route("/webhook/new-ticket", methods=["POST"])
def new_ticket():
    payload = request.get_json()
    ticket_id = payload["ticket_id"]
    ticket_text = payload["description"]

    ticket = classify_ticket(ticket_text)

    # urgency check comes FIRST — a high-urgency order question must escalate, not auto-draft
    if ticket["urgency"] == "high":
        zendesk.assign_agent(ticket_id, queue="escalations")
        return jsonify({"status": "escalated"})
    elif ticket["category"] == "order_status":
        order = orders_api.get_order(payload["customer_email"])  # real system of record
        reply = draft_reply(ticket, order)
    else:
        reply = draft_reply(ticket, None)

    zendesk.post_internal_note(ticket_id, f"AI suggested reply:\n{reply}")
    return jsonify({"status": "drafted"})

Simple explanation: a webhook fires when a ticket arrives. The classifier routes it; known categories get a drafted reply posted as an internal note so the agent stays in control. High-urgency tickets skip the AI entirely.

Integration gotchas worth knowing:

• Rate limits. Support spikes hit your LLM provider's limits. Add a queue (Redis/Celery) between the webhook and the model call.
• Idempotency. Webhooks fire twice sometimes. Guard with the ticket ID so you never draft twice.
• PII. Mask emails and phone numbers before sending ticket text to an external API.

Deployment

Ship v1 as "agent assist" (drafts only), not "auto-reply". Deploy with the same boring stack every company trusts: a small Flask/FastAPI service in Docker, behind the company's load balancer.

# docker-compose.yml (simplified)
services:
  support-ai:
    build: .
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ZENDESK_SUBDOMAIN=${ZENDESK_SUBDOMAIN}
      - ZENDESK_TOKEN=${ZENDESK_TOKEN}
      - ORDERS_API_URL=${ORDERS_API_URL}
      - ORDERS_API_KEY=${ORDERS_API_KEY}
    restart: unless-stopped

Rollout plan:

1. Phase 1: AI drafts replies for one category only (order status). Agents review.
2. Phase 2: Expand to all high-volume categories. Track acceptance rate — how often agents send the draft unchanged.
3. Phase 3: Auto-send for categories with >95% agent acceptance and low severity.

Best practice: the acceptance rate is your true quality metric, not model accuracy benchmarks. A >95% acceptance rate on a category is what unlocks auto-send for it.

Monitoring

Deployment isn't the finish line — it's the start of measurement.

# app/metrics.py — imported by the webhook handler and the classifier
import functools, logging, time

logger = logging.getLogger("support_ai")

def track(ticket_id: str, category: str, latency_s: float,
          accepted: bool, escalated: bool = False):
    logger.info("ticket_metric", extra={
        "ticket_id": ticket_id,
        "category": category,
        "latency_s": round(latency_s, 2),
        "accepted": accepted,
        "escalated": escalated,
        "ts": time.time(),
    })

def track_error(source: str, message: str):
    logger.error("component_error", extra={"source": source, "message": message})

def track_latency(op: str):
    """Decorator: logs how long the wrapped function took."""
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            start = time.time()
            try:
                return fn(*args, **kwargs)
            finally:
                logger.info("latency", extra={
                    "op": op,
                    "latency_s": round(time.time() - start, 2),
                })
        return wrapper
    return decorator

Simple explanation: every AI interaction logs one structured event. Pipe these logs into any dashboard (Grafana, Metabase, even a spreadsheet at first) and review weekly with the support lead.

Also sample 20 AI replies per week and read them yourself. No dashboard catches a subtle tone problem — reading real replies does.

Full Build — Production Codebase

The prototype was good enough to prove the idea. Production code needs error handling, retries, config management, and structure. Here is the actual service, file by file.

Project structure

support-ai/
├── app/
│   ├── main.py          # FastAPI webhook endpoint
│   ├── classifier.py    # LLM classification
│   ├── drafter.py       # LLM reply generation
│   ├── integrations.py  # Zendesk + Orders API clients
│   ├── config.py        # settings from environment
│   └── metrics.py       # structured logging
├── tests/
│   └── test_classifier.py
├── Dockerfile
├── docker-compose.yml
└── requirements.txt

config.py — no secrets in code

from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    openai_api_key: str
    zendesk_subdomain: str
    zendesk_token: str
    orders_api_url: str
    orders_api_key: str
    auto_send_categories: list[str] = []   # empty until acceptance rate proves it's safe
    escalation_queue_id: int = 9

    model_config = SettingsConfigDict(env_file=".env")

settings = Settings()

Simple explanation: every credential and every business decision (which categories auto-send, which queue gets escalations) lives in configuration, not code. Changing "auto-send" is a config change reviewed by the support lead — not a code deploy.

classifier.py — with retry and fallback

import json, openai
from openai import APIError, RateLimitError
from .config import settings
from .metrics import track_latency

VALID_CATEGORIES = {"order_status", "return_request", "refund_delay", "complaint", "other"}

CLASSIFY_PROMPT = """Classify this support ticket into exactly one category:
order_status, return_request, refund_delay, complaint, other.

Ticket: "{ticket_text}"

Reply in JSON: {{"category": "...", "urgency": "low|medium|high"}}"""

@track_latency("classify")
def classify(ticket_text: str) -> dict:
    client = openai.OpenAI(api_key=settings.openai_api_key)
    for attempt in range(3):
        try:
            response = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{"role": "user", "content": CLASSIFY_PROMPT.format(ticket_text=ticket_text[:4000])}],
                response_format={"type": "json_object"},
                temperature=0,
                timeout=15,
            )
            result = json.loads(response.choices[0].message.content)
            if result.get("category") in VALID_CATEGORIES:
                return result
        except (RateLimitError, APIError, json.JSONDecodeError) as e:
            if attempt == 2:
                return _fallback(ticket_text, str(e))
    return _fallback(ticket_text, "invalid output")

def _fallback(ticket_text: str, reason: str) -> dict:
    """Never silently drop a ticket — route to a human with context."""
    return {"category": "other", "urgency": "high", "fallback": True, "reason": reason}

Simple explanation: three retries with a hard 15-second timeout, and a fallback that escalates to a human instead of guessing. The worst outcome for a support system is a ticket vanishing — the fallback guarantees that never happens, and the reason field tells you exactly why the AI failed so you can fix it.

main.py — the complete webhook handler

import time
from fastapi import FastAPI, Request, BackgroundTasks
from .classifier import classify
from .drafter import draft_reply
from .integrations import zendesk, orders_api
from .config import settings
from .metrics import track, track_error

app = FastAPI()
processed: set[str] = set()   # use Redis in production

@app.post("/webhook/new-ticket")
async def new_ticket(request: Request, background: BackgroundTasks):
    payload = await request.json()

    # 1. Acknowledge fast — Zendesk retries if you're slow
    ticket_id = payload["ticket_id"]
    if ticket_id in processed:
        return {"status": "duplicate_ignored"}
    processed.add(ticket_id)

    background.add_task(handle_ticket, ticket_id, payload)
    return {"status": "accepted"}

def handle_ticket(ticket_id: str, payload: dict):
    start = time.time()
    text = mask_pii(payload["description"])
    ticket = classify(text)

    if ticket.get("fallback") or ticket["urgency"] == "high":
        zendesk.assign_agent(ticket_id, queue=settings.escalation_queue_id)
        track(ticket_id, ticket["category"], time.time() - start, accepted=False, escalated=True)
        return

    order = None
    if ticket["category"] == "order_status":
        order = safe_order_lookup(payload["customer_email"])   # None on API failure

    reply = draft_reply(ticket, order)

    if order is None and ticket["category"] == "order_status":
        # can't answer without facts — send to human, don't send a vague reply
        zendesk.assign_agent(ticket_id, queue=settings.escalation_queue_id)
    elif ticket["category"] in settings.auto_send_categories:
        zendesk.post_public_reply(ticket_id, reply)   # only proven-safe categories
    else:
        zendesk.post_internal_note(ticket_id, f"AI suggested reply:\n{reply}")

    track(ticket_id, ticket["category"], time.time() - start, accepted=False)

def safe_order_lookup(email: str) -> dict | None:
    try:
        return orders_api.get_order(email, timeout=5)
    except Exception as e:
        track_error("orders_api", str(e))
        return None

def mask_pii(text: str) -> str:
    import re
    text = re.sub(r"\b[\w.+-]+@[\w-]+\.[\w.]+\b", "[email]", text)
    text = re.sub(r"\b\d{7,}\b", "[number]", text)
    return text

Simple explanation: the webhook responds accepted in milliseconds and does the real work in a background task — support platforms retry webhooks aggressively, and a slow endpoint causes duplicate processing. Two rules protect customers: PII is masked before text leaves the company, and if the Orders API fails, the ticket escalates rather than the AI answering without facts.

tests/test_classifier.py — evals are unit tests too

import pytest
from app.classifier import classify, VALID_CATEGORIES

CASES = [
    ("my order #4521 hasn't arrived and it's been 2 weeks", "order_status"),
    ("how do I return these shoes, they don't fit", "return_request"),
    ("where is my refund, you said 5 days", "refund_delay"),
    ("this is the worst service ever, I want to speak to a manager", "complaint"),
    ("do you ship to Canada?", "other"),
]

@pytest.mark.parametrize("text,expected", CASES)
def test_classification(text, expected):
    assert classify(text)["category"] == expected

def test_gibberish_falls_back():
    result = classify("asdfgh 12345 !!!!")
    assert result["category"] in VALID_CATEGORIES   # never crashes, never unknown

Simple explanation: these five cases came from discovery (real tickets) and run on every deploy. When a prompt change breaks one, you know before the customers do.

Prompt Library

Keep prompts in one reviewed file, versioned like code — drafter.py implements these prompts, and the classifier uses CLASSIFY_PROMPT from classifier.py. Every prompt states its rules explicitly.

Example — the most sensitive one:

DRAFT_COMPLAINT = """You are a support assistant for an e-commerce store.
A customer wrote an angry message. Write a reply that:
1. Acknowledges their frustration in one sentence.
2. States that a specialist will personally follow up within 4 hours.
3. Does NOT apologize in legal terms.
4. Does NOT offer any compensation, discount, or refund.
Max 60 words.

Customer message: "{ticket_text}"
"""

Simple explanation: complaints are where an AI can create legal or financial liability ("we're so sorry, here's 20% off"). The prompt forbids compensation entirely — offering it is a human decision, every time.

Evaluation Plan

Before launch, and every week after, score the system against a labeled set.

1. Build the golden set. Take 150 real historical tickets. Have the support lead label each with the correct category, urgency, and an acceptable reply.
2. Score automatically on every deploy:

def run_eval(golden_set: list[dict]) -> float:
    correct = sum(
        classify(t["text"])["category"] == t["expected_category"]
        for t in golden_set
    )
    return correct / len(golden_set)

3. Gates to pass before rollout expands:

4. Regression policy: any hallucinated factual claim in a sent reply triggers a review of that category; two in a month pauses auto-send for it.

Project Timeline and Cost

A realistic 6-week timeline for one FDE:

Estimated running cost at 2,000 tickets/day:

Simple explanation: compare that against ~1.5 support headcounts saved at peak — the ROI conversation with leadership writes itself. But always present cost with the acceptance-rate metric; a cheap system agents ignore costs more than it saves.

Security and PII Checklist

• PII (emails, phones, addresses) masked before any text is sent to the LLM provider
• API keys in environment variables / secret manager, never in git
• LLM data-retention settings configured (zero-retention endpoint if policy requires)
• Webhook endpoints verify a shared secret or signature before processing
• Access to logs restricted — logs contain ticket content
• Legal/DPO review of the data flow diagram before launch (one page: what leaves, where, why)
• Kill switch: one config flag (enabled_categories=[]) turns the AI off without a deploy

Runbook — When Things Break

Note: review the runbook with the support lead — they are usually the first to notice something is wrong, and they need to know exactly who to call.