AI Evaluation, Hallucination Detection & Guardrails

A model that's fluent but wrong is dangerous in production. Before you ship, you need to measure how good it is (evaluation), catch when it's making things up (hallucination detection), and constrain what it's allowed to do (guardrails). This tutorial covers all three as one discipline: trust.

Table of Contents

• Why Trust Is the Real Product
• AI Evaluation Fundamentals
• Evaluation Metrics and Test Sets
• Human vs. Model-Based Evaluation
• What Hallucination Is
• Detecting Hallucinations
• What Are AI Guardrails?
• Types of Guardrails

Why Trust Is the Real Product

An AI system that answers confidently but wrong erodes user trust fast, and once trust is gone, adoption dies. Evaluation, hallucination detection, and guardrails are how you make a probabilistic model trustworthy enough to ship. They're not nice-to-haves — they're what turn a demo into a product.

AI Evaluation Fundamentals

Evaluation is measuring how well your AI system performs on a known set of cases. You run the system against a test set of inputs with expected outcomes, then score the results.

Why it matters: without measurement, "it seems good" is a guess. With evaluation, you can prove a change helped, catch regressions, and compare prompts or models objectively.

Test set: 100 questions with expected answers
Run system on each  ->  compare to expected  ->  compute a score (e.g. accuracy)

Evaluation Metrics and Test Sets

Choose metrics that match the task. Different tasks need different scores:

# Accuracy: the simplest score
correct = sum(actual == expected for actual, expected in zip(outputs, expecteds))
accuracy = correct / len(outputs)

Explanation: accuracy counts how many outputs match expected. But watch out — if 95% of cases are "low priority," a system that always says "low" scores 95% yet is useless. That's why imbalanced tasks use precision/recall.

Best Practice: Build a permanent test set of real, hard cases and rerun it on every prompt or model change. That's your regression safety net.

Human vs. Model-Based Evaluation

Two ways to score outputs:

• Human evaluation — people judge correctness/quality. Most accurate but slow and costly; use for small, high-stakes sets.
• Model-based evaluation — another LLM (or a cheaper model) judges the output. Fast and cheap, and good enough for most checks.

# Model-based check: is the answer grounded in the source?
resp = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=50,
    system="Return 'GROUNDED' if the answer is fully supported by the "
           "source text, otherwise 'NOT_GROUNDED'.",
    messages=[{"role": "user", "content": f"SOURCE:\n{source}\n\nANSWER:\n{answer}"}],
)
grounded = resp.content[0].text == "GROUNDED"

Explanation: a judge model compares the answer against the source text and flags whether the answer is supported. This is a fast, scalable way to catch unfaithful outputs across thousands of cases.

Best Practice: Use cheap/fast checks at scale and spend human review only on the outputs that matter most or that automated checks flag.

What Hallucination Is

A hallucination is a confident statement the model makes up — facts, names, citations, numbers — that aren't true or aren't in the source. It happens because the model predicts plausible text, not truth. It's especially dangerous in customer-facing or high-stakes answers.

Why it matters for FDEs: your solution sits inside a real business. A hallucinated order status or invented policy has real consequences, so you build detection around it.

Detecting Hallucinations

Detection means catching fabricated content before it reaches the user. Three practical approaches:

• Grounding checks — in RAG, verify the answer is supported by the retrieved context (the judge example above).
• Citation requirement — force the model to cite the source passage, then verify the citation exists.
• Consistency checks — ask the model the same question twice (or re-answer) and look for disagreement; a hallucinated fact often shifts between tries.

# Ask twice with a different phrasing; disagreement is a warning sign
a = ask("Who is the CFO of Acme?")
b = ask("Acme's chief financial officer is?")
if normalize(a) != normalize(b):
    flag_for_review(a)   # likely hallucination or uncertainty

Explanation: a real fact tends to reproduce consistently; a fabrication drifts. When the two answers disagree, don't trust it — escalate or re-prompt.

Best Practice: Never ask the model to just "be careful." Verify against a source (grounding) or cross-check answers. That's what catches hallucination.

What Are AI Guardrails?

Guardrails are the constraints and controls you put around the model to keep its behavior inside acceptable bounds. They don't rely on the model behaving well — they enforce limits from outside.

Why they're needed: prompts reduce errors but can't eliminate them. Guardrails are the safety net that catches what the prompt missed: jailbreaks, off-topic output, harmful content, or risky actions.

Types of Guardrails

Guardrails come in layers:

• Input filters — block toxic, private, or off-scope requests before the model sees them.
• Output filters — check the model's reply for banned content before showing it.
• Action controls — require approval or confirmation before irreversible actions (sending, paying, deleting).
• Scope limits — restrict which tools/data the model can reach (allowlist tools, mask sensitive fields).
• Rate limits — cap usage to control cost and abuse.

def safe_reply(user_input):
    if is_blocked_input(user_input):        # input filter
        return "I can't help with that."
    reply = generate(user_input)
    if violates_policy(reply):              # output filter
        return "I can't provide that."
    return reply

Explanation: the request and the reply each pass through a filter before and after the model. Even if the model tries something off-limits, the filter catches it before the user — or your systems — sees it.

Best Practice: Treat guardrails as a security layer you can't skip, not an optional add-on. Assume the model will be tested by users and design the guardrails to hold regardless.