Database Monitoring, API Monitoring, and AI Application Monitoring: Watching Every Layer

Monitoring, Errors & Performance covered the application itself. But a modern solution isn't one process — there's a database underneath, an API surface in the middle, and often an AI layer on top. Each layer fails differently and hides different symptoms. This tutorial covers how to monitor all three so a problem in any of them shows up on your dashboard before it becomes a user-visible outage.

Table of Contents

• Database Monitoring
  • What to Watch in a Database
  • Finding Slow Queries
• API Monitoring
  • Synthetic Checks
  • What to Track per Endpoint
• AI Application Monitoring
  • What's Different About AI
  • Instrumenting an LLM Call

Database Monitoring

Database monitoring watches the health and performance of the database layer — connections, query speed, storage, replication — so it stays fast and available before users have a reason to complain.

The database is where most application failures actually live. It's the shared resource every request hits, and its failure modes — disk filling up, a missing index, replication lag — are slow and silent. That combination is exactly what makes it dangerous: "nobody noticed until users complained."

What to Watch in a Database

A shortlist that covers most setups:

Most managed databases — RDS, Cloud SQL, Azure DB — expose these in a console with almost no setup. For self-managed databases, exporters like postgres_exporter for Prometheus scrape the same numbers automatically.

Finding Slow Queries

If you do one thing for your database, make it this: find slow queries before users do. The highest-leverage habit in database operations.

-- PostgreSQL: the slowest queries of the last week
SELECT query, calls, mean_exec_time, max_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 5;

This lists the five queries with the highest average execution time. The classic next step is running EXPLAIN on one of them to see where the time goes — usually a missing index or a full table scan. One added index frequently turns a 2-second query into a 5-millisecond one. It's the closest thing there is to free performance.

API Monitoring

API monitoring watches the endpoints your solution exposes and the third-party APIs it depends on, checking that they're reachable, fast, and returning the right kind of answer.

APIs are the contract between your solution and everything that uses it. When your own endpoint breaks, every consumer breaks with it. When a third-party API you depend on breaks, your app breaks too — even though your code is perfectly fine. Monitoring both sides tells you who to blame and where to start fixing.

Synthetic Checks

Synthetic monitoring is a robot acting like a user: on a schedule, it calls your API exactly the way a real client would, and alerts if the call fails or takes too long. It catches the failures that real-traffic monitoring misses — a login page that breaks only when nobody logged in that day, for example.

import requests

CHECK = {"url": "https://api.example.com/health", "timeout": 3.0}

def run_check():
    try:
        resp = requests.get(CHECK["url"], timeout=CHECK["timeout"])
        ok = resp.status_code == 200
        print(f"{'PASS' if ok else 'FAIL'} status={resp.status_code} "
              f"time={(resp.elapsed.total_seconds() * 1000):.0f}ms")
    except requests.Timeout:
        print("FAIL timed out after 3s")

run_check()

Run on a cron every minute and you've got a complete uptime monitor. It reports status code and latency of the health endpoint, and it fails loudly on timeout. Point it at the critical paths — login, checkout, the main read — because those are the ones that cost the most when they break.

What to Track per Endpoint

• Uptime — is it reachable right now?
• Status code distribution — what's the ratio of 2xx to 4xx to 5xx over time?
• Latency percentiles — p95 and p99 per endpoint, not just averages.
• Error rate — the 5xx ratio; your SLO from Tutorial 1 lives here.
• Dependencies — which upstream services each endpoint hits, so you know the blast radius when something breaks.

AI Application Monitoring

AI application monitoring tracks the special signals of an LLM-powered feature — cost, latency, output quality, drift, safety — in addition to whatever normal monitoring the rest of the app already has.

This one matters because an LLM call is not a normal API call. It can succeed technically while returning garbage. It costs money per token. It can leak data. And its quality decays silently as prompts or models change. Your normal monitoring sees "200 OK" and celebrates. AI monitoring asks whether the 200 OK was actually a good answer.

What's Different About AI

That last row is the sneaky one. A prompt tweak or a new model version can silently change every answer. Versioning and tracking what you sent is the only way to catch it after the fact.

Instrumenting an LLM Call

import time
import json

def call_and_log(prompt, model="gpt-4o"):
    start = time.time()
    response = call_model(model, prompt)          # your real call
    duration_ms = (time.time() - start) * 1000

    log_row = {
        "prompt_version": "support-v3",
        "model": model,
        "latency_ms": round(duration_ms, 1),
        "input_tokens": response.usage.prompt_tokens,
        "output_tokens": response.usage.completion_tokens,
        "cost_estimate": estimate_cost(response.usage),
        "flagged": is_refusal(response.text),
        "satisfies_guardrails": guardrail_check(response.text),
    }
    log_to_lake(json.dumps(log_row))
    return response.text

Every call records the prompt version, latency, token counts, estimated cost, and a couple of safety signals. Over a day, this log lets you answer the questions managers actually ask: how much does this cost us? How slow is it? How often does it refuse? Did the new prompt change anything? Without these rows, those questions are unanswerable.

For quality drift — is it still answering correctly? — keep a small set of known questions with known-good answers and run them on a schedule. A regression on that set is the AI equivalent of a failed health check.