ChatGPT and LLM APIs: Chat, API Calls & Streaming

You don't need to run a model to build with one. Every major lab exposes its models through two surfaces: chat products you click on (ChatGPT, Claude.ai) and programmatic APIs your code calls. This tutorial covers both and shows you how to make your first real API call.

Table of Contents

• Chat Products vs. APIs
• How an LLM API Call Is Structured
• Messages: System, User, Assistant
• Calling an LLM API from Python
• Streaming Responses
• Parameters That Change Behavior
• Managing Keys and Secrets

Chat Products vs. APIs

A chat product (ChatGPT, Claude.ai) is a finished UI wrapping an API. You type, the model replies. An API is the raw interface your own application calls — no UI, just a request and a response you parse in code.

For an FDE, chat is great for prototyping and quick questions. The API is what ships inside the solution.

How an LLM API Call Is Structured

Most modern LLM APIs follow one shape: you send a list of messages plus parameters, and get back the model's generated message. The API is stateless — every call stands alone, so you send the full conversation history each time.

You send:   model + list of messages + parameters
You get:    the model's reply message + usage (token counts)

Messages: System, User, Assistant

The conversation is a list of messages, each tagged with a role:

• system — instructions that set how the model should behave (covered fully in Tutorial 3).
• user — what the person typed.
• assistant — the model's previous replies, so the model can follow a thread.

[system]   "You are a support assistant. Be brief."
[user]     "What's our refund policy?"
[assistant]"Our refunds are processed within 5 business days."
[user]     "How do I request one?"

Explanation: you replay the history so the model can answer the latest question in context. Drop the earlier turns and it forgets the thread.

Calling an LLM API from Python

Here's a minimal call using the Anthropic SDK (anthropic). The pattern is the same for OpenAI and other providers — different client, same shape.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from env

resp = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=200,
    messages=[
        {"role": "user", "content": "Summarize this bug report in one line."}
    ],
)

print(resp.content[0].text)

Explanation: create the client, call messages.create, pass the model, a token cap, and the user message. The reply comes back in resp.content[0].text. The key is read from the environment, never typed in code.

Best Practice: Pick the model at one place in your config, not scattered through code — you'll upgrade it without editing everywhere.

Streaming Responses

Real users notice the first token quickly, and long answers feel broken if nothing appears for 20 seconds. Streaming returns text as it's generated, token by token.

with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=500,
    messages=[{"role": "user", "content": "Write a short changelog."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="")

Explanation: stream.text_stream yields chunks as they arrive. You render each chunk to the user instead of waiting for the whole reply. The final assembled text is also available on the stream.

Best Practice: Use streaming for any user-facing response; reserve non-streaming for background jobs where latency doesn't matter.

Parameters That Change Behavior

A few knobs control cost, speed, and randomness:

• max_tokens — the cap on the reply length. Cuts worst-case cost.
• temperature — randomness. Low (0–0.5) for factual/extraction work; high for creative writing. For most FDE tasks, low is right.
• system — behavior instructions (separate from the user message).

resp = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=100,
    temperature=0.2,          # low for deterministic extraction
    system="Extract the customer name and order ID from the text.",
    messages=[{"role": "user", "content": "Order #8821 for Dana Reyes."}],
)

Explanation: the low temperature and the extraction instruction make the output predictable — the right settings when you're feeding a reply into downstream logic.

Note: temperature is not a reliability switch. Even at 0, models can drift. Determinism is bought with structure and evaluation, not just a low knob.

Managing Keys and Secrets

API keys are credentials that bill you. Treat them like passwords: never hard-code them, never commit them to git, never paste them into logs.

# Never this
client = anthropic.Anthropic(api_key="sk-ant-...")   # secret in code!

# Do this — load from the environment
import os
key = os.environ["ANTHROPIC_API_KEY"]

Why useful: a leaked key means someone else bills you and may hit your rate limits. Keeping it in the environment (or a secrets manager) also makes the same code run on your laptop and in production unchanged.

Best Practice: Add .env and any key files to .gitignore; store secrets in the platform's secret manager in production.