Data Fundamentals for FDEs: What Data Is and How It Moves

Before writing a single query or pipeline, an FDE needs a shared vocabulary for talking about data. This tutorial builds the mental model: what data is, how it's shaped, how it moves, and the basic distinctions (structured vs unstructured, batch vs streaming) that decide every tool you'll pick later.

Table of Contents

• What Is Data?
• Structured, Semi-Structured, and Unstructured
• Common Data Types
• The Data Lifecycle
• OLTP vs OLAP
• Batch vs Streaming
• Schema vs Schemaless
• Why This Matters for an FDE

What Is Data?

Data is a recorded fact about the world — a sale, a temperature reading, a click, a support ticket. On its own a number means little; arranged and interpreted, it becomes information a business can act on.

For an FDE, data is the raw material of every solution. Your job is rarely to create data — it's to move it, clean it, combine it, and make it useful inside a customer's systems.

Note: "Data" is plural; "datum" is one item. In practice everyone says "data is," and that's fine.

Structured, Semi-Structured, and Unstructured

The single most important split in data engineering is how regular the data's shape is:

Structured:      [ id | name      | signup_date ]   ← rigid grid
Semi-structured: { "id": 1, "tags": ["a","b"] }      ← flexible, nested
Unstructured:    "Q3 earnings call recording.mp3"     ← just bytes

Explanation: most business data an FDE touches is structured (it lives in tables) or semi-structured (it arrives as JSON). Knowing which you have tells you which tool fits — you don't reach for a spreadsheet to store MP3s.

Why it matters: the shape of the data is the first thing that determines the database, the cleaning approach, and even whether AI can use it.

Common Data Types

Whatever the shape, individual values have types. Getting these wrong is the #1 source of silent data bugs.

Common Mistake: storing money as a floating-point float. 0.1 + 0.2 is not exactly 0.30 in binary, so totals drift. Use decimal/numeric for currency.

The Data Lifecycle

Data rarely sits still. It moves through stages, and each stage is a job an FDE does:

Generate → Collect → Store → Process → Analyze → Act
  (app)     (API)    (DB)     (clean)   (report)  (decision)

Example: a customer's webshop generates an order → an integration collects it via API → it's stored in Postgres → a pipeline processes it into a daily summary → a dashboard analyzes it → a manager acts on it. Your code usually lives in the Collect, Store, and Process boxes.

OLTP vs OLAP

Two flavors of database workload you'll hear constantly:

• OLTP (Online Transaction Processing): many small, fast reads/writes of current state. "Insert this order," "update this user." This is the live app database.
• OLAP (Online Analytical Processing): big, slow reads that summarize history. "Total revenue per region last quarter." This is the reporting/warehouse side.

OLTP:  app writes 1 row        → fast, frequent, current
OLAP:  report scans 10M rows   → slow, rare, historical

Note: The same data often feeds both. Raw rows land in an OLTP database, then get copied into an OLAP warehouse for analysis. Don't force one database to do both badly.

Batch vs Streaming

How fresh must the data be?

• Batch: collect data, then process it in chunks on a schedule (every night, every hour). Simple, cheap, good enough for most reporting.
• Streaming: process each event the moment it arrives (clickstream, live location). More complex, needed only when "within minutes" isn't good enough.

Best Practice: default to batch. Most "we need real-time" requests are really "we need it within the hour," and batch is far easier to build and debug.

Schema vs Schemaless

• Schema = a declared shape: "this table has columns id, name, email, all required." The database enforces it. Safer for business data; catches bad input early.
• Schemaless = no fixed shape declared up front; each record carries its own structure (typical of document stores). Faster to start, but you must validate in your code instead.

Note: "Schemaless" doesn't mean "no rules." It means the database won't enforce them — so you must, usually in the transformation step.

Why This Matters for an FDE

FDEs live at the customer's data edge. You'll be handed a messy export, an undocumented API, or a 20-year-old table, and asked to make it flow into a product. The fundamentals above are the lens you use to decide:

• Is this structured or not? → which store (Tutorial 3)
• Batch or streaming? → which pipeline (Tutorial 5)
• OLTP or OLAP? → which query pattern
• Schema or schemaless? → where validation goes (Tutorial 6)

Get the fundamentals right and every later choice becomes obvious instead of scary.