Datasets

Datasets

A dataset is a table of examples. Each example (row) has an input, an optional output, and optional metadata. A dataset is the unit you evaluate a prompt or application against: you run something over every example and compare what comes out against what the example says.

Rows in one dataset should look roughly alike — the same shape of input, the same shape of output, the same metadata keys. The dataset has no enforced schema, so this consistency is a convention you maintain, not something the system guarantees. When you add or edit rows, match the shape of the rows already there; a dataset where row 3 has a question field and row 4 has a prompt field is harder to evaluate and harder to reason about.

What an output actually means

The most common mistake is to treat a saved output as "the correct answer." It usually is not.

An output on a row is a reference, not a verdict. Treat it as "an answer that was recorded for this input," and ask where it came from before trusting it:

• A dataset is golden only when its outputs are genuinely ideal — hand-written or hand-verified to be the answer you want. Here the output really is the target.
• Far more often, outputs are baselines: captured from whatever produced them when the dataset was built (a model run, a production trace, an import). A baseline output records what the system did, which may be mediocre, outdated, or wrong. It is a point of comparison, not a goal.

So: never assume an output is right because it is present. If the user implies the dataset is golden, you can lean on the outputs as targets; if you don't know its provenance, say so and treat outputs as references. A row with no output at all is normal and fine — it just means "we have an input to run, but no recorded answer to compare against."

How datasets feed evaluators and experiments

• Evaluators are attached to a dataset to judge outputs along a dimension (correctness, relevance, format, and so on).
• Running a prompt or application over every example is an experiment. Each example produces an experiment output, and the evaluators score those outputs — often against the example's reference output. This is why the reference/golden distinction matters: an experiment compared against weak baselines tells you "did this change behavior," while an experiment compared against golden outputs tells you "is this correct."
• Every example-level change creates a new dataset version, recording who made it. Experiments are run against a specific version, so the dataset's history is part of the evidence.

Matching a dataset to the prompt you run over it

When you run a prompt over a dataset in the playground, each prompt template variable {{x}} is filled from the matching example field input.x — bound by name. So the dataset's input keys have to cover the prompt's template variables, or the run produces nothing useful.

Reconcile the two before you run (or before you build a dataset for a run):

• List the prompt's template variables and the dataset's input keys and make them line up. A {{customer_message}} variable needs an input.customer_message field — not input.message or input.question. A field that matches no variable is unused; a variable with no matching field renders empty.
• Cover every variable the prompt uses. A prompt with five variables run over a dataset that only carries one field fills one variable and blanks the other four.
• A playground run that finishes with empty output and no error almost always means this binding did not match — treat that as a misconfiguration to fix (align the field names, or map them), not as success.

If you are creating a dataset specifically to exercise a prompt, read the prompt's variables first and name the example input keys to match them from the start.

Splits

A split is a named slice of the examples. The same mechanism serves three distinct purposes — know which one the user means before acting:

1. Honest measurement (train / validation / test). The classic ML division. A held-out test split that you never tune against gives a trustworthy estimate of real performance; tuning until the test numbers look good (overfitting) destroys that. If the user is optimizing a prompt or a model, respect the test split as blind.
2. Facets (category / difficulty / type). Splits like single-hop vs. multi-hop, easy vs. hard, or by topic let you break an experiment down and see where a task is weak instead of reading one aggregate number. Use these when the user asks "where is it failing," not just "how well does it do."
3. Quick iteration (small chunks). A small split is useful for a fast pass in the playground before committing to a full regression run over the whole dataset. Cheaper and faster to learn from while iterating.

A single example can belong to more than one split, and a split is just a label on rows — moving a row between splits doesn't change the row's input or output.

How to work with datasets well

• Look before you change. Read existing rows before adding — to match their shape — and before editing or deleting — to act on the real current content, not an assumption. This is the same habit the prompt tools follow.
• Be honest about outputs. When you add a row, only present its output as the right answer if it genuinely is; otherwise call it a baseline/reference.
• Keep rows uniform. New and edited rows should match the field names and structure of the rest of the dataset.
• Respect splits. Don't blur a held-out test split into training material, and surface per-split breakdowns when the user wants to know where a task is weak.
• Treat changes as versioned and attributable. Edits and deletes are real mutations to a shared artifact others may be using, and each creates a new dataset version. Preview and confirm before applying.