Datasets, trajectories, tasks and environments: what is the difference?

Four different deliverables

An archive of business records is a source. It does not, by itself, define a model test. An AI team may need a documented dataset, an example of work, a task with success criteria, or a runnable environment. Each adds a different layer of context and control.

The terms overlap in practice. A dataset can contain traces; a task can use records from that dataset; an environment can run many tasks. Naming the deliverable helps a source owner and research team agree on what must actually be prepared.

One fictional case, four forms

Consider a fictional packaging business that notices several cartons from one production run are underweight. Its systems hold check-weigher events, batch identifiers, a shift note, a maintenance record, later inspection results and a release decision. This is an invented example, not a Metis customer record or a tested benchmark.

As a dataset, the business might describe a selected period of weight events and linked batch, inspection and maintenance records. The useful inventory says how many runs are covered, how events join, which measurements are absent and whether rejected runs were retained. Documentation of composition, collection and intended use follows the discipline proposed by Datasheets for Datasets; it is separate from permission to share a particular archive.

As a historical work trace, one case follows the alert, the measurements seen by the operator, the inspection, the adjustment and the later check. If the shift note was written after the outcome, it must be labelled as later evidence rather than information available at the start. A record of a human workflow is not automatically a trace of a model run. During an evaluation, the agent’s own tool calls and observations would form a new trial trace.

Read the dataset documentation proposal

Turn the case into a checkable task

A possible task starts before the maintenance adjustment. The agent receives the affected batch, an underweight alert and access to permitted read-only records. Its objective is to identify the likely cause and recommend a justified next check. The task definition states which records are visible, which actions are permitted and what evidence a successful answer must cite.

A grader could check whether the answer identifies the affected batch and supports its diagnosis with the available measurements. A separate human review might be needed for competing explanations. Merely repeating the eventual maintenance note would not prove reasoning if that note was visible at the start. The task designer must check for answer leakage and valid alternative solutions.

Anthropic describes a task as a test with inputs and success criteria, a trial as one attempt, a transcript as the record of that attempt and an outcome as the final environment state. These distinctions matter when an agent can take actions as well as write an answer.

Read Anthropic’s agent evaluation definitions

Make the environment repeatable

If researchers want the agent to investigate through tools rather than read a static packet, the environment must provide a stable starting state. For this fictional case, that could mean a copy of the relevant batch records, a read-only query tool, documented time limits and a reset that restores the same records before each trial. The later inspection and release decision stay outside the agent’s starting view and can inform outcome review.

Repeated attempts need the same task version, data snapshot and tool behavior. If a live system changes between runs, score differences may reflect a changed environment. Record environment version, permitted actions, failures, resets and grader version with the result. A claimed uplift needs a defined comparison and enough repeated trials to separate a real change from variation; this example reports no model result.

A minimum handoff card

Before calling any source a benchmark, write down the following details for one candidate case. A short card makes missing work visible and gives both sides a basis for deciding whether a static evaluation or a runnable task is feasible.

• Source boundary: systems, dates, case-selection rule, exclusions and permitted use.
• Join keys: how the alert, batch, inspection, intervention and outcome connect.
• Starting point: the exact information available before the decision.
• Action surface: tools, permissions, side effects, limits and reset behavior.
• Checks: observable success, acceptable alternatives and known grader gaps.
• Repeatability: task and environment versions, trial count and reported variation.

Choose the smallest useful output

If the links and coverage are unknown, start with a documented inventory and a bounded sample. If a complete case exists but cannot be replayed, a carefully labelled historical trace may still help assess the workflow. A task requires a defensible starting point and outcome check. A runnable environment requires the additional engineering to isolate and reset state.

The distinction prevents a familiar mistake: treating a large archive as evidence of an agent capability. The useful question is which deliverable the records can support, with what gaps, under which permissions and with which checks.