Capability and baseline
Name the model capability and task. Agree which models, existing benchmarks or held-out checks would make a pilot informative.
Train your models on
economically valuable work
Business data and task environments scoped to a model capability,
with source rights and outcome checks defined for each pilot.
From business records
to research material.
Start with the capability and workflow you want to study. We scope the material around that use and describe what the source can actually support.
Business datasets
Operational records organised around a research question, with their source, coverage, permissions and known gaps described.
Orders, support cases, documents and software records.
Connected work histories
Linked records of what was known, the actions and tool use that followed, and the recorded outcome where those details exist.
A request connected to its decisions, changes and resolution.
Task environments
A defined starting state, available tools, reset behaviour and outcome checks, scoped to the capability you want to examine.
The task definition and scoring logic form part of the proposed pilot.
Start with your
research question.
A focused conversation establishes the fit before a broader delivery is agreed.
- 01
Define the capability
Tell us the workflow, target model capability and evaluation or training use you are exploring.
- 02
Assess a first sample
Where suitable material and permissions exist, review representative records or a task definition, including what is missing and how success would be checked.
- 03
Agree a pilot
Set its scope, volume, timeline, delivery format, permitted use, access and commercial terms before work begins.
Illustrative taskSoftware engineering
A workflow. An objective.
A result you can check.
A fictional software task: resolve a date-range export bug. Connected issues, code history, reviews and tests explain the work. An evaluation also needs a defined starting point, available tools and explicit outcome checks.
- Starting point
- The reported issue and repository before the fix, with a scoped workspace and test runner.
- Objective
- Correct the export behaviour without introducing regressions.
- Outcome checks
- Check the affected date range and relevant existing tests against agreed criteria.
Useful evidence.
Clear boundaries.
A credible pilot makes its contents, checks and rights inspectable. We identify what is available, what needs testing and what is not yet established.
What a work trace contains
Describe the observations, actions, tool calls, human turns and timestamps actually present. Identify missing links and fields.
Scoring and repeatability
Define the outcome checks or rubric, test for shortcuts and compare repeated runs. Validate automated scoring against human review where relevant.
Rights and delivery
Specify source permissions, allowed training or evaluation use, ownership, any exclusivity, delivery format and access for each proposed package.
From the blog.
Practical guides to the records, context and permissions behind a research package.
