Outcome
Is the result correct, complete and usable?
Fig. 1 · Data analysis
Revenue overstated 6% by duplicate rows.
Evaluation infrastructure for AI agents
Measure AI agents against how experts actually work.
We turn expert knowledge and real work processes into structured tasks, reference solutions and rigorous criteria — so AI developers can see whether an agent can perform the work, not just produce a plausible answer.
Fig. 1 Agent vs. professional, step by step
Concept illustration · example data
Task
0.00/ 1.00
Expert bar 0.80
Key difference
- Same
- Different
- Missing
§1 The gap
Benchmarks check answers. Work is a sequence of decisions.
An agent can return a convincing result through a process no professional would accept. As agents move from answering to doing, evaluation has to look at the work itself.
| Typical benchmark | Expert-grounded evaluation | |
|---|---|---|
| What is tested? | A prompt and a single answer | A complete task, done in a working environment |
| What counts as correct? | Matching an answer key | The expert's result, reached in a way the expert would accept |
| What gets scored? | Only the final output | Outcome, process and judgment, each with its own criteria |
| Who decides what "good" means? | Benchmark authors | Practitioners who do the work |
| Applied to the Fig. 1 examples | Every output looks finished, so all pass | Process and judgment fall short, so all score below 0.80 |
§2 Method
From expert work to a repeatable measurement.
Every evaluation starts with real work by someone who does it for a living.
-
01
Capture
Practitioners perform real tasks. We record every step, decision and outcome.
OutputExpert trace
-
02
Structure
Each task becomes a reproducible evaluation with a reference solution and weighted criteria.
OutputTask specification
-
03
Run
The agent does the same task under the same conditions. Every action is recorded.
OutputAgent trajectory
-
04
Evaluate
Each step is compared with the professional's, scored, and validated by people.
OutputScorecard
§3 What we measure
Three layers. One standard: would an expert accept it?
Process
Did the agent take steps a practitioner would accept, in a sensible order?
Fig. 1 · Software engineering
Loosened the test instead of fixing the bug.
Judgment
When the task called for a decision, did it make the right call?
Fig. 1 · Customer support
Refunded outside policy and missed a fraud pattern.
Criteria are written with practitioners and reviewed by people before they are used to score an agent.
§4 What we are building
Infrastructure, not services.
We are building
- Tooling for experts to author tasks, reference solutions and criteria
- A harness for running agents in realistic task environments
- Scoring that combines automated checks, expert rubrics and human validation
- A library of evaluations that grows across professions
We are notThe difference
- A project-management tool
- We don't organize or track work. We measure whether an agent can do it.
- An AI consultancy
- We don't deliver one-off projects. Each task becomes a versioned evaluation that can be rerun on any agent.
- A data-annotation vendor
- We don't label model outputs. Experts define tasks, reference solutions and criteria that score complete agent runs.
Expert knowledge is the input. The product is the platform that turns it into repeatable, comparable measurement.
§5 Stage and direction
Starting narrow. Built to expand.
-
Now
In development
A focused first suite
Expert-grounded tasks in a single professional domain, built and validated with practitioners.
-
Next
Planned
Adjacent professions
New domains and tooling that lets experts author and validate tasks directly.
-
Long term
Vision
The measurement layer for agent work
Infrastructure for measuring, understanding and improving what AI agents can do in real-world work, across industries.
§6 Work with us