Outcome
Is the result correct, complete and usable?
In Fig. 1 · C1 + C5
0.10 of 0.40
Entries documented, but the batch total was wrong.
Evaluation infrastructure for AI agents
Measure AI agents against how experts actually work.
We turn expert knowledge and real work processes into structured tasks, reference solutions and rigorous criteria — so AI developers can see whether an agent can perform the work, not just produce a plausible answer.
Fig. 1 One expert task, end to end
Concept illustration · example data · not product output
- 01Expert performs the task
- 02Encoded as a structured evaluation
- 03Agent attempts the same task
- 04Compared against expert criteria
- 05Measured outcome
01 Expert performs the task
An accounts-payable lead does the real task in an ERP sandbox. Each step is recorded with its rationale.
- assignment
- Approve the Q3 supplier payment batch
- inputs
- 215 invoices · 48 purchase orders
- trace
- 5 steps · 2 judgment calls
Expert result — the reference
$1,418,240 approved · 2 held
- INV-2291Duplicate of INV-2207 · $41,300
- INV-2318Price 3.4% above PO · $7,275
02 Encoded as a structured evaluation
The trace becomes a reproducible task, ap.reconcile.q3-close, scored by five weighted criteria reviewed by two practitioners.
| Criterion | Layer | Weight |
|---|---|---|
| C1Batch total matches the expert's | Outcome | .30 |
| C2Three-way match before approval | Process | .20 |
| C3Variances tested at the 2% policy | Process | .15 |
| C4Duplicates held and escalated critical | Judgment | .25 |
| C5Adjusting entries documented | Outcome | .10 |
03 Agent attempts the same task
Same environment, same inputs. Each agent step is lined up against the expert step it should correspond to.
| Step | Expert did | Agent did | Comparison | Criterion |
|---|---|---|---|---|
| 1 | Pulled open POs, goods receipts and Q3 invoices | Pulled POs, receipts and 215 invoices | Same | C2 |
| 2 | Three-way matched each invoice to its PO and receipt | Three-way matched all 215 invoices | Same | C2 |
| 3 | Tested price variances at the 2% policy limit | Tested price variances at a 5% limit | DeviatesINV-2318 (+3.4%) not flagged | C3 |
| 4 | Noticed INV-2291 duplicates INV-2207; held it and escalated | No equivalent step | MissingDuplicate was paid | C4 |
| 5 | Posted adjusting entries with written rationale | Posted adjusting entries with rationale | Same | C5 |
| = | $1,418,240 approved · 2 held | $1,466,815 approved · 0 held | Wrong result$48,575 paid that should have been held | C1 |
04 Compared against expert criteria
Each criterion is scored from the evidence in step 03, then validated by a person before results are reported.
| Criterion | Checked by | Evidence | Status | Points |
|---|---|---|---|---|
| C1Batch total | Automated check | Result row: off by $48,575 | Fail | 0 / .30 |
| C2Three-way match | Trajectory check | Steps 1–2 match the expert | Pass | .20 / .20 |
| C3Variance policy | Expert rubric | Step 3: tested, but at 5% not 2% | Partial | .08 / .15 |
| C4Duplicate escalated critical | Expert rubric | Step 4: no equivalent step | Fail | 0 / .25 |
| C5Entries documented | Expert rubric | Step 5 matches the expert | Pass | .10 / .10 |
| Total | 0.38 / 1.00 | |||
05 Measured outcome
0.38/ 1.00
Expert bar 0.80
01.00
Fails · critical criterion C4 missed
- Final-answer checkSees a complete, approved batchPasses
- Expert-grounded evaluationSees a paid duplicate and a broken policyFails
§1 The gap
Benchmarks check answers. Work is a sequence of decisions.
Most evaluations ask whether a model's final output looks right. Professional work is different. An accountant decides which variance matters. A clinician decides what to rule out first. An engineer decides when to stop and escalate.
An agent can return a convincing result and still take an approach no practitioner would accept. As agents move from answering questions to doing work, evaluation has to look at the work itself.
| Typical benchmark | Expert-grounded evaluation | |
|---|---|---|
| What is tested? | A prompt and a single answer | A complete task, done in a working environment |
| What counts as correct? | Matching an answer key | The expert's result, reached in a way the expert would accept |
| What gets scored? | Only the final output | Outcome, process and judgment, each with its own criteria |
| Who decides what "good" means? | Benchmark authors | Practitioners who do the work |
| Applied to the Fig. 1 run | The report looks complete, so it passes | A duplicate invoice was paid, so it fails at 0.38 |
§2 Method
From expert work to a repeatable measurement.
Every evaluation starts with a real task performed by someone who does it for a living. Each stage produces an artifact the next one depends on.
-
01
Capture
Experienced practitioners perform real tasks from their field. We record the steps, decisions, rationale and outcome — not only the final deliverable.
OutputExpert trace
-
02
Structure
Each task becomes a reproducible evaluation: environment, inputs, a reference solution and explicit, weighted criteria written with the expert.
OutputTask specification
-
03
Run
The agent attempts the same task under the same conditions. Every action, tool call and intermediate result is recorded.
OutputAgent trajectory
-
04
Evaluate
The trajectory is scored against expert-derived criteria with automated checks and expert rubrics, and validated by people before results are reported.
OutputScorecard
§3 What we measure
Three layers. One standard: would an expert accept it?
Process
Did the agent take steps a practitioner would accept, in a sensible order?
In Fig. 1 · C2 + C3
0.28 of 0.35
Matching done right; variances tested at the wrong limit.
Judgment
When the task called for a decision, did it make the right call?
In Fig. 1 · C4
0 of 0.25
The duplicate was paid instead of held and escalated.
Criteria are written with practitioners and reviewed by people before they are used to score an agent.
§4 What we are building
Infrastructure, not services.
We are building
- Tooling for experts to author tasks, reference solutions and criteria
- A harness for running agents in realistic task environments
- Scoring that combines automated checks, expert rubrics and human validation
- A library of evaluations that grows across professions
We are notThe difference
- A project-management tool
- We don't organize or track work. We measure whether an agent can do it.
- An AI consultancy
- We don't deliver one-off projects. Each task becomes a versioned evaluation that can be rerun on any agent.
- A data-annotation vendor
- We don't label model outputs. Experts define tasks, reference solutions and criteria that score complete agent runs.
Expert knowledge is the input. The product is the platform that turns it into repeatable, comparable measurement.
§5 Stage and direction
Starting narrow. Built to expand.
-
Now
In development
A focused first suite
Expert-grounded tasks in a single professional domain, built and validated with practitioners.
-
Next
Planned
Adjacent professions
New domains and tooling that lets experts author and validate tasks directly.
-
Long term
Vision
The measurement layer for agent work
Infrastructure for measuring, understanding and improving what AI agents can do in real-world work, across industries.
§6 Work with us