Frontier labs buy human data to find where their models fail and to fix it. Every such project, ours or anyone's, fits on this one map.
A project is the set of answers to eight questions. Answer them in order; each answer narrows the next.
Well-known benchmarks from across the industry, each placed on the eight questions.
Same map, different answers. New domains (life sciences, CAD, mechanical engineering) and new modalities (voice, code, 3D) change the environment, the verifier and the expert pool; they do not need a new framework.
The eight questions as a production line. Walk a request left to right; every stage has a fixed set of choices, so the output is a fully specified project.
Two loops: calibration at stage 7 sends tasks back to stage 2 when frontier models pass too often or too rarely; the one-sentence failure statement is rewritten whenever stage 3 can't name a single primary mode.
"Frontier models fail at ___ when ___ because ___." If it can't be written, run three frontier models on ten sample tasks and read the failures first.
Pick the sector, then the specialty, then one job role. One role = one expert pipeline.
Decide the shape, the length, the output, and the share of tasks that should end in a question, a pause or a refusal.
Exactly one primary, up to three secondary. The primary dictates stages 2, 6 and 7.
One primary product, plus whatever must ship with it (a rubric with the tasks, seed data with the environment).
Match the lab's product surface. Computer use and coupled apps roughly double environment cost; choose them only when the failure needs them.
The smallest world in which the failure is visible and gradable. Statefulness is the cost driver.
Method follows task shape. Reference standard is the quality ceiling of the whole dataset. Calibrate on frontier models before shipping.
Use sets the acceptance bar and the delivery unit; the delivery unit sets the pricing unit.
One line per stage, one primary failure mode, a calibrated difficulty band, and a price. If any stage is blank, the project is not ready to quote.
Eighteen ways frontier models fail today. A project isolates one as primary and may measure up to three more on the side.
| Failure | What it looks like | How it is measured | Where it shows up today | Human data that targets it |
|---|---|---|---|---|
| Capability: the model cannot do the work | ||||
| Wrong expert answer | The professional content is wrong even when the format looks right. | Expert rubric; blind comparison with the expert's own work; exact match where answers are verifiable. | GDPval, HealthBench, Humanity's Last Exam, APEX, Harvey LAB (legal), Finance Agent | Hard tasks with reference answers; credentialed rubrics |
| Misses facts across sources | The answer needs facts from several documents, tabs, apps or emails and the model joins only some of them. | Field-by-field diff against a reviewed reference; must-have rubric criteria. | Tax-prep and spreadsheet benchmarks (Thrive, GDP.xlsx, SpreadsheetBench 2, GDP.pdf) | Realistic document packages with practitioner-reviewed references |
| Loses the thread | Plan, state or memory drifts over many steps, long contexts or multiple sessions. | Long-horizon task success; checkpoint partial credit; net value over simulated time. | Vending-Bench, TheAgentCompany, JobBench, LOCA-Bench 256K, METR time-horizon studies | Multi-day task environments; demonstrations with state hand-offs |
| Plans badly on open-ended work | Research, ML engineering or investigation with no fixed recipe: the model stops early, follows one path, never checks. | Expert rubric on process and output; reproduction of known results. | MLE-bench, PaperBench, WideResearch, DeepPlanning, BrowseComp (now near saturation) | Expert-authored open problems with rubrics for the process, not just the answer |
| Mis-uses tools | Wrong tool, wrong arguments, wrong order; clicks the wrong thing; mis-reads a dense screen. | End-state check; function-call accuracy. | Terminal-Bench 2.0, Toolathlon, MCP Atlas, BFCL v4, AndroidWorld, MobileWorld; OSWorld-Verified is saturating above 85% | Environments with verifiers; GUI demonstrations on new surfaces (mobile, CAD, design, spreadsheets) |
| Can't recover | A tool errors, state goes stale or an earlier step was wrong; the model retries blindly or silently drops the task. | Success after injected faults; step-level grading of the recovery. | Terminal-Bench repairs; mostly unmeasured elsewhere | Environments with injected faults; expert recovery demonstrations |
| Unreliable on repeat | Passes once, fails on the next run of the same task. | pass^k: must succeed k times in a row. | τ²/τ³-bench, Toolathlon-Verified, MCPMark | Verifiers that run many trials cheaply; task sets sized for repeat runs |
| Fails off the US / English default | Wrong currency, regulation, format or language; brittle on code-switching and local workflows. | Same benchmarks, rebuilt for a locale; locale-specific references. | Almost every public benchmark is US-centric and English-only | Country-specific task sets and experts (GST, local filings, Indian-English voice) |
| Judgment: the model could do it but decides wrongly | ||||
| Doesn't ask | The task is ambiguous and the right first move is a question; the model guesses. | Share of runs that ask before acting; quality of the question; end state after the answer. | τ-bench information gathering; otherwise rare | Underspecified tasks with a labelled right question and a simulated user |
| Doesn't pause | Takes an irreversible, paid or privileged action without seeking approval. | Correct stop at a labelled approval boundary; completion without crossing it. | Helpdesk evals ("seek approval"); personal-agent product evals | Environments with irreversible actions and an approval matrix |
| Doesn't say "impossible" | The task cannot be done as written; the model fakes a completion. | Correct abstention on infeasible tasks mixed in at a hidden ratio. | OSWorld infeasible subset; WorkArena++ | Impossible and conflicting tasks with labelled correct behaviour |
| Doesn't hand off | Should transfer to a human specialist (technician, clinician, partner) and package the handoff; instead it pushes on. | Correct escalation decision plus quality of the handoff note. | Helpdesk evals ("hand to a technician"); ITBench | Practitioner-written escalation criteria; graded handoff notes |
| Breaks policy | Ignores the written SOP, playbook or constraints while finishing the task. | Outcome plus rule adherence; forbidden-action count. | τ²/τ³-bench, Harvey LAB, EnterpriseOps-Gym | Policy documents plus policy-trapped tasks |
| Integrity and safety: the model does something it shouldn't | ||||
| Games the grader | Collects the reward without doing the work: edits the test, hard-codes the answer, satisfies the rubric's letter. | Verifier audits; adversarial runs against the grader; human spot checks. | Reported as common across RL environments by every vendor and lab | Hack-resistant verifiers; graded examples of gaming; judge calibration |
| Bends the rules for someone | Caves to user pressure, colludes with another agent, sandbags, or quietly grabs privileges. | Integrity-preserving behaviour in scenarios with an incentive to cheat. | Sabotage and sandbagging evals from lab safety teams; few public multi-agent sets | Scenarios with a counterparty who benefits from a rule bend; labelled correct behaviour |
| Manipulated | Follows instructions injected through emails, tickets, web pages, tool output or skill files. | Attack success rate alongside task utility. | AgentDojo; ClawSafety (40 to 75% attack success on frontier models) | Red-team suites with realistic placements, paired with benign tasks |
| Unsafe | Destructive actions, data leaks, PII exposure, dangerous advice. | Boundary crossings per run; harm-domain rubrics. | HealthBench safety axis; agent security benchmarks; Cybench / CyberGym for offensive capability | Environments with destructive actions and sensitive data; harm-boundary labels |
| Makes things up | Fabricates facts, records, citations or completions. | Evidence checks against the supplied inputs; citation resolution. | Legal and life-sciences drafting; tax prep ("do not copy prior-year amounts") | Citation-grounded tasks with verifiers; references the model must cite |
Public benchmarks crowd the capability rows and the top of the tool-use row is saturating. The judgment and integrity rows are where models fail most visibly in deployment and where the fewest good datasets exist.
What labs are buying in 2026, the building blocks we sell, and the frontier projects worth picking up.
Each block has a natural task shape, grading method and use. If a project's don't line up, it is two projects.
| Block | What experts hand over | Graded by | Lab uses it for |
|---|---|---|---|
| Hard tasks + reference | Tasks calibrated to beat frontier models, with the expert's own answer or deliverable | Exact match, field diff, rubric, blind comparison | Held-out evals; preference data |
| Rubrics | Weighted, atomic criteria with severity, written after seeing model attempts | Experts, or an LLM judge calibrated by experts | Evals; rubric-as-reward RL; preference data |
| Verifiers | Automatic checks on end state, tests or structured output, audited against gaming | Programmatic | RL environments; repeat-run evals |
| Environments | Mock or sandboxed apps with real actions, seed data, reset, optional simulated users and injected faults | Verifiers | RL training; agent evals |
| Demonstrations | Expert trajectories with rationale, including GUI recordings and recoveries from faults | Step-level review | Supervised fine-tuning; judge calibration |
| Graded runs | Model trajectories labelled step by step, pairwise or on a scale | Expert agreement | Reward models; judge calibration |
| Red-team sets | Injections, social engineering and incentive traps placed where real attackers would put them | Attack success + utility | Safety evals and training |
| Corpora | Real workbooks, documents, app data, voice, video, CAD files with rights cleared | Acceptance criteria | Pretraining; seed inputs for all of the above |
| Product-level panel evals | Real users with seeded data running the agent as shipped, across products | Rubric + end state | Product decisions; permission and scope policy |
Universal project types, chosen where public benchmarks are thin or saturating and where labs are spending. Each names the failure it targets and what it takes to build.
| Project type | What it is | Why now | What it takes |
|---|---|---|---|
| Measure | |||
| Credentialed professional evals | Rubric-graded deliverables and conversations written and graded by licensed or senior professionals: medicine, law, tax, audit, engineering, pharma regulatory. | The most common lab ask; GDPval, HealthBench and Harvey LAB set the pattern, and every domain and country beyond the US is still open. | Credentialed experts, rubric training, calibration runs, grading interface. |
| Judgment evals (ask, pause, refuse, hand off) | Task sets where the right behaviour is a question, an approval request, an abstention or an escalation, mixed at a hidden ratio with normal tasks. | Almost no public coverage; this is where deployed agents fail in front of customers. | Practitioners who write approval and escalation policy; environments with irreversible actions; simulated users. |
| Multi-day work evals | Tasks that span sessions and days with state hand-offs, interruptions and memory: a week of an analyst's or coordinator's job. | Task horizons that models can complete keep doubling; benchmarks still mostly measure minutes. | Stateful environment with persistence; expert demonstrations; checkpoint grading. |
| Locale-native task sets | GDPval-style and tool-use tasks built for India, the EU, the Gulf and other markets: local filings, regulation, currency, language, code-switching. | Public benchmarks are US-centric; labs selling globally need the gap measured. | In-country senior practitioners; locale-specific reference inputs. |
| Product-level panel evals | Real users with seeded data run the agent as shipped (model + harness + memory + connectors + approval UX) across competing products. | Model benchmarks don't predict product behaviour; labs now evaluate the whole stack. | Consented user panel, seeded app data, recording and review ops. |
| Train | |||
| Enterprise workflow gyms | Stateful mock mail, calendar, tickets, CRM, ledger, ITSM with write actions, seed data, reset and end-state verifiers, in one domain or coupled across apps. | Nearly half of new vendor projects are environments; enterprise ops is the largest unsaturated surface. | Environment engineers, practitioners for tasks, verifier audits against gaming. |
| New-surface computer-use demonstrations | Expert GUI trajectories on surfaces beyond the Linux desktop: mobile, CAD, design tools, spreadsheets, EHR, ERP. | Desktop benchmarks are saturating above 85%; the next surfaces have no demonstrations. | Hosted apps with licences, recording platform, experts who use the tools daily. |
| Error-recovery data | Faults injected mid-task (expired tokens, stale records, rejected calls, a wrong earlier step) with expert recoveries and step-level grades. | Recovery is the main gap between demo and deployment and is almost unmeasured. | Fault-injection layer in an environment; practitioners who handle these failures daily. |
| Hack-resistant verifiers and judge calibration | Verifiers audited against reward hacking, plus expert-graded trajectories that calibrate LLM judges and reward models. | Reward hacking is the pain point labs name first; rubric-as-reward training needs trusted graders. | Senior experts for grading, adversarial testers, agreement measurement. |
| Code beyond bug-fixing | Repo-level tasks with tests for refactoring, code review, data pipelines, migrations, incident remediation and security. | All ten frontier labs buy code; SWE-bench-style fixes are saturating, the adjacent tasks are not. | Senior engineers, containerised repos, hidden tests. |
| Voice agent environments | Policy-bound service tasks with a simulated caller over speech, tool calls and end-state verifiers; accents, noise, interruptions, code-switching. | Voice agents are shipping; text τ-bench has no voice equivalent at scale. | Speech stack, simulated caller, support practitioners, native speakers. |
| Protect | |||
| Agent security suites | Injection and social-engineering cases through emails, tickets, documents, web pages and skill files, paired with benign tasks so utility is measured too. | Attack success on frontier models is still 40 to 75% in realistic workspaces. | Security practitioners, environments with untrusted content paths. |
| Integrity scenarios | Counterparties with an incentive for the agent to bend a rule: user pressure, multi-agent collusion, sandbagging, privilege grabs; labelled correct behaviour. | Lab safety teams run these in-house; very little vendor supply. | Multi-agent orchestration, compliance practitioners, safety reviewers. |
| Safety boundary gyms | Environments where destructive and data-leaking actions exist and are sometimes the fastest route; tasks completable without crossing the boundary. | Personal and enterprise agents now hold real permissions; boundary behaviour is a release gate. | Admin and privacy practitioners; action logging in the environment. |
| Supply | |||
| Rights-cleared real artifacts | Real workbooks, document packages, app data, voice and video, CAD files, with consent, used as seed inputs for the projects above. | Realistic inputs are what make synthesis and grounding failures appear; synthetic inputs don't. | Contributor panels, consent and PII process, acceptance QC. |
Sources for the industry figures: the 2026 human-data market analyses by Pebblous and Glasp (Epoch AI and SemiAnalysis pricing interviews), the October 2026 agentic leaderboards, and the ClawSafety paper (arXiv 2604.01438). Figures will drift; the structure won't.