Sign in

ResearchBenchmarks

χ-BenchCan AI agents automate end-to-end, policy-rich healthcare workflows?

Thirty frontier agents, 75 long-horizon healthcare workflows, and a best score of 28%. This is what broke, why, and where it matters.

What we built

U.S. healthcare runs on long, fragmented, policy-heavy workflows, the kind that take a nurse, a coordinator, and a medical director hours of clicking, calling, and chart-reading to push to a terminal action. Frontier AI agents are increasingly pitched as the natural automation candidate here. We wanted to know how close they actually are.

So we built a high-fidelity simulator of 21 real healthcare apps (provider EHRs, payer UM portals, care-management consoles), wired together by their own state machines, and dropped frontier agents into them with a 1,279-document Managed-Care Operations Handbook and 200+ role-scoped MCP tools. Each task starts with a real clinical case (a knee-replacement prior auth, a chemotherapy review, a diabetes care-plan kickoff) and asks the agent to drive it end-to-end to a submitted packet, a finalized determination, or a complete care plan with patient consent. When the agent stops, a composite verifier reads the full workspace and grades the run.

Agent
Frontier harness and model
  • 200+ role-scoped MCP tools
  • 1,279-document handbook
  • Workspace files and chart access
Simulated healthcare world
21 apps behind 3 MCP servers
  • Prior authorization · Provider5 apps
  • Utilization management · Payer10 apps
  • Care management · Care manager5 apps
Verifier
Two-layer scorecard
  • Deterministic contract
  • LLM judge
  • Passes only if both pass
3 domains75 tasks30 agent setups3 trials per task
Figure 1.The χ-Bench environment. The agent works through 21 simulated healthcare apps over MCP, writes its work to a shared workspace, and a two-layer verifier grades the whole run.

Each of the three domains has 25 long-horizon tasks. To produce ground truth, a clinician (or an author wearing that hat) walked every task end to end on the live UI before it shipped: the average task takes 21 steps and the longest 40, and each is graded independently across three trials.

Figure 2.χ-Bench in 2:39: how the benchmark tests whether agents can work through real healthcare administration, across policies, applications, roles, evidence and long sequences of actions.

Why this is hard

Coding benchmarks like SWE-Bench and Terminal-Bench measure long-horizon execution against a clean, deterministic backdrop: the file system doesn't talk back, the build either passes or it doesn't. Healthcare workflows look superficially similar but are wired differently in three ways that almost no current benchmark combines.

Policy density. Every decision has to be grounded in a specific rule: a payer's medical-necessity criterion, an internal escalation procedure, a state regulation. There are thousands of them, they vary by payer and plan year, and “misreading the rule” and “applying the right rule incorrectly” are different failure modes that we score separately.

Multi-role composition. A single PA case touches a coordinator, a UM nurse, a medical director, and sometimes a peer-to-peer call between an MD on each side. The agent has to switch role context, authority boundaries, and what it's safe to write, and every handoff is terminal. Submit the wrong packet and there's no rollback.

Multilateral interaction. Some steps aren't tool calls but live multi-turn conversations: a peer-to-peer review with the payer's MD, an RFI back to the provider, a twenty-minute outreach call with a chronic-disease patient. The agent has to drop out of execute-mode, hold a real conversation, and carry the result back into the workflow.

Task pa_t00869 steps · 7m41s
Reward0.00 · 7/11
Seat 1 · provider intake
Clinical Intake Clerk
chart.search_patients
chart.get_patient_chart
chart.list_candidate_orders
people.list_ordering_providers
Seat 2 · case construction
PA Coordinator
cases.create_from_ordercode=43775 · site=outpatient
forms.list_required_forms
Seat 3 · documentation review
Clinical Reviewer
docs.upload_evidencechart_note
docs.upload_evidencelab_result
forms.save_form_responsequestionnaire
docs.upload_evidenceletter_of_medical_necessity
Seat 4 · disposition + submit
PA Submitter
docs.create_submission_bundle
auth.finalize_provider_determinationfinal_action=submit_pa
auth.submit_authorization
auth.check_status
Medical Policy · 2026T0362QQ
Bariatric Surgery
✓
BMI ≥ 40 documented
42.3
✓
Two supervised programs documented
✓
Procedure code 43775 on covered list
✗
Site of service must be inpatient_hospital
agent submitted outpatient
✗
Pre-op psych eval completed (referral ≠ done)
!
final_action should be gather_more_evidence
00:00
00:32↻ auto-loops
Figure 3.One agent, one trial, one final scorecard: a replay of an actual trajectory from the claude-code-opus-4-6 submission, with its real tool calls, policy gates and scorecard outcome.

How we grade a run

When the agent stops, the verifier reads everything it touched: world-store updates, files in the workspace, the full tool-call event trail, and any conversation transcripts. Two layers grade in parallel and both have to pass.

The deterministic contract covers checks that have a single right answer: did the case status reach the expected terminal state? did the right CPT codes land on the request? was a peer-to-peer raised when policy required one? The rubric LLM judge (pinned to claude-opus-4-7) handles items the contract can't formalize: was the medical-necessity rationale grounded in the cited policy? did the outreach call respect patient autonomy or quietly badger a refusing member into “yes”?

We report pass@1, pass@3 (any of three independent trials), and the strict pass^3 (all three) as a reliability metric.

What the run leaves
  • Per-stage states
  • Side-effect artifacts
  • Workspace documents
  • Event log
  • Conversation transcripts
Layer 1
Deterministic contract
Code-checkable assertions
Terminal status reachedStage payload valuesRequired event log entriesDocument field valuesCross-app side effects
and
Layer 2
LLM judge
Opus 4.7, majority of three votes
Policy alignmentReasoning soundnessInternal coherencePatient engagementAutonomy-first outreach
Trial passes
only if both layers pass
Figure 4.How a run is graded. Every trial leaves a persisted record; a deterministic contract and an LLM judge grade it in parallel, and the trial passes only when both do.

Headline results

We ran 30 harness × model combinations: every frontier proprietary stack (Claude Code, OpenAI Codex, Gemini CLI) paired with that lab's closed-weight models, plus four open-source frameworks (OpenClaw, Hermes, OpenAI Agents SDK, DeepAgents) sweeping five open-weight models. Best in class is Claude Code + Opus 4.6 at 28.0% pass@1. The per-domain leaders split cleanly across three labs: Codex + GPT-5.5 leads PA at 29.3%, Opus 4.6 leads UM at 41.3%, and Opus 4.7 leads CM at 32.0%. There is no single “best agent” for healthcare workflows yet.

OpenAI · CodexAnthropic · Claude CodeGoogle · Gemini CLI
Prior authorization
Best 29.3%, GPT-5.5
  • GPT-5.529.3
  • GPT-5.424.0
  • Opus 4.724.0
  • Sonnet 4.624.0
  • Opus 4.618.7
  • Gemini 3 Flash18.7
  • Gemini 3.1 Pro14.7
  • GPT-5.4 Mini10.7
  • Haiku 4.50.0
Utilization management
Best 41.3%, Opus 4.6
  • Opus 4.641.3
  • Sonnet 4.634.7
  • GPT-5.532.0
  • Gemini 3 Flash18.7
  • GPT-5.417.3
  • Opus 4.717.3
  • Haiku 4.514.7
  • GPT-5.4 Mini13.3
  • Gemini 3.1 Pro6.7
Care management
Best 32.0%, Opus 4.7
  • Opus 4.732.0
  • Opus 4.624.0
  • Sonnet 4.620.0
  • GPT-5.46.7
  • Haiku 4.54.0
  • GPT-5.51.3
  • GPT-5.4 Mini1.3
  • Gemini 3.1 Pro0.0
  • Gemini 3 Flash0.0
Figure 5.pass@1 by domain for the nine proprietary harness and model setups, sorted within each domain. The leaders split across all three labs, so there is no single best agent for healthcare workflows yet.

Reliability is the bigger problem. No agent clears 20% on pass^3, the metric where the same agent has to pass the same task three runs in a row. Agents that win one trial often fail the next on the same case. Two stress tests collapse the headline number further:

The marathon. Load all 25 tasks of a domain into a single agent session and ask it to finish them in any order. The best agents finish with overall 3.8%. On PA, neither leading agent submits a single authorization across 25 queued cases despite touching most of them. Long context doesn't save you: Opus 4.7 (1M-token context, no compaction) and GPT-5.5 (auto-compacts 4–6× per session) fail in roughly the same shape.

The end-to-end arena. Wire two Codex + GPT-5.5 agents together, one as the provider and one as the payer, and let them exchange information only through MCP tools. The PA configuration that scores 30.4% solo collapses to 0% when the payer agent and cross-role checks join: 18 of 23 cases never reach an MD decision, and on the five PA cases that policy requires a peer-to-peer call, neither side raises one.

Cost vs accuracy

Spending more does not buy reliability in healthcare workflows. Plot every harness × model configuration by mean per-trial spend and pass@1 and the field falls into four quadrants. The Sweet Spot (cheap and accurate) is sparsely populated and dominated by a handful of frugal open-weight pairings. The Premium tier (Claude Code paired with Opus 4.6 or Sonnet 4.6, and Codex with GPT-5.5) buys the top of the leaderboard but at 4–50× the trial cost of the frontier's budget end. Everything to the right of the $1 line and below the 13% bar is Overpriced: Claude Opus 4.7 on OpenClaw spends almost $12 per trial to land at 17.3% pass@1.

The connecting line traces the cost-accuracy Pareto frontier: no other configuration is both cheaper and more accurate than the points on it. Its seven configurations, labelled on the chart, span two orders of magnitude in spend for a 22-point accuracy spread.

Sweet spot: cheap and accuratePremiumBudgetOverpriced0%10%20%30%$0.1$1$10Average cost per trial (log scale)pass@1GPT-5.4 · Codex: 16.0% at $1.30GPT-5.4 Mini · Codex: 8.4% at $0.27Claude Opus 4.7 · Claude Code: 24.4% at $9.91Gemini 3.1 Pro · Gemini CLI: 7.1% at $2.11Gemini 3 Flash · Gemini CLI: 12.5% at $0.33Claude Opus 4.7 · OpenClaw: 17.3% at $11.48Kimi K2.6 · OpenClaw: 10.2% at $0.91DeepSeek V4 Pro · OpenClaw: 11.1% at $0.53GLM-5.1 · OpenClaw: 16.9% at $0.96Qwen 3.6 Max · OpenClaw: 4.9% at $2.80Grok 4.3 · OpenClaw: 0.4% at $2.66Kimi K2.6 · OAI Agents: 15.1% at $0.43Qwen 3.6 Max · OAI Agents: 15.6% at $0.58Grok 4.3 · OAI Agents: 5.8% at $1.54Kimi K2.6 · Hermes: 15.6% at $1.07DeepSeek V4 Pro · Hermes: 13.8% at $2.19GLM-5.1 · Hermes: 18.7% at $1.04Qwen 3.6 Max · Hermes: 16.4% at $4.12Grok 4.3 · Hermes: 4.4% at $1.05Kimi K2.6 · DeepAgents: 3.1% at $0.55GLM-5.1 · DeepAgents: 11.1% at $0.26Qwen 3.6 Max · DeepAgents: 9.3% at $0.57Grok 4.3 · DeepAgents: 2.2% at $1.43Opus 4.7 · OpenClaw, $11.48 for 17.3%Haiku 4.5 (Claude Code): 6.2% at $0.16Haiku 4.5 · Claude CodeDS V4 Pro (DeepAgents): 10.7% at $0.21DeepSeek V4 · DeepAgentsDS V4 Pro (OAI Agents): 14.2% at $0.25DeepSeek V4 · OAI AgentsGLM-5.1 (OAI Agents): 18.7% at $0.27GLM-5.1 · OAI AgentsGPT-5.5 (Codex): 20.9% at $1.29GPT-5.5 · CodexSonnet 4.6 (Claude Code): 26.2% at $1.30Sonnet 4.6 · Claude CodeOpus 4.6 (Claude Code): 28.0% at $6.47Opus 4.6 · Claude Code
Pareto frontier Other configurations (hover for details)
Figure 6. Average cost per trial against pass@1 for all 30 harness × model configurations. Dashed lines split the plane at $1 per trial and 13% pass@1. The rose line is the Pareto frontier: seven configurations where no other point is both cheaper and more accurate.

Where agents break down

A 28% vs. 21% leaderboard gap suggests the leading agents are doing roughly the same thing, only at different success rates. They aren't. We pulled the per-check, per-tool-call data from 225 trials each for the two leading configurations from different labs, Claude Code + Opus 4.6 (28.0% pass@1) and Codex + GPT-5.5 (20.9% pass@1), and the failure shapes are wildly different.

Claude is the careful process-follower that submits packets it shouldn't. It nails every mandatory peer-to-peer (12 of 12) and writes care plans with 100% goal-structure compliance, but fails 0/33 of the “gather more evidence and put the case on hold” tasks because it always submits. Codex is the fast shortcutter: 1.5–2× faster per trial, better at form-filling, but 88% of its outreach calls fail the patient-conversation rubric, and it racked up 122 consecutive retries on a single UM trial when it couldn't format a tool call correctly.

Each agent is good at the things the other one is bad at. Both fail in ways that would cause actual harm in production. Below: their strengths and weaknesses side by side, then their failure rate on each rubric check.

Claude Code + Opus 4.6
The thorough process-follower
28.0%
pass@1
96%
consistent across trials
Strengths
  • Procedural fidelityPayerNever skips a mandatory step; peer-to-peer raised 12 of 12 times
  • Role-switchingPayerKeeps the right authority boundaries across 5 UM roles
Weaknesses
  • Slow execution6.6 min per PA trial and 68 steps on average, vs 3.5 min and 46 for Codex
Codex + GPT-5.5
The efficient shortcutter
20.9%
pass@1
64%
consistent across trials
Strengths
  • Documentation gap detectionProviderPasses 6 of 33 hold tasks, where Claude passes none
  • Intake efficiencyProvider40% pass on structured form-filling tasks
Weaknesses
  • Skips mandatory stepsPayerBypasses the peer-to-peer call when the clinical answer seems obvious
  • Hard-coded triage routingPayerA bug that breaks gold-card auto-approvals
Claude Code + Opus 4.6 Codex + GPT-5.5
  • Rushed to submit?
  • Fabricated paperwork?
  • Wrong case status?
  • Wrong treatment location?
  • Missing decision on file?
Share of trials failing0%25%50%75%100%
Retry attempts (per UM task)
44122
Format errors (worst single task)
0163
Evidence uploaded (per PA task)
3.71
Words generated (thousands)
146.5
Figure 7.Failure rate by rubric check for the two leading agents over 225 trials each; further right means more trials failing. Below, how each handled errors and output.

Cite this

If you reference χ-Bench in a paper, post, or eval, here's the BibTeX. The data, code, and all 75 task definitions are open-sourced at the GitHub repo linked above.

@misc{chen2026chibenchaiagentsautomate,
      title={CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?},
      author={Haolin Chen and Deon Metelski and Leon Qi and Tao Xia and Joonyul Lee and Steve Brown and Kevin Riley and Frank Wang and T. Y. Alvin Liu and Hank Capps MD and Zeyu Tang and Xiangchen Song and Lingjing Kong and Fan Feng and Tianyi Zeng and Zhiwei Liu and Zixian Ma and Hang Jiang and Fangli Geng and Yuan Yuan and Chenyu You and Qingsong Wen and Hua Wei and Yanjie Fu and Yue Zhao and Carl Yang and Biwei Huang and Kun Zhang and Caiming Xiong and Sanmi Koyejo and Eric P. Xing and Philip S. Yu and Weiran Yao},
      year={2026},
      eprint={2605.16679},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.16679},
      note={Accepted to NeurIPS 2026},
}
ModelsCura 1T: Healthcare Foundation Model via Recursive Self-ImprovementLearningRecursive Self-Improvement in AI: A Survey
All research