Sign in

Researchχ-Bench

LeaderboardEvery agent setup scored on χ-Bench, ours and the community's.

Submissions run χ-Bench's 75 tasks across prior authorization, utilization management and care management, graded by the two-layer verifier under pass@1 with a binary reward; a run that skipped a domain shows a dash there. The paper explains how a run is graded.

45
Agent setups scored
54.7%
Best overall, erius · claude-opus-5
2026-08-12
Last updated
  1. 1claude-opus-5erius · Michael Johnson (MJ)54.7%
  2. 2claude-opus-4-8erius · Michael Johnson (MJ)37.3%
  3. 3claude-opus-5claude-code37.3%
  4. 4claude-opus-4-8claude-code33.3%
  5. 5claude-opus-4-6claude-code28.0%
  6. 6claude-sonnet-4-6claude-code26.2%
  7. 7gpt-5.6-solcodex25.3%
  8. 8kimi-k3openai-agents25.3%
  9. 9claude-opus-4-7claude-code24.4%
  10. 10claude-fable-5claude-code24.0%
  11. 11MedGuardhermes · cuilinke22.7%
  12. 12gpt-5.5codex20.9%
  13. 13claude-sonnet-5claude-code20.0%
  14. 14glm-5.1openai-agents18.7%
  15. 15glm-5.1hermes18.7%
  16. 16glm-5.2openai-agents18.7%
  17. 17claude-opus-4-7openclaw17.3%
  18. 18glm-5.1openclaw16.9%
  19. 19qwen-3.6-maxhermes16.4%
  20. 20gpt-5.4codex16.0%
  21. 21qwen-3.6-maxopenai-agents15.6%
  22. 22kimi-k2.6hermes15.6%
  23. 23kimi-k2.6openai-agents15.1%
  24. 24deepseek-v4-proopenai-agents14.2%
  25. 25deepseek-v4-prohermes13.8%
  26. 26gpt-5.6-terracodex13.3%
  27. 27gpt-5.6-lunacodex13.3%
  28. 28gemini-3-flashgemini-cli12.5%
  29. 29deepseek-v4-proopenclaw11.1%
  30. 30glm-5.1deepagents11.1%
  31. 31deepseek-v4-prodeepagents10.7%
  32. 32kimi-k2.6openclaw10.2%
  33. 33qwen-3.6-maxdeepagents9.3%
  34. 34gpt-5.4-minicodex8.4%
  35. 35TML Inkling 256Kopenai-agents8.0%
  36. 36gemini-3.1-progemini-cli7.1%
  37. 37claude-haiku-4-5claude-code6.2%
  38. 38grok-4.3openai-agents5.8%
  39. 39qwen-3.6-maxopenclaw4.9%
  40. 40grok-4.3hermes4.4%
  41. 41kimi-k2.6deepagents3.1%
  42. 42grok-4.3deepagents2.2%
  43. 43grok-4.3openclaw0.4%
  44. 44Nemotron 3 Ultra 256Kopenai-agents0.0%
  45. 45gpt-5.5pa-codex-agent · MedArise–

Ranked by pass@1 on all domains. A setup that did not run a domain shows a dash and sorts last. The arrow opens each run's submission packet.

Have an agent to put on the board?
Run the suite with your harness and model, package the results with cb submission prepare, and open a pull request.
How to submit a run