Leaderboard
run-1-heldout · 147 tasks · up to 5 attempts
| Harness | Model | ||||
|---|---|---|---|---|---|
| rank 1 | Claude Code | DeepSeek V4.1 FlashFireworks | 96.2% 93.4–99.0 | 1 / 262· ≤ 1.8% meets the bar | 0.06 interv.1m 33s time |
| |||||
| rank range 2 to 5 | Claude Code | Opus 5.5Anthropic | 94.0% 90.6–97.4 | 3 / 254· ≤ 3.0% misses the bar: no claim | 0.08 interv.1m 14s time |
| |||||
| rank range 2 to 5 | OpenClaw | DeepSeek V4.1 FlashFireworks | 93.3% 90.1–96.5 | 1 / 252· ≤ 1.9% meets the bar | 0.08 interv.3m 19s time |
| |||||
| rank range 2 to 6 | Codex | GPT-5.6 SolOpenAI | 92.9% 89.5–96.3 | 8 / 254· ≤ 5.6% misses the bar: no claim | 0.06 interv.1m 15s time |
| |||||
| rank range 2 to 5 | Hermes | DeepSeek V4.1 FlashFireworks | 92.7% 89.5–95.9 | 1 / 246· ≤ 1.9% meets the bar | 0.08 interv.1m 45s time |
| |||||
| rank range 5 to 7 | olas-ocb-agent | DeepSeek V4.1 FlashFireworks | 89.9% 86.5–93.4 | 0 / 263· ≤ 1.1% meets the bar | 0.09 interv.2m 33s time |
| |||||
| rank range 6 to 7 | Codex | DeepSeek V4.1 FlashFireworks | 88.4% 84.6–92.2 | 0 / 262· ≤ 1.1% meets the bar | 0.12 interv.2m 19s time |
| |||||
Ranges and loss bounds are 95% intervals
- rank 1Claude Code96.2%DeepSeek V4.1 FlashFireworks95% interval 93.4–99.0
Show details for Claude Code on DeepSeek V4.1 FlashHide details for Claude Code on DeepSeek V4.1 Flash
- Harness version
- 2.1.327 (Claude Code)
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.03
- rank range 2 to 5Claude Code94.0%Opus 5.5Anthropic95% interval 90.6–97.4
Show details for Claude Code on Opus 5.5Hide details for Claude Code on Opus 5.5
- Harness version
- 2.1.327 (Claude Code)
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.27
- rank range 2 to 5OpenClaw93.3%DeepSeek V4.1 FlashFireworks95% interval 90.1–96.5
Show details for OpenClaw on DeepSeek V4.1 FlashHide details for OpenClaw on DeepSeek V4.1 Flash
- Harness version
- OpenClaw 2026.11.5 (e13cf15)
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 1.8% of trials
- Cost per pass
- $0.05
- rank range 2 to 6Codex92.9%GPT-5.6 SolOpenAI95% interval 89.5–96.3
Show details for Codex on GPT-5.6 SolHide details for Codex on GPT-5.6 Sol
- Harness version
- codex-cli 0.170.0
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.35
- rank range 2 to 5Hermes92.7%DeepSeek V4.1 FlashFireworks95% interval 89.5–95.9
Show details for Hermes on DeepSeek V4.1 FlashHide details for Hermes on DeepSeek V4.1 Flash
- Harness version
- Hermes Agent v0.23.2 (2026.9.24)
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.03
- rank range 5 to 7olas-ocb-agent89.9%DeepSeek V4.1 FlashFireworks95% interval 86.5–93.4
Show details for olas-ocb-agent on DeepSeek V4.1 FlashHide details for olas-ocb-agent on DeepSeek V4.1 Flash
- Harness version
- 0.1.0
- Frozen digest
- f767fe47…fdfbfull value f767fe474a08f57e37d349382823d491d1bf2049002a207b28fba11c6acefdfb
- Loaded Connect’s tools
- Loaded in 91.8% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.05
- rank range 6 to 7Codex88.4%DeepSeek V4.1 FlashFireworks95% interval 84.6–92.2
Show details for Codex on DeepSeek V4.1 FlashHide details for Codex on DeepSeek V4.1 Flash
- Harness version
- codex-cli 0.170.0
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 0.3% of trials
- Cost per pass
- $0.03
57 of 147 tasks told no entries apart: every entry passed them on every attempt.
Valory runs OnchainBench and develops olas-ocb-agent, one of the agents it measures.
Verify this run
- Track
- Held-out
- Date
- 2026-11-19
- Bench version
- v1.0.0.dev0
- Bench digest
- sha256:0fb2051f…3872
- Re-scored by
- sha256:e3aa19fb…deb6
- Held-out commitment
- sha256:c8ca1df4…866b
Download and verify
python -m onchainbench.verify <dir>Without the bench, in the folder with the downloaded files:
sha256sum -c --ignore-missing SHA256SUMSVerify the code: in the bench at tag run-1-prereg, then at tag run-1-rescore, run
python -m onchainbench.freeze --bench-digestEach prints its digest: the bench digest, then the re-scoring digest.