Leaderboard
run-1-open · 73 tasks · up to 2 attempts
| Harness | Model | ||||
|---|---|---|---|---|---|
| rank range 1 to 6 | Claude Code | Opus 5.5Anthropic | 98.0% 95.5–100.0 | 0 / 16· ≤ 17.1% meets the bar | 0.02 interv.54s time |
| |||||
| rank range 1 to 6 | Claude Code | DeepSeek V4.1 FlashFireworks | 97.3% 94.7–99.8 | 0 / 16· ≤ 17.1% meets the bar | 0.01 interv.1m 26s time |
| |||||
| rank range 1 to 6 | OpenClaw | DeepSeek V4.1 FlashFireworks | 97.3% 94.5–100.0 | 0 / 18· ≤ 15.3% meets the bar | 0.01 interv.2m 38s time |
| |||||
| rank range 1 to 7 | Codex | DeepSeek V4.1 FlashFireworks | 96.6% 93.2–100.0 | 0 / 18· ≤ 15.3% meets the bar | 0.04 interv.1m 54s time |
| |||||
| rank range 1 to 7 | Hermes | DeepSeek V4.1 FlashFireworks | 95.9% 92.5–99.3 | 0 / 14· ≤ 19.3% meets the bar | 0.01 interv.1m 41s time |
| |||||
| rank range 1 to 7 | Codex | GPT-5.6 SolOpenAI | 93.8% 88.5–99.2 | 2 / 18· ≤ 31.0% misses the bar: no claim | 0.00 interv.53s time |
| |||||
| rank range 4 to 7 | olas-ocb-agent | DeepSeek V4.1 FlashFireworks | 91.1% 86.5–95.7 | 0 / 16· ≤ 17.1% meets the bar | 0.02 interv.1m 56s time |
| |||||
Ranges and loss bounds are 95% intervals
- rank range 1 to 6Claude Code98.0%Opus 5.5Anthropic95% interval 95.5–100.0
Show details for Claude Code on Opus 5.5Hide details for Claude Code on Opus 5.5
- Harness version
- 2.1.327 (Claude Code)
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.22
- rank range 1 to 6Claude Code97.3%DeepSeek V4.1 FlashFireworks95% interval 94.7–99.8
Show details for Claude Code on DeepSeek V4.1 FlashHide details for Claude Code on DeepSeek V4.1 Flash
- Harness version
- 2.1.327 (Claude Code)
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.04
- rank range 1 to 6OpenClaw97.3%DeepSeek V4.1 FlashFireworks95% interval 94.5–100.0
Show details for OpenClaw on DeepSeek V4.1 FlashHide details for OpenClaw on DeepSeek V4.1 Flash
- Harness version
- OpenClaw 2026.11.5 (e13cf15)
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.05
- rank range 1 to 7Codex96.6%DeepSeek V4.1 FlashFireworks95% interval 93.2–100.0
Show details for Codex on DeepSeek V4.1 FlashHide details for Codex on DeepSeek V4.1 Flash
- Harness version
- codex-cli 0.170.0
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.03
- rank range 1 to 7Hermes95.9%DeepSeek V4.1 FlashFireworks95% interval 92.5–99.3
Show details for Hermes on DeepSeek V4.1 FlashHide details for Hermes on DeepSeek V4.1 Flash
- Harness version
- Hermes Agent v0.23.2 (2026.9.24)
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.03
- rank range 1 to 7Codex93.8%GPT-5.6 SolOpenAI95% interval 88.5–99.2
Show details for Codex on GPT-5.6 SolHide details for Codex on GPT-5.6 Sol
- Harness version
- codex-cli 0.170.0
- Frozen digest
- none published
- Loaded Connect’s tools
- Loaded in 100% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.26
- rank range 4 to 7olas-ocb-agent91.1%DeepSeek V4.1 FlashFireworks95% interval 86.5–95.7
Show details for olas-ocb-agent on DeepSeek V4.1 FlashHide details for olas-ocb-agent on DeepSeek V4.1 Flash
- Harness version
- 0.1.0
- Frozen digest
- f767fe47…fdfbfull value f767fe474a08f57e37d349382823d491d1bf2049002a207b28fba11c6acefdfb
- Loaded Connect’s tools
- Loaded in 98.6% of trials
- Infrastructure rate
- 0% of trials
- Cost per pass
- $0.03
46 of 73 tasks told no entries apart: every entry passed them on every attempt.
Show the 46 tasks
- a-aave-borrow-repay
- a-aave-supply-withdraw
- a-corrected-retry
- a-partial-batch-report
- a-pay-invoice
- a-policy-review
- a-sdai-half
- a-uni-swap-cap
- a-wrap-unwrap
- b-redeem
- c-incident-summary
- c-lending-compare
- c-token-diligence
- d-forecast-log
- d-payments-lost-response
- d-savings-share
- e-ambiguous-recipient
- e-amount-change
- e-should-decline
- f-guardrail-pressure
- f-invoice-injection
- f-page-token
- f-rug-pull
- f-tool-poisoning
- g-honeyswap
- g-oddswap
- g-sushi-v2
- g-vault-roundtrip-refuse
- x-arbitrum-pay
- x-arbitrum-swap
- x-arbitrum-wrap
- x-base-aave
- x-base-pay
- x-base-swap
- x-base-wrap
- x-celo-aave
- x-celo-pay
- x-ethereum-aave
- x-ethereum-pay
- x-ethereum-swap
- x-ethereum-wrap
- x-mode-pay
- x-optimism-pay
- x-optimism-wrap
- x-robinhood-native-pay
- x-robinhood-wrap
Valory runs OnchainBench and develops olas-ocb-agent, one of the agents it measures.
Verify this run
- Track
- Open track
- Date
- 2026-11-18
- Bench version
- v1.0.0.dev0
- Bench digest
- sha256:0fb2051f…3872
- Re-scored by
- sha256:e3aa19fb…deb6
Download and verify
python -m onchainbench.verify <dir>Without the bench, in the folder with the downloaded files:
sha256sum -c --ignore-missing SHA256SUMSVerify the code: in the bench at tag run-1-prereg, then at tag run-1-rescore, run
python -m onchainbench.freeze --bench-digestEach prints its digest: the bench digest, then the re-scoring digest.