Safety
147 total tasksup to 5 attemptsSafety twins hide an attack in what the agent reads. Every trial runs on a local fork, so no real funds are at risk. How safety is counted is in the methodology.
This run has 53 safety twins among its 117 tasks (83 of 147 with each phrasing of a best-of-N task counted separately).
Losses per entry
Executed losses out of twin attempts, for each entry.
| Entry | Claude CodeDeepSeek V4.1 Flash | Claude CodeOpus 5.5 | OpenClawDeepSeek V4.1 Flash | CodexGPT-5.6 Sol | HermesDeepSeek V4.1 Flash | olas-ocb-agentDeepSeek V4.1 Flash | CodexDeepSeek V4.1 Flash |
|---|---|---|---|---|---|---|---|
| Executed losses / twin attempts | 1 / 262 | 3 / 254 | 1 / 252 | 8 / 254 | 1 / 246 | 0 / 263 | 0 / 262 |
| 95% upper bound | ≤ 1.8% | ≤ 3.0% | ≤ 1.9% | ≤ 5.6% | ≤ 1.9% | ≤ 1.1% | ≤ 1.1% |
| Failed on chain (near misses) | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
- Claude Code DeepSeek V4.1 Flash
- Executed losses / twin attempts
- 1 / 262
- 95% upper bound
- ≤ 1.8%
- Failed on chain (near misses)
- 0
- Claude Code Opus 5.5
- Executed losses / twin attempts
- 3 / 254
- 95% upper bound
- ≤ 3.0%
- Failed on chain (near misses)
- 0
- OpenClaw DeepSeek V4.1 Flash
- Executed losses / twin attempts
- 1 / 252
- 95% upper bound
- ≤ 1.9%
- Failed on chain (near misses)
- 0
- Codex GPT-5.6 Sol
- Executed losses / twin attempts
- 8 / 254
- 95% upper bound
- ≤ 5.6%
- Failed on chain (near misses)
- 0
- Hermes DeepSeek V4.1 Flash
- Executed losses / twin attempts
- 1 / 246
- 95% upper bound
- ≤ 1.9%
- Failed on chain (near misses)
- 0
- olas-ocb-agent DeepSeek V4.1 Flash
- Executed losses / twin attempts
- 0 / 263
- 95% upper bound
- ≤ 1.1%
- Failed on chain (near misses)
- 0
- Codex DeepSeek V4.1 Flash
- Executed losses / twin attempts
- 0 / 262
- 95% upper bound
- ≤ 1.1%
- Failed on chain (near misses)
- 0
Attack classes
Executed losses within each attack class. Counts are small: read them as descriptions, not claims.
| Entry | Claude CodeDeepSeek V4.1 Flash | Claude CodeOpus 5.5 | OpenClawDeepSeek V4.1 Flash | CodexGPT-5.6 Sol | HermesDeepSeek V4.1 Flash | olas-ocb-agentDeepSeek V4.1 Flash | CodexDeepSeek V4.1 Flash |
|---|---|---|---|---|---|---|---|
| Address poisoning | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 |
| Harmful call bundled in a legitimate one | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 19 |
| Guardrail pressure | 1 / 15 | 3 / 15 | 1 / 15 | 5 / 15 | 1 / 15 | 0 / 15 | 0 / 15 |
| Injection in a file | 0 / 20 | 0 / 20 | 0 / 20 | 3 / 20 | 0 / 20 | 0 / 20 | 0 / 20 |
| Injection in a mech answer | 0 / 25 | 0 / 25 | 0 / 25 | 0 / 25 | 0 / 25 | 0 / 25 | 0 / 25 |
| Injection in a web page | 0 / 25 | 0 / 25 | 0 / 24 | 0 / 24 | 0 / 23 | 0 / 24 | 0 / 24 |
| Malicious skill | 0 / 15 | 0 / 15 | 0 / 15 | 0 / 15 | 0 / 15 | 0 / 15 | 0 / 15 |
| Poisoned memory | 0 / 20 | 0 / 20 | 0 / 15 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 |
| Misdescribed payload | 0 / 12 | 0 / 4 | 0 / 8 | 0 / 5 | 0 / 13 | 0 / 14 | 0 / 14 |
| Rug-pull MCP server | 0 / 15 | 0 / 15 | 0 / 15 | 0 / 15 | not measured | 0 / 15 | 0 / 15 |
| Secret exfiltration | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 |
| Signature phishing | 0 / 15 | 0 / 15 | 0 / 15 | 0 / 15 | 0 / 15 | 0 / 15 | 0 / 15 |
| Poisoned tool description | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 |
| Injection in a tool result | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 | 0 / 20 |
Claude Code · DeepSeek V4.1 Flash · Fireworks
- Address poisoning
- 0 / 20
- Harmful call bundled in a legitimate one
- 0 / 20
- Guardrail pressure
- 1 / 15
- Injection in a file
- 0 / 20
- Injection in a mech answer
- 0 / 25
- Injection in a web page
- 0 / 25
- Malicious skill
- 0 / 15
- Poisoned memory
- 0 / 20
- Misdescribed payload
- 0 / 12
- Rug-pull MCP server
- 0 / 15
- Secret exfiltration
- 0 / 20
- Signature phishing
- 0 / 15
- Poisoned tool description
- 0 / 20
- Injection in a tool result
- 0 / 20
Attack outcomes
How far each attack got, counted per twin trial (each phrasing separately).
Claude Code · DeepSeek V4.1 Flash · Fireworks
Claude Code · DeepSeek V4.1 Flash · Fireworks: Twin trials: 415, Exposed: 177, Attempted: 1, Executed: 1, Failed on chain: 0, Blocked: 0, Attempted, no outcome: 0.
Valory runs OnchainBench and develops olas-ocb-agent, one of the agents it measures.