Skip to content

Safety

147 total tasksup to 5 attempts

Safety twins hide an attack in what the agent reads. Every trial runs on a local fork, so no real funds are at risk. How safety is counted is in the methodology.

This run has 53 safety twins among its 117 tasks (83 of 147 with each phrasing of a best-of-N task counted separately).

Losses per entry

Executed losses out of twin attempts, for each entry.

Safety per entry
Entry
Claude CodeDeepSeek V4.1 Flash
Claude CodeOpus 5.5
OpenClawDeepSeek V4.1 Flash
CodexGPT-5.6 Sol
HermesDeepSeek V4.1 Flash
olas-ocb-agentDeepSeek V4.1 Flash
CodexDeepSeek V4.1 Flash
Executed losses
/ twin attempts
1 / 2623 / 2541 / 2528 / 2541 / 2460 / 2630 / 262
95% upper bound≤ 1.8%≤ 3.0%≤ 1.9%≤ 5.6%≤ 1.9%≤ 1.1%≤ 1.1%
Failed on chain
(near misses)
0000000
  • Claude Code DeepSeek V4.1 Flash
    Executed losses / twin attempts
    1 / 262
    95% upper bound
    ≤ 1.8%
    Failed on chain (near misses)
    0
  • Claude Code Opus 5.5
    Executed losses / twin attempts
    3 / 254
    95% upper bound
    ≤ 3.0%
    Failed on chain (near misses)
    0
  • OpenClaw DeepSeek V4.1 Flash
    Executed losses / twin attempts
    1 / 252
    95% upper bound
    ≤ 1.9%
    Failed on chain (near misses)
    0
  • Codex GPT-5.6 Sol
    Executed losses / twin attempts
    8 / 254
    95% upper bound
    ≤ 5.6%
    Failed on chain (near misses)
    0
  • Hermes DeepSeek V4.1 Flash
    Executed losses / twin attempts
    1 / 246
    95% upper bound
    ≤ 1.9%
    Failed on chain (near misses)
    0
  • olas-ocb-agent DeepSeek V4.1 Flash
    Executed losses / twin attempts
    0 / 263
    95% upper bound
    ≤ 1.1%
    Failed on chain (near misses)
    0
  • Codex DeepSeek V4.1 Flash
    Executed losses / twin attempts
    0 / 262
    95% upper bound
    ≤ 1.1%
    Failed on chain (near misses)
    0

Attack classes

Executed losses within each attack class. Counts are small: read them as descriptions, not claims.

Executed losses per attack class
Entry
Claude CodeDeepSeek V4.1 Flash
Claude CodeOpus 5.5
OpenClawDeepSeek V4.1 Flash
CodexGPT-5.6 Sol
HermesDeepSeek V4.1 Flash
olas-ocb-agentDeepSeek V4.1 Flash
CodexDeepSeek V4.1 Flash
Address poisoning0 / 200 / 200 / 200 / 200 / 200 / 200 / 20
Harmful call bundled in a legitimate one0 / 200 / 200 / 200 / 200 / 200 / 200 / 19
Guardrail pressure1 / 153 / 151 / 155 / 151 / 150 / 150 / 15
Injection in a file0 / 200 / 200 / 203 / 200 / 200 / 200 / 20
Injection in a mech answer0 / 250 / 250 / 250 / 250 / 250 / 250 / 25
Injection in a web page0 / 250 / 250 / 240 / 240 / 230 / 240 / 24
Malicious skill0 / 150 / 150 / 150 / 150 / 150 / 150 / 15
Poisoned memory0 / 200 / 200 / 150 / 200 / 200 / 200 / 20
Misdescribed payload0 / 120 / 40 / 80 / 50 / 130 / 140 / 14
Rug-pull MCP server0 / 150 / 150 / 150 / 15not measured0 / 150 / 15
Secret exfiltration0 / 200 / 200 / 200 / 200 / 200 / 200 / 20
Signature phishing0 / 150 / 150 / 150 / 150 / 150 / 150 / 15
Poisoned tool description0 / 200 / 200 / 200 / 200 / 200 / 200 / 20
Injection in a tool result0 / 200 / 200 / 200 / 200 / 200 / 200 / 20

Claude Code · DeepSeek V4.1 Flash · Fireworks

Address poisoning
0 / 20
Harmful call bundled in a legitimate one
0 / 20
Guardrail pressure
1 / 15
Injection in a file
0 / 20
Injection in a mech answer
0 / 25
Injection in a web page
0 / 25
Malicious skill
0 / 15
Poisoned memory
0 / 20
Misdescribed payload
0 / 12
Rug-pull MCP server
0 / 15
Secret exfiltration
0 / 20
Signature phishing
0 / 15
Poisoned tool description
0 / 20
Injection in a tool result
0 / 20

Attack outcomes

How far each attack got, counted per twin trial (each phrasing separately).

Claude Code · DeepSeek V4.1 Flash · Fireworks

Claude Code · DeepSeek V4.1 Flash · Fireworks: Twin trials: 415, Exposed: 177, Attempted: 1, Executed: 1, Failed on chain: 0, Blocked: 0, Attempted, no outcome: 0.

Valory runs OnchainBench and develops olas-ocb-agent, one of the agents it measures.