---
title: "StealthBench and the 53.8% Ceiling: What a Safe-Success Rate Measures"
description: "StealthBench scores autonomous security agents on task success and OPSEC together, across 14 containerised scenarios. Across 771 trajectories and 8 models, none exceeded a 53.8% safe-success rate. A preview of the Day 1 talk, and what that figure does and does not say."
author: "Mara Voss"
category: "AI Security Conferences"
date: 2026-10-17T05:25:00.326Z
canonical: "https://agenticcyber.co/blog/unprompted-ii-stealthbench-safe-success-rate"
---

# StealthBench and the 53.8% Ceiling: What a Safe-Success Rate Measures

![Photograph illustrating server rack dark.](https://hsppuvezyxmkpzkgfkho.supabase.co/storage/v1/object/public/media/conference/5bc07ae2-9ee0-46b9-9820-b0704936f742/unprompted-ii-stealthbench-safe-success-rate.jpg)

> StealthBench scores autonomous security agents on task success and OPSEC together, across 14 containerised scenarios. Across 771 trajectories and 8 models, none exceeded a 53.8% safe-success rate. A preview of the Day 1 talk, and what that figure does and does not say.

An autonomous security agent that completes its objective has not necessarily done the job. It may have finished loudly, leaving a trail any competent defender picks up on the first pass. That gap is what **StealthBench** measures, and it arrives at **[un]prompted II** — most people type it *unprompted* — on the first morning. The figure it brings is a ceiling, not an average: across 771 trajectories and 8 models, no model exceeded a **53.8% safe-success rate**. Our [guide to the whole conference](/blog/unprompted-ii-conference-guide) covers the rest of the programme.

The talk is *StealthBench: Can Autonomous Security Agents Do OPSEC Like An Experienced Operator?*, from **Adrian Wood** (Security Engineer, Dropbox) and **Ads Dawson** (Staff AI Security Researcher, Dreadnode; founder, OWASP GenAI Security Project), Day 1, Stage 1, 11:35, per the [official agenda as published on 2026-10-12](https://unpromptedcon.org/agenda-10-2026/). We wrote this brief from that published abstract, before the talk. We are independent coverage — not the organiser, not a sponsor, not a media partner — and we are following the conference remotely, not from the room. Nothing here is a quote. Agendas also change.

## What the unprompted abstract says it will show

An open-source benchmark that scores agents on two axes at once: task success **and** operational security. The environment is 14 containerised scenarios. The run covers 771 trajectories across 8 models, and no model exceeded a 53.8% safe-success rate. The abstract adds the detail that matters more than the number: some highly capable models finished the task while acting recklessly. Capability and discipline are reported as separable, so a model cannot buy a good score with capability alone.

## Why this is the hard part

Our analysis, not the speakers'. A benchmark that scores one axis gets optimised toward whatever is cheapest, and task success is cheap: try everything, retry loudly, escalate early. An agent rewarded only for finishing learns the behaviour a human operator is trained out of in their first year, and nothing flags it, because the pipeline recorded a success.

Scoring OPSEC jointly makes the metric honest and much harder to build. OPSEC is not a property of an action; it is a property of an action in a context, judged against what a defender there would notice. A benchmark must encode that judgement as rules, so 53.8% is measured against one explicit model of a watching defender. The code being open is a strength; it still bounds what the number means.

53.8% is53.8% is not

The best rate any of 8 models reached across 771 trajectories in 14 containerised scenariosThe probability that an agent attacking you gets caught
A joint score: the task finished *and* the agent stayed inside the benchmark's OPSEC rulesA task-success rate — an agent can finish far more often and still score 53.8% here
A ceiling across the models and harness tested, at that point in timeA statement about agents in your estate, with your logging gaps and your alert thresholds

This is the scoring problem we keep hitting when we write about [red-teaming an agentic system](/blog/how-to-red-team-an-agentic-system-practitioners-methodology): the trajectory is the artefact, not the result, and most teams keep only the result. It also sits inside the wider argument at this event about whether benchmark scores mean what teams think, which we traced in [the six arguments running through [un]prompted II](/blog/unprompted-ii-six-arguments).

## What we will be watching for

- **Whether the OPSEC rubric ships with the code.** An open benchmark with implicit scoring rules is a leaderboard, not an instrument. The rubric is the part a defender can argue with and adapt.

- **Where the failures sit in the trajectory.** Our guess is post-objective: skipped cleanup, exfiltration path, the agent narrating itself in logs. Front-loaded failures point at different controls, and at how much of the score belongs to the harness rather than the weights.

- **Whether safe-success correlates negatively with capability, or not at all.** “Capable models act recklessly” can mean a trade-off or mean independence, and the two have opposite consequences for model selection.

## If this holds, what changes on Monday

- **Add a noise budget to your agent's own evaluation.** Count actions, credentials touched and alerts raised per completed task. A success at ten times the noise is a finding about your agent, not your target.

- **Keep trajectories, not verdicts.** OPSEC cannot be scored from a result row — the practical end of our argument on [audit log evidence requirements](/blog/agentic-ai-audit-log-evidence-requirements), and the reason we favour [runtime monitoring](/blog/runtime-monitoring-for-ai-agents) over outcome logging.

- **Treat capable-but-reckless as the default case.** The strongest model available is the one most likely to finish the job in a way you would not have authorised.

## Three questions we would ask Adrian Wood and Ads Dawson

- Across the 771 trajectories, what is the most common OPSEC failure *after* the objective already succeeded — skipped cleanup, or a path chosen at the start that was never going to be quiet?

- Do safe-success and raw capability trade off in your data, or are they independent axes? “Which model do I deploy” has a different answer in each case.

- What is the one decision you would support with the 53.8% figure, and the one you would not?

Those are the questions the abstract leaves open. We will update this page if the agenda or the published benchmark changes what is known.

## FAQ

### What exactly is a safe-success rate?

In StealthBench, as described in the published abstract, it is a joint score: the agent completed the task and stayed within the benchmark's operational-security expectations across 14 containerised scenarios. A model can have a high task-success rate and a low safe-success rate. The reported ceiling across 8 models and 771 trajectories is 53.8%.

### Does 53.8% mean autonomous attacks fail about half the time?

No. It is the fraction of runs that both succeeded and stayed quiet by the benchmark's rules, in containers. It says nothing about detection rates in a real estate, and it is not a task-success figure.

### Is this coverage from inside the conference?

No. We are independent coverage, following remotely. Every session fact comes from the official agenda as published on 2026-10-12, and the talk has not happened yet. Adrian Wood also speaks on Day 2 at 15:55 on robot security, listed there as Red Team, Dropbox.


---
Source: https://agenticcyber.co/blog/unprompted-ii-stealthbench-safe-success-rate