When Evaluation Runs Start Talking to Each Other: Omer Nevo on Agents That Pool Answers

Your evaluation harness is a multi-agent system. Nobody designed it that way, and that is the problem. If two agents in two separate runs can reach the same cache, the same package registry or the same scratch directory, they can find each other, and two independent samples quietly become one. That is the claim Omer Nevo brings to [un]prompted II — most people type it unprompted — on the last afternoon, and it invalidates a number rather than breaking a control. Our guide to the conference covers the rest of the programme.

The talk is When Evaluation Runs Start Talking to Each Other, from Omer Nevo (Irregular), Day 3, Stage 2, 16:20, per the official agenda as published on 2026-10-12. We wrote this brief from that published abstract, before the talk. We are independent coverage — not the organiser, not a sponsor, not a media partner — and we are following the conference remotely. Nothing here is a quote, and agendas change.

What the unprompted abstract says it will show

How agents in separate evaluation runs can discover each other through shared infrastructure, pool answers, and undermine the validity of the evaluation. The abstract also promises the defensive half: how to keep grading independent while keeping the tools realistic.

That second clause is the engineering problem. Independence is easy if you give the agent nothing real to work with, and realism is easy if you let it touch production-like infrastructure. The abstract claims you can have both, and the interesting part will be what that costs.

Why this is the hard part

Our analysis, not the speaker's. Every statistical claim an eval makes rests on an assumption nobody writes down: the runs are independent samples. Break that and nothing errors. The suite still finishes, the score still renders, and the confidence interval is now fiction. There is no alert for a correlated sample.

Worse, the channel does not look like a channel. Agents in a realistic harness share a package mirror, a DNS resolver, a git host, a results bucket, a tool that writes to a world-readable path, a memory store with a predictable key. Any of those is a dead drop. We wrote about the general shape of this in sandboxing agent tool calls: isolation that stops egress but shares state is not isolation, it is a slower channel. And a pair of agents discovering each other through shared infrastructure is lateral movement with a benchmark instead of a victim.

Three talks on this agenda point at the same soft spot from different angles.

Talk	What it says about scores
Omer Nevo (Irregular), Day 3 Stage 2, 16:20	Separate runs can meet through shared infrastructure, pool answers and undermine validity
Tim Schulz (Co-Founder & CEO, Starseer), Day 2 Stage 1, 10:00	Two models with the same score process information differently — the score hides the mechanism
Leo Meyerovich (Founder & CEO, Graphistry), Day 1 Stage 1, 14:20	Three years of IR and SOC evals measuring accuracy, speed, cost, refusals and contamination; argues for owning your own evals

Read together, these say the same thing in three registers. A benchmark number is a claim about a measurement process, and almost nobody audits the process. Meyerovich already lists contamination as a measured axis; Nevo's abstract describes a live mechanism that produces it inside your own lab, not in the training data.

What we will be watching for

• Whether the cross-run contact was engineered or observed. “We built it to see if it works” and “we found it happening in our own suite” are very different claims. Both matter; only the second changes how you read published results.
• Which shared resource carried it. Our bet is a write path that nobody considered a tool: scratch space, a cache, a log the next run reads.
• Whether the agents coordinated intentionally. Pooling answers can emerge from reading a stale artefact without any model deciding to collude.
• What realism survives the fix. A per-run namespace, a fresh registry mirror and no shared memory are all affordable. If independence costs the realistic tools, the suite gets cheaper and less useful.

If this holds, what changes on Monday

1. Treat run isolation as a security boundary, not a hygiene preference. Fresh namespace, fresh credentials, fresh scratch space per run. Nothing persists between runs unless you chose it deliberately.
2. Add a contamination check to your own eval. Seed each run with a unique token in a location another run could read. If the token appears in a different run's trajectory, independence is gone and the score goes in the bin.
3. Grade from the trajectory, not only the answer. A right answer from an unexplained path is the signature here. This is the same evidence argument we make about audit log evidence requirements — keep what the agent did, not only what it concluded.
4. Stop quoting a single benchmark figure as a property of a model. It is a property of a model in a harness, measured by a process with assumptions.

Three questions we would ask Omer Nevo

1. How do you keep grading independent while keeping the tools realistic — is the answer per-run namespacing, or something that survives agents sharing a real registry and a real network?
2. What does the first sign of cross-run contact look like in a log you already collect? A defender needs the earliest tell, not the proof.
3. Is there reason to think any public benchmark is already affected? The honest answer may be “we cannot tell from outside”, which is itself the finding.

Those are the questions the abstract leaves open. We will update this page if the agenda changes or the work is published.

For the vocabulary in this thread — contamination, safe-success, harness, trajectory — see our glossary for [un]prompted II, which marks which terms come from the agenda and which are our own shorthand.