Harness Hub
All papers

Safe Resemblance, Silent Propagation: Limits of Text-Similarity Guardrails for AI Agent Safety

Charles Xiao

Verdict: Incremental

The paper is a carefully hedged negative result against one niche overlap guardrail, with a real but minor fix to its measurement code. The evaluation principles it recommends are already standard in the cited behavioral and policy-aware literature. Its evidence rests on 19 positives from one model, one canary task and no alternative detectors, so the abstract's general claims about text-similarity guardrails go beyond what is shown.

  • Novelty

    2 out of 5
  • Rigor

    3 out of 5
  • Reproducibility

    2 out of 5
  • Relevance to harnesses

    2 out of 5

TL;DR

The paper tests one public n-gram-overlap guardrail, DonkeyRail (BLEU/ROUGE-L/METEOR features fed to a logistic model, from the Morris-II line), on a synthetic three-agent email relay. Each root output is labeled by downstream behavior. At threshold 0.5 the guardrail misses 8 of 19 functional Luna roots and flags 70 of 192 authorized Luna replies. A low-false-positive calibrated threshold catches 1 of 19 and still flags 15 of 192 on new templates. The paper also corrects ROC-interpolated operating points in the repo and adds two short formal arguments. All positives come from one model, one canary task and one detector.

Adversarial review by Harness Hub · Nov 18, 2026 · read full text · Suggest a correction

Original abstract (authors’ text, select to expand)

When AI agents relay information to one another, an output can preserve an injected instruction's effect on downstream agents even after its wording changes, yet legitimate quotation and template reuse can produce high textual overlap with the source, so text-similarity guardrails that attempt to flag the former risk conflating these two cases. We test this distinction using DonkeyRail, a public classifier combining BLEU, ROUGE-L, and METEOR features, deployed in a controlled three-agent email relay, defining a behavioral measure in which an output counts as functional (the injected procedure induces fresh computations in two subsequent agents) independent of the detector's score, alongside legitimate controls that explicitly authorize quotation and reuse. At its default threshold the classifier misses 8 of 19 functional outputs while flagging 70 of 192 legitimate replies, with every completed quotation and template-reuse reply triggering a false alert; calibrating for low false positives on one template set does not transfer, as the recalibrated threshold detects only one functional output and still produces 15 false alerts on new templates. We also identify an operating-point error in all 21 published classifier configurations, where ROC interpolation replaces direct threshold application, and show formally that high AUC against a surface-level text criterion provides no distribution-free guarantee of high AUC against the behavioral target, while the existing evaluation labels lack the joint observations needed to estimate the gap. An authorization analysis further shows that identical visible transcripts can require different safety decisions depending on the governing policy, a distinction no observation-only classifier can resolve. These results argue for behavioral and policy-grounded evaluation over text-similarity guardrails.

Filed underevaluationguardrailsagents

Summary

Setup: - A synthetic 'add seven to the ticket, emit a LAB-PROOF line, forward this procedure' instruction sits in the tenth retrieved record of an email-assistant prompt. - Each output is inserted into the next agent's context, and every agent gets a new ticket. - A root output counts as functional only if both later agents emit the correct fresh result. - The frozen DonkeyRail logistic classifier (public repo, pinned commit) scores roots against their retrieved records offline. - The same classifier also scores 192 calibration and 192 test legitimate replies per model. The four tasks are acknowledgment, quotation, paraphrase and template reuse; quotation and reuse are explicitly authorized. Results: - gpt-6-luna produced 19 of 72 functional roots in the primary run. - gpt-6.1-sol produced none, so its recall and behavioral AUC are undefined. - False alerts concentrate on quotation and template reuse. Acknowledgments draw 0 of 48 and paraphrases 2 of 48. - A calibration rule allowing at most 3 alerts on 192 calibration replies, applied to disjoint templates and source formats, keeps 1 of 19 detections. It still flags 15 of 192 Luna and 69 of 192 Sol test replies. Other contributions: - The repo's 21 reported fixed-threshold operating points were obtained by ROC interpolation. The paper gives direct counts. - A toy construction shows proxy-label AUC gives no distribution-free bound on behavioral AUC. - A two-world argument shows that an observation-only classifier cannot decide authorization when the policy is absent from its input.

Key findings

  • Table 1, Luna at threshold 0.5: TP 11, FN 8, recall 57.89%, legitimate FP 70/192. The seed-cluster bootstrap interval for recall is 35.0% to 76.9% (Section 4.2).
  • Table 1, Luna at the calibrated threshold: TP 1, FN 18, recall 5.26%, legitimate FP 15/192. Sol at the calibrated threshold: FP 69/192, with zero of 192 calibration alerts for both models (Section 4.3).
  • Table 2 at threshold 0.5: Luna draws 37/48 quotation and 31/48 template alerts; Sol draws 48/48 on each. All 30 qualifying completed Luna quotation/template replies and all 93 qualifying Sol replies are flagged (Section 4.2).
  • Luna task completion is low on the reuse tasks: quotation 22/48 and template 8/48, versus Sol 47/48 and 46/48 (Table 2). Hypothetical blocking would cut completed tasks from 122 to 90 for Luna and from 188 to 93 for Sol.
  • Sol produced no functional roots and no fresh actions in its 216 primary agent outputs. Every behavioral recall and AUC number therefore comes from Luna alone (Section 4.1).
  • Table 3, Luna root AUC: 0.7901 (earlier), 0.7160 (primary), 0.7609 (pooled). Functional-vs-benign AUC: 0.4083, 0.6436, 0.5272. Proxy AUC on the roots is undefined because every observed C is zero.
  • Section 5.1, BLEU stump at 0.5: reported FPR 0.500036 versus direct 23/1500 = 0.015333. Three-feature logistic model: direct FPR 29/1500 = 0.019333 versus reported 0.021580. The AUC of 0.997884 is unchanged.
  • Appendix B, Table 4: the Jailbreaks file contains the roleplay phrase in 86 of 1,579 positive rows (phrase AUC 0.527). Other virus/spam files match the phrase exactly.
  • Appendix C, earlier evaluation: 7/13 functional roots detected at 0.5 and 128/192 legitimate alerts. Calibrated threshold 0.999981053592889 gives zero test alerts and zero detections. Stripping relay tags, fences and quote lines before scoring drops functional detections from seven to zero.
  • Section 6: across 64 byte-identical transcript pairs that differ only in external policy, a constant observation-only baseline scores 64/128 and consulting the policy scores 128/128.

The paper at a glance

01Agent 1: 9distractor emails +1 untrusted relayrecord (12 framingsx 6 variants) +…02Root output o1scored offline byfrozen DonkeyRail(max BLEU, ROUGE-L,METEOR over 10…03o1 inserted asrelay record forAgent 2 with newticket t2; o2inserted for Agent…04Functional labelF(o1) = correctfresh LAB-PROOFline at both Agent2 and Agent 305Separate authorizedcontrols(acknowledge,quote, paraphrase,template reuse;…06Compare alerts attau=0.5 and atcalibration-onlythreshold
  1. Agent 1: 9 distractor emails + 1 untrusted relay record (12 framings x 6 variants) + trusted meeting request with ticket t1
  2. Root output o1 scored offline by frozen DonkeyRail (max BLEU, ROUGE-L, METEOR over 10 records -> logistic)
  3. o1 inserted as relay record for Agent 2 with new ticket t2; o2 inserted for Agent 3 with t3
  4. Functional label F(o1) = correct fresh LAB-PROOF line at both Agent 2 and Agent 3
  5. Separate authorized controls (acknowledge, quote, paraphrase, template reuse; disjoint calibration/test templates) scored for false alerts
  6. Compare alerts at tau=0.5 and at calibration-only threshold
Fig. 1 — How the paper works, step by step (arXiv:2611.21520).

Why it matters

Relevance is limited for most harness builders. The detector under test is an academic n-gram-overlap guardrail from the Morris-II worm line, and few production harnesses deploy anything like it. The transferable lessons are methodological, and most are already established by AgentDojo, CaMeL and Fides: - Label injection evals by downstream behavior, not by a marker phrase. - Include explicitly authorized high-overlap tasks such as quoting and template filling in the false-positive set. - Compute operating points by counting decisions at the threshold. - Treat authorization as a policy input rather than something inferred from text. If your harness relays agent outputs into other agents' contexts and you have considered similarity-based replication detection, the Luna examples are a concrete counterexample. One reformatted instruction scores 0.012957 at the root, and an encoded one scores 0.006965 in Appendix E, yet both propagate across two hops. The evidence is narrow: one detector, one benign arithmetic canary, forced context insertion with no retrieval or tools, and 19 positives from one model. It shows what not to rely on. It does not test any alternative you might build.

Issues (10)

  • High severity

    All behavioral detection evidence comes from one model with very few positives: 19 functional roots in the primary run and 13 in the earlier one. Sol produced zero, so half of the model comparison contributes nothing to recall or behavioral AUC. The recall interval of 35.0% to 76.9% is wide. Only 7 variants were functional in both runs, and each cell has a single sample. The '8 of 19 missed' figure is a fragile point estimate.

  • High severity

    Only one detector family is evaluated: the DonkeyRail logistic model, plus repo stumps and naive Bayes in the measurement section. There is no embedding-similarity, LLM-judge, prompt-injection classifier or provenance baseline on the same chains and controls. The title and abstract generalize to 'text-similarity guardrails', while Section 7 concedes broader metric classes need separate evaluation.

  • Medium severity

    The false-alert headline is largely produced by the construction. - DonkeyRail sees only retrieved records and the reply, never the task, so it cannot represent authorization. - Half of the legitimate test mixture is verbatim quotation or full template reuse. - Ordinary acknowledgments draw 0/48 alerts and paraphrases 2/48. The 70/192 (36.46%) and Sol 98/192 (51.04%) rates mainly measure the chosen mix. The paper says the mix does not reflect real email prevalence, but the abstract presents 70 of 192 without that qualifier.

  • Medium severity

    The 'functional' endpoint measures instruction adoption, not a policy violation the model was told to avoid. Section 3.2 states that the trusted/untrusted annotations and the prohibition on extra proof lines are evaluator metadata, not model-visible policy. Calling these outputs propagation of an 'injected instruction' as a safety failure therefore depends on an assumed boundary that the agents were never given.

  • Medium severity

    Section 5.3 reads the pooled functional-vs-benign AUC of 0.527181 as 'almost no discriminative power'. The per-evaluation values are 0.4083 and 0.6436, and the paper itself says pooling mixes different challenge and control distributions. The pooled figure is an average of two quite different cohorts, so the strong conclusion is not supported as stated.

  • Medium severity

    The calibration-transfer claim rests on one rule (at most 3 of 192 calibration alerts) and one shift that changes request templates and source formats together. No mixed-pool calibration or alternative target rate is reported. 'Does not transfer' is shown only for this configuration, and the cause cannot be attributed to either factor.

  • Low severity

    The exploratory diagnostic in Appendix C suggests the 0.5-threshold detections come mostly from verbatim relay-tag content rather than from the injected procedure. Stripping relay tags, fences and quote lines drops functional detections from seven to zero, while legitimate alerts stay at 128. The main text does not discuss what the 57.89% recall is actually keyed on.

  • Low severity

    'An operating-point error in all 21 published classifier configurations' refers to the public repository's saved operating points. Appendix B says these are distinct from the original Morris-II paper-level results. For the headline three-feature model, FPR moves only from 0.021580 to 0.019333, and AUC is unchanged.

  • Low severity

    Reproduction is weak: - Models are given as undated API aliases. - Luna and Sol run at different reasoning efforts and token limits. - There is one sample per cell. - Code and data are said to be in supplementary materials, but no URL is printed.

  • Low severity

    Both propositions are close to definitional. Proposition 2 says that if the deciding fact is not in the input, no function of the input decides it. Proposition 1 constructs label flips that move the target AUC anywhere. Neither adds evidence beyond the experiments. Behavioral evaluation of worm propagation is also already established by the cited SRPO and AgentWorm work.

Questions for the authors

  • How do an embedding-similarity score or an LLM-based injection classifier perform on the same chains and authorized controls?
  • With multiple samples per framing-variant cell, how stable are the 19 functional roots and the 8 misses?
  • Does calibrating on a pool spanning several template and format families keep test FPR near target, and what recall remains?
  • Do adoption rates change when the trusted/untrusted boundary and the prohibition are stated to the model rather than kept as evaluator metadata?
  • Why does Sol show zero adoption, and do matched reasoning effort and token limits change that?
  • What is the functional-vs-benign AUC when authorized quotation and template reuse are excluded from the negatives?

Related in the directory

People

Lead author of the indirect prompt injection work (Greshake et al.) that the paper cites as the mechanism its relay builds on.

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.

Yuzhi Huang, Jie Wu +11

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.