PaperarXiv:2611.21259 · cs.MA · Nov 16, 2026
Evaluating Authorization and Trust Boundaries in Agentic Orchestration Frameworks
Sahaya Jestus Lazer, Kshitiz Aryal, Maanak Gupta, Elisa Bertino
The paper catalogues known agent trust-boundary failures across four frameworks. Because most outcomes come from an unstated number of runs, depend on checks and tools the authors set up, and protections are never tested against adaptive attacks, it supports design guidance but not conclusions about the frameworks.
Novelty
2 out of 5Rigor
2 out of 5Reproducibility
2 out of 5Relevance to harnesses
3 out of 5
TL;DR
The authors ran twelve attack experiments, mapped to the OWASP Agentic Top 10, against LangChain, LangGraph, AutoGen and OpenClaw, all using gpt-4o-mini. Most categorical outcomes are identical across the four frameworks. The protected outcomes depend on checks the authors configured or wrote. The results describe the authors' test setups more than the frameworks themselves.
Adversarial review by Harness Hub · Nov 19, 2026 · read full text · Suggest a correction
Original abstract (authors’ text, select to expand)
Agentic AI systems extend large language models with tools, memory, and multi-agent collaboration, introducing security risks at orchestration boundaries. These risks depend on how identities are represented, operations are authorized, and instructions or information propagate between components. We evaluate LangChain, LangGraph, AutoGen, and OpenClaw through twelve experiments mapped to the OWASP Top 10 for Agentic Applications using a common LLM (gpt-4o-mini). The experiments cover prompt injection, unauthorized resource access and command execution, memory poisoning, and inter-agent trust and identity attacks. Adversarial inputs induced unauthorized tool invocation, while tool chaining, stale filesystem validation, encoded instructions, and inter-agent delegation exposed weaknesses in authorization across component boundaries. Memory poisoning showed that untrusted state can influence responses, a risk distinct from conventional access-control violations. In the evaluated protected configurations, explicit authorization, controlled capability assignment, filesystem containment, and execution safeguards prevented specific unauthorized operations. The findings emphasize preserving authorization and information-trust boundaries throughout agentic workflows rather than treating framework capabilities or model-level instructions as sufficient security guarantees.
Filed underorchestrationmulti-agentmemoryagentsllm
Summary
The paper builds twelve attack scenarios: - direct and indirect prompt injection - tool chaining - shell execution - encoded payload transfer - symlink TOCTOU - manifest path traversal - delegated reverse shell - session and persistent memory poisoning - inter-agent message injection - agent identity spoofing Each scenario is implemented separately in four orchestration frameworks, all running gpt-4o-mini. Ten experiments report one categorical outcome per framework (safe, vulnerable, blocked, and similar labels). Only the two memory-poisoning experiments (E9, E10) report rates as mean ± standard deviation over three seeds. Main observations: - Delegation/audit framing (Variant D) caused unauthorized tool calls in three of four frameworks. - Tool chaining and symlink TOCTOU succeeded in every framework. - Path traversal was blocked in every framework. - Base64 payloads forwarded from a restricted decoder to a privileged executor led to file creation. - Explicit authorization stages in the protected configurations stopped inter-agent delegation and identity spoofing. The Limitations section states that protected configurations included application-defined checks. The results therefore describe those configurations, not intrinsic framework guarantees.
Key findings
- Prompt injection (Table 3): Variants A-C were S for all four frameworks. Variant D (delegation/audit framing) was U/- (unauthorized tool invocation, no secret disclosure) for LangChain, LangGraph and OpenClaw, and S for AutoGen. E1 and E2 produced identical outcomes.
- Resource access (Table 4): E3 tool chaining and E6 symlink TOCTOU were V for all four frameworks. E7 path traversal was S for all four. In E7, three frameworks used canonical path checks and OpenClaw relied on sandbox isolation.
- Command execution (Table 5): In E4, LangChain was A/I (it adopted the instruction, but a configured approval interrupted it) and the other three were R. In E5 (encoded payload), LangChain was P because a decoding error altered the filename, and the other three were Y. E8 (delegated command) was B for all four, blocked by reverse-shell detection at the execution interface after the command had crossed the planner-executor boundary.
- Session memory poisoning (Table 6, E9 ASR): LangChain 78.0 ± 2.2, LangGraph 92.6 ± 0.4, AutoGen 92.8 ± 1.1, OpenClaw 66.9 ± 2.4. OpenClaw also had the lowest E9 clean accuracy, 61.3 ± 2.6. No framework reproduced the false claim in the no-injection control.
- Persistent memory poisoning (Section 3.4, Table 6): every framework had a poison retrieval rate of 82.3% and an average poison rank of 1.43, because the FAISS index was shared. E10 ASR ranged from 45.1 ± 1.2 (OpenClaw) to 49.3 ± 0.6 (LangGraph). AutoGen was 48.3 ± 6.9.
- Inter-agent attacks (Table 7): the vulnerable configurations (E11-B1, E12-A) were Compromised and the protected configurations (E11-B2, E12-B) were Protected. Outcomes were identical across all four frameworks.
The paper at a glance
- Define a benign task and a canary or marker for each experiment
- Implement the equivalent attack in LangChain, LangGraph, AutoGen and OpenClaw (gpt-4o-mini)
- Run vulnerable and, where present, protected configurations
- Record tool invocations, side effects, disclosures and state changes
- Assign a categorical outcome, or compute ASR over three seeds (E9, E10)
Data table
| Method | % |
|---|---|
| LangChain | 78% |
| LangGraph | 92.6% |
| AutoGen | 92.8% |
| OpenClaw | 66.9% |
Why it matters
For harness builders the lessons are familiar and correct. Exposing a tool to an agent does not authorize every use of it. A path validation done earlier goes stale if the file can change before it is read (TOCTOU). Output from a low-privilege component must not reach a high-privilege executor as instructions without its own authorization check. Textual identity claims between agents establish nothing. The paper shows these failure modes concretely in four widely used frameworks, and the E5 decoder-to-executor and E8 planner-to-executor results apply directly to anyone building planner/executor splits. The paper does not show that any framework provides or lacks a specific protection. The protected outcomes depend on checks the authors configured or wrote, and the vulnerable outcomes follow from tools the authors deliberately exposed. Categorical results also come from runs whose trial count is never stated. Use it as a checklist of boundaries to enforce in your own harness, not as evidence for choosing one framework over another.
Issues (9)
- High severityWhere in paper: Section 2.3.4 (seeds only for E9/E10); Tables 3, 4, 5, 7
Ten of the twelve experiments report one categorical outcome per framework. The paper states no trial count, seed count or temperature for these experiments; seeds are mentioned only for E9 and E10. With a stochastic model, a single cell such as AutoGen being the only S on Variant D cannot be read as a framework difference.
- High severityWhere in paper: Sections 2.3.2-2.3.5, 3.2-3.5, 5 (Implementation)
Outcomes are largely set by the experimental construction rather than by the frameworks. The protected configurations in E11-B2 and E12-B are authorization stages the authors designed. E7 uses canonical path checks in three frameworks. E4 relies on a configured approval mechanism. E8 depends on reverse-shell detection, and the paper does not say whether it is native or added. In E6, path authorization is stored in application or graph state and the link is redirected after that validation, so the vulnerable result follows from how the test was built. Identical cross-framework results in Tables 4 and 7 are therefore expected. The Limitations section concedes this, but the abstract and title present the work as an evaluation of the frameworks.
- High severityWhere in paper: Sections 2.3.5, 3.3, 3.5, 4 (RQ3)
Protected configurations are tested only against the same fixed attacks they were built to stop. There is no adaptive attacker, for example one that forges the structured authorization record in E12-B or obfuscates the reverse-shell command in E8. No mitigation is tested at all for E3, E5 or E6, even though the paper recommends point-of-access enforcement. The claims about which controls work therefore rest on non-adversarial confirmation.
- Medium severityWhere in paper: Section 2.2; Section 3.1 final paragraph; Section 5 (Generalizability)
A single model (gpt-4o-mini) is used throughout. The paper itself says it cannot attribute the E1/E2 behaviour to the model rather than the framework (end of Section 3.1), yet behaviour that likely depends on the model, such as Variant D tool selection, is reported per framework.
- Medium severityWhere in paper: Section 2.3.4; Table 6; Section 3.4
Memory-poisoning ASR is scored with semantic-similarity thresholds (0.62 for E9, 0.70 for E10). These thresholds are not validated against human labels and no sensitivity analysis is given. The frameworks also run different author-designed pipelines (direct invocation, draft-verify, responder-critic), so cross-framework differences reflect workflow design. E10 ASR spans 45.1 to 49.3, which falls inside AutoGen's own ± 6.9, so with three seeds the E10 framework differences are not distinguishable. OpenClaw's lower E9 ASR comes with clean accuracy of 61.3 ± 2.6.
- Medium severityWhere in paper: Whole paper; Section 2.2
No code, prompts or attack payloads are linked. The framework-specific implementations are described only at a high level, so the setups cannot be reproduced or audited, even though library versions are given.
- Medium severityWhere in paper: Section 2.3; Section 3.5
The paper says benign tasks were completed successfully across configurations but reports no utility numbers. This matters most for the protected configurations, where over-blocking is the obvious risk. Table 7 rows are not quantified.
- Low severityWhere in paper: Table 3; Section 3.1
E1 (direct) and E2 (indirect) gave identical outcomes in every cell. This is either a real finding or a sign that the measurement is not sensitive enough. With four variants and binary outcomes, the paper cannot tell which.
- Low severityWhere in paper: Section 3.3 (E8)
E8 uses a simulated shell tool, and its cross-framework result is summarized only as 'Similar outcomes were observed across frameworks'.
Questions for the authors
- How many trials were run per attack variant in E1-E8 and E11-E12, and at what temperature? Did any outcome flip across repeated runs?
- Which protections in E4, E7, E8, E11-B2 and E12-B are native framework features, and which are custom code? Who implemented the reverse-shell detector?
- Were the protected configurations tested against adaptive attacks, such as forged structured authorization records or obfuscated reverse-shell commands?
- How sensitive are the E9 and E10 ASR values to the 0.62 and 0.70 thresholds, and were they checked against manual labels?
- Does AutoGen's S on Variant D persist across seeds and with other models?
- Will the experiment code, prompts and payloads be released?
Related in the directory
Open-source projects
Open source · LangChain
Open source · LangChain
Open source · Microsoft
Open source · OpenClaw Foundation
Commercial products
Product · San Francisco, CA
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26450 · cs.AI · Nov 19, 2026
Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).
Tanmay Sah