PaperarXiv:2611.20922 · cs.CL · Nov 16, 2026
Experience Rot: Experienced Agents Fail Where Novices Succeed
Zhen Xu, Hang Yin, Zitao Zhang, Kevin Zhao, Xuefeng Liu
The design is controlled and partly preregistered, and the remedy comparisons are ones harness builders can act on: docs in the user turn rather than the system prompt, read-back after writes, and writes that return the stored object. The underlying behaviour (experience-following, stale beliefs about tools) is already reported in cited prior work. The magnitudes depend on deliberately silent services, no argument validation, a single minimal loop and a non-preregistered incomplete-docs design.
Novelty
3 out of 5Rigor
3 out of 5Reproducibility
3 out of 5Relevance to harnesses
5 out of 5
TL;DR
Across three small synthetic services, each with one silent interface change, agents that already did a similar task copy their earlier call, skip the documentation and report success. With incomplete documentation, out-of-date experience gives 0.23 success against 0.63 with no experience. Reading back the stored state after writing is the only tested remedy that also helps when the docs are incomplete, at more than twice the model calls.
Adversarial review by Harness Hub · Nov 19, 2026 · read full text (truncated) · Suggest a correction
Original abstract (authors’ text, select to expand)
Agents are increasingly built to learn from experience: they keep conversation histories, memory files, retrieved notes and skills, and reuse what worked before. We show that this experience turns against them when their tools change silently. We test a range of models with tools defined as in the Model Context Protocol (MCP). After a silent change, experienced agents fail on most tasks, while agents new to the task mostly succeed. We call this failure Experience Rot. Experienced agents copy their earlier call instead of checking the current interface, even when the tool definition shows the change. Then they report success. When the documentation is incomplete, experience is what makes agents succeed. After a change, the same experience leaves them worse off than no experience at all. The failure appears in every form of memory we test and grows with the amount of experience. Simple remedies fall short: announcing the update barely helps, and pointing the agent to the documentation fails whenever the documentation does not spell out the answer.
Summary
The authors built three services: a calendar, a task list and a mail client. Each has versions A and B that differ in one detail: - a field name; - a structure; - an enum value set; - the meaning of an integer (priority direction, weekday, month or duration numbering). A write that is valid only in the other version is accepted with the same 'ok' response, so failures are silent. An agent first does an earlier task on one version, retried up to ten times until the stored state is correct. The final task then runs on the same version (valid experience), on the other version (out-of-date experience), or with no earlier task. Six models run in one minimal loop at temperature 0.7 with at most 16 model calls per task. There are 12,624 episodes, of which 136 are excluded. Most complete-documentation conditions were preregistered; the incomplete-documentation conditions were not. Reported results: - Out-of-date experience lowers success sharply. - Lookups before the first write fall to at most 6%. - Nearly all failed runs are reported to the user as done. - The effect appears with conversation history, memory files, retrieved notes and skills, and grows with the number of earlier tasks. - Update notices and system-prompt instructions barely help or make things worse. - Documentation or a read-docs instruction placed in the task message helps only when the documentation contains the answer. - A verify-and-correct instruction, or write responses that return the stored object, recover most failures. - Hand-written histories that include a read-back step hide most of the cost.
Key findings
- Incomplete documentation (Fig. 1c, Table 4): success is 0.63 with no experience, 1.00 with valid experience and 0.23 with out-of-date experience. The cost is 0.77 ± 0.01.
- Complete documentation (Table 4): out-of-date success is 0.26 with an untyped tool, 0.49 when the typed definition lists the new field, and 0.42 for a changed meaning of a number. Valid experience gives 1.00 in all three.
- Lookup before the first write (Table 4): for the untyped field case it falls from 1.00 with no experience to 0.01 with out-of-date experience. For meaning-of-a-number changes it falls from 0.70 to 0.00.
- Completion reports (Section 4): 347 of 348 failed runs with complete documentation end with a reply saying the task is done. With incomplete documentation the share is 92%.
- Storage form (Table 5): the cost is 0.53 for conversation, 0.27 for a memory file, 0.20 for retrieved memory and 0.30 for a skill. 92% of the notes that come with a skill record the old usage.
- Amount of experience (Table 7): out-of-date success is 0.46 with one earlier task, 0.33 with three and 0.26 with ten.
- Additions to the final task (Table 6, complete documentation): nothing added gives 0.46 and a version number plus update notice gives 0.50.
- Documentation placement (Table 6): documentation in the task message gives 0.99, in the system prompt 0.54, and in the tool description 0.55.
- Instructions (Table 6): 'read the documentation first' in the system prompt gives 0.29. Verify-and-correct gives 0.91 at 6.5 calls, against 2.9 calls with valid experience.
- Incomplete documentation (Fig. 5): 'read docs first' in the task message gives 0.25. Verify-and-correct in the task message gives 0.69 at 10.6 calls, against 3.1 with valid experience.
- Hand-written histories (Table 7): with a read-back step, out-of-date success is 0.88 (cost 0.11). The agent's own earlier task gives 0.46 (cost 0.54).
- Per model (Table 8, complete docs, typed tools): Qwen3.8 reads back after every write and reaches 1.00 out-of-date success, although its first write still uses the old usage in 0.88 of runs. MiniMax reaches 0.05 with a read-back rate of 0.00.
The paper at a glance
- Build a synthetic service with versions A and B that differ in one interface detail
- Agent does the earlier task live on A or B (up to ten attempts until the stored state is correct)
- Final task follows in the same conversation on A or B, or is run with no earlier task
- Silent acceptance: writes valid only for the other version return the same ok response
- A program checks the stored state; lookup, old usage, read-back and completion claim are logged
- Vary docs completeness, memory form, amount of experience, context additions and history authorship
Data table
| Method | fraction |
|---|---|
| Nothing added | 0.46 fraction |
| Docs in task message | 0.99 fraction |
| Docs in tool description | 0.55 fraction |
| Docs in system prompt | 0.54 fraction |
| Version number and update notice | 0.5 fraction |
| Instruction: read docs first (system prompt) | 0.29 fraction |
| Instruction: verify and correct | 0.91 fraction |
Why it matters
This paper is directly relevant to anyone building memory, skills or long sessions on top of MCP-style tools. It isolates a failure that harnesses can cause. Once an agent has a successful earlier call in context or in a memory file, it often stops consulting the current tool definition and docs, repeats the old call, and tells the user it succeeded. The remedy comparisons are concrete: - Version tags on memories and update notices do little. - A system-prompt instruction to check the docs lowers success. - Docs or instructions attached to the user turn help only when the docs contain the answer. - A read-back step, or write tools that return the stored object, help regardless of the docs, at more than twice the model calls for read-back. The finding that hand-written histories with a check step make memory look safe matters for anyone evaluating a memory layer. The evidence has limits: - The services are synthetic and deliberately permissive: they silently drop unknown fields and do not validate enums. - All models run in one minimal loop with no client-side schema validation. - The incomplete-docs headline depends on a non-preregistered design in which the user explicitly stated the old rule. Absolute failure rates may therefore overstate what a production harness sees. The direction of the effect and the ranking of remedies are the more transferable parts.
Issues (8)
- Medium severityWhere in paper: Abstract; Table 4; Table 8 (Appendix G.4)
The abstract says experienced agents 'fail on most tasks'. This does not hold for every condition or model. With complete docs and the change visible in the typed definition, out-of-date success is 0.49 (Table 4). Per model, GLM reaches 0.70 and Qwen3.8 reaches 1.00 (Table 8), although Qwen's first write still uses the old usage in 0.88 of runs. Pooling over models hides that the outcome depends largely on each model's read-back habit, not on experience alone.
- High severityWhere in paper: Section 3.2 (incomplete docs design); Section 4; Limitations; Appendix D (‡ not preregistered)
The headline incomplete-documentation result (0.63 vs 1.00 vs 0.23, Fig. 1c) comes from conditions that were not preregistered, and the design favours failure. - In the earlier request, the user explicitly states the meaning ('For this service, priority 4 is the highest'). The final request in the same conversation does not, and the docs do not contain the answer. - An out-of-date agent that follows the user's earlier statement is arguably behaving reasonably. It can only discover the change by reading back the stored state. - The no-experience baseline of 0.63 depends on whether the running version matches the model's default. Every run succeeds where January is month 1, but only 40% where it is month 0. The Limitations section admits these points, but the abstract presents 'worse off than no experience at all' as a general finding.
- High severityWhere in paper: Section 3.1; Appendix B (typed definition, enum values); Appendix A (Tool design: 'we did not test it')
Silence is engineered into the services, and ordinary schema validation would catch some changes. - Services ignore unknown fields and leave state unchanged on unknown values, while returning the same 'accepted' response. - In the shown typed definition, additionalProperties false applies only to the top-level parameters. The payload object is unconstrained, so 'attendees' passes. - For the value change, the typed definition lists the current enum (p0..p3 vs urgent..low). A client that validates arguments against the current schema would reject the old value outright. - A strict payload schema would likewise catch the field-name and structure changes. Rejecting unknown fields is explicitly untested. The failure rates for typed field and value changes therefore describe harnesses with no argument validation, not MCP tooling in general. Only the meaning-of-a-number changes are genuinely invisible to validation.
- Medium severityWhere in paper: Section 7; Table 6; Appendix G.2
The 'read docs first' system-prompt result (0.29, below nothing-added 0.46) is presented as a failed remedy, but it is confounded. Agents follow the instruction in the earlier task 98% of the time, so the old version's documentation enters the conversation. They follow it in the final task only 4% of the time. The drop may come from old-version docs being added to the experience, not from the instruction being harmful.
- Medium severityWhere in paper: Section 3.3; Appendix E; Limitations
The setting is narrow: - six models, none of them Claude, GPT or Gemini, several of them 'Flash' variants; - a single minimal agent loop at temperature 0.7 with no planning, verification or argument validation; - three synthetic services, with one interface difference per version; - one to ten earlier tasks. Most remedies were tested only with experience held in the conversation. Real frameworks with built-in verification may behave differently: the verify instruction alone moves success to 0.91.
- Medium severityWhere in paper: Section 3.3; Limitations
The standard errors are likely optimistic. They come from a cluster bootstrap over task instances that treats models and changes as fixed, and several pooled numbers rest on a few change types. The authors acknowledge this.
- Low severityWhere in paper: Section 3.3; Limitations; Appendix J
Exclusions condition on success. Of the 127 episodes dropped because the earlier task never succeeded, most are on the weekday change, mainly from three models. Weekday estimates for those models therefore use only runs where the earlier task succeeded.
- Low severityWhere in paper: Section 4; Section 5; Appendices G.7, G.9
Two measures rely on keyword patterns, not human annotation: - 'failed runs reported as done' (347 of 348; 92%); - the reasoning-text categories (plans lookup, refers to earlier). No validation of these patterns appears in the text provided.
Questions for the authors
- How much of the field and value failure remains if the client validates arguments against the current schema, including enums and additionalProperties false on the payload, or if the service rejects unknown fields?
- Do the results hold for Claude, GPT or Gemini models, and in real harnesses with their own plan or verification steps?
- In the incomplete-docs design, what happens if the earlier meaning is learned by trial rather than stated by the user?
- Does the system-prompt 'read docs first' penalty persist if old-version docs are removed from the earlier-task context?
- What does read-back cost in long tool chains where many writes cannot be read back directly?
- Will the services, task pools, prompts and preregistration be released? No code or data URL appears in the text provided.
Related in the directory
Open-source projects
Open source · Anthropic (community)
Open source · Letta
Open source · Mem0
Commercial products
People
Person · Co-founder & CEO
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26450 · cs.AI · Nov 19, 2026
Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).
Tanmay Sah