PaperarXiv:2611.20927 · cs.MA · Nov 16, 2026
LEDGER: Auditable Agent Memory Under Change
Md Aidul Islam, Md Nasir Uddin Shuvo, Muhammad Waseem, Pekka Abrahamsson
The design is clear, and the evidence-level evaluation is directly usable by memory-layer builders. However, the large gains come from a self-built workload that encodes the system's own policy and on which the system was tuned. The deletion advantage is partly adapter-driven, and the external benchmark result is a statistical tie.
Novelty
3 out of 5Rigor
3 out of 5Reproducibility
4 out of 5Relevance to harnesses
4 out of 5
TL;DR
LEDGER is a fact-centric agent memory: an LLM proposes facts, and fixed rules decide what is current, which source wins and what is deleted. On a synthetic coding workload the authors built, LEDGER+EP fallback beats full replay by 7.8 points on held-out seeds and sends far fewer stale or deleted values to the model. On a 210-question LongMemEval subset it matches full replay at 74.8%, a tie within wide intervals.
Adversarial review by Harness Hub · Nov 19, 2026 · read full text · Suggest a correction
Original abstract (authors’ text, select to expand)
Agent memory systems are mostly evaluated on fixed histories, and only their final answers are scored. In real use, facts change, sources differ in reliability, and users ask the agent to forget things. We present LEDGER, a memory layer in which a language model proposes facts and explicit rules decide which value is current, which source to trust, and what to delete. Each fact keeps its source and the period in which it held. LEDGER+EP adds a search over raw conversation turns for questions the ledger does not cover, and scrubs these turns on deletion. We compare both against no memory, full-history replay, retrieval over raw transcripts, and Mem0, with the same answering model and prompt, on a synthetic multi-session coding-agent workload and a 210-question subset of LongMemEval, and we score both the evidence each memory sends to the model and the answer. On the coding workload, LEDGER+EP in fallback mode answers 94.3% of questions correctly on five development seeds and 91.1% on five held-out seeds, against 82.1% and 83.3% for full replay, the strongest baseline. When a deletion request only describes the fact, no deleted value reached our systems' evidence in any evaluated probe. On LongMemEval, LEDGER+EP in augment mode has the same observed accuracy as full replay (74.8%) with 1% of its context. The main limitation is that the coding workload is synthetic and built by us; on LongMemEval, the differences between the strongest systems are within the confidence intervals.
Summary
LEDGER keeps per-user slots (entity/attribute) with validity intervals, provenance and a three-level source trust order. An LLM extractor proposes claims, and a lexical grounding check drops values that do not appear in the cited event. Five write rules then decide state: restate, supersede, dispute, introduce and hold. An LLM router maps each question or deletion request to known slots. LEDGER+EP adds a BM25 store of raw turns, used in one of two modes: - fallback, only when the ledger has no fact; - augment, always filling the remaining budget. Episode turns are scrubbed on deletion. The evaluation scores both the evidence packet sent to a shared answerer (gpt-4.1-mini) and the answer. Wrong values are labelled stale, future, contaminated or resurrected. Baselines are no memory, full replay, transcript RAG and Mem0 OSS 1.0.11, plus a Mem0 variant with a coding prompt. Results cover: - a synthetic coding workload: 5 development seeds, 5 held-out seeds, and a 540-day long preset; - a stratified 210-question LongMemEval_S subset; - restart, model-change and run-to-run noise checks. The paper states its main limits itself: the workload is synthetic, the development seeds flatter the system, the LongMemEval intervals overlap, and no downstream agent task is tested.
Key findings
- Coding workload, development seeds 0-4 (Table V): LEDGER+EP fallback answers 94.3 ± 4.2%, augment 94.4 ± 2.2%, LEDGER (LLM) 90.9 ± 5.5%, full replay 82.1 ± 3.0%, transcript RAG 74.8 ± 3.6%, Mem0 66.8 ± 1.8%.
- Held-out seeds 5-9 (Table VI): LEDGER+EP fallback scores 91.1 ± 5.4% against full replay at 83.3 ± 2.8%. The lead shrinks from 12.2 to 7.8 points; LEDGER's lead shrinks from 8.8 to 4.9 points.
- Evidence quality (Table V): full replay's evidence is clean in 21.9% of probes, and 249 of its probes carry a forbidden value. LEDGER+EP fallback has 16 such probes and augment has 193.
- Description-only deletion (Table V, Sec. V-A): full replay sent the deleted value to the model in 46 probes, transcript RAG in 39, and Mem0 in 18 (17 reached the user). LEDGER and LEDGER+EP sent 0.
- Contamination and history (Sec. V-A): full replay answers 31 of 67 contamination probes and LEDGER answers 62 of 67. Mem0 answers 27 of 93 historical questions because its updates overwrite in place.
- LongMemEval subset (Table VII): LEDGER+EP augment reaches 74.8% with 1,087 injected tokens, against full replay at 74.8% with 108,447. The paired 95% CI of the difference is -6.2 to +6.2. Mem0 scores 71.0% and LEDGER alone 41.0%.
- Long preset (Fig. 2, Sec. V-C): full replay falls from 89.4% to 70.5% and transcript RAG from 88.0% to 66.5%. LEDGER stays between 85.0% and 93.3%. Mem0 drops to 56-63% from the fourth checkpoint on.
- Operational checks (Sec. V-D): the accuracy sd across repeated runs is 0.7 and 0.4 points per seed. Restoring from disk reproduced the verdicts of all 1,379 probes.
The paper at a glance
- Session events (user, tool, doc, assistant)
- LLM extractor proposes <event, slot, value> claims
- Enumerated-id and lexical grounding checks
- Ledger core write rules by source trust: restate / supersede / dispute / introduce / hold
- LLM router maps question or deletion to known slots
- Evidence: current fact lines plus BM25 raw turns (fallback or augment)
- Shared answerer gpt-4.1-mini
- Deletion: erase entries, keep tombstone, scrub episode turns
Data table
| Method | % |
|---|---|
| No memory | 29.4% |
| Full replay | 83.3% |
| Transcript RAG | 78.5% |
| Mem0 | 66.2% |
| Mem0 (coding prompt) | 68.4% |
| Ledger (rules) | 37% |
| Ledger (LLM) (this paper) | 88.2% |
| Ledger+Ep fallback (this paper) | 91.1% |
| Ledger+Ep augment (this paper) | 91.8% |
Why it matters
For anyone building a memory layer for a coding agent, this paper offers a concrete design and evaluation template. It separates LLM extraction from deterministic state rules: supersession with validity intervals, a source trust ranking, and tombstoned deletion with raw-turn scrubbing. It also scores what the memory actually sends to the model, not only the final answer. The evidence-versus-answer gap it documents is the most transferable result. Full replay answers 82.1% of probes while its evidence is clean in only 21.9%, and a deleted value the model does not repeat has still been sent to the provider. The headline accuracy gains come from a workload the authors wrote together with the system and its trust policy. Read them as a demonstration that the rules do what they claim, not as evidence that LEDGER beats Mem0 or replay on real traces. The LongMemEval result is a tie within wide intervals. The parts most worth adopting are the write-rule table, the trust policy, the deletion and scrubbing semantics, and evidence-level scoring.
Issues (8)
- High severityWhere in paper: Sec. IV-A (Ground truth), Sec. VII Construct/Internal validity
The coding workload, its oracle and LEDGER were written together. The workload also encodes LEDGER's own trust policy: contradictions come only from documents and the assistant. The headline 94.3% / 91.1% therefore measures conformance to the authors' policy on templated text with nine fact kinds, not correctness on real agent traces. Held-out seeds use the same templates and mutation rules, so they do not test generalisation beyond the generator.
- High severityWhere in paper: Sec. II-B, Sec. III-E, Sec. V-A Deletion, Table V
The deletion comparison is largely decided by the adapter. Transcript baselines and the Mem0 adapter delete only by event id, so a description-only request removes nothing through their delete call by construction. Only LEDGER's router acts on the description. The abstract's 'no deleted value reached our systems' evidence' (0 against 46, 39 and 18) therefore partly reflects integration choices, not memory design. The guarantee also holds only on a workload where each event states at most one fact and values are unique per user. It is conditional on A1/A2, leaves an open case for missed trusted turns, does not reach caches or logs, and was not tested on LongMemEval.
- Medium severityWhere in paper: Sec. IV-C Two scores; Table V Evidence column
Evidence accuracy is structurally unfavourable to the baselines. On abstention probes, only an explicit memory-level abstention counts, so no memory, full replay and transcript RAG cannot score correct there at the evidence level. Part of the 91.7% vs 21.9% evidence gap is an artifact of the metric rather than of retrieval quality.
- Medium severityWhere in paper: Table V vs Table VI, Sec. V-B, Sec. V-G, Sec. VII Internal validity
Tuning effort is asymmetric. LEDGER's rules, safeguards and extraction prompts were revised while inspecting failures on seeds 0-4, and the abstract's 94.3% headline is reported on those seeds. Mem0's coding prompt was fixed before any run and never iterated. On held-out seeds the lead over full replay drops from 12.2 to 7.8 points. Safeguard ablations are single development runs, with no leave-one-out on the final system.
- Medium severityWhere in paper: Table V, Table VI, Sec. VII Conclusion validity
Coding-workload comparisons rest on five seeds with sds up to 5.5 points for the authors' systems (5.4 for fallback on held-out seeds). No confidence interval or test is reported for the coding-workload system differences.
- Medium severityWhere in paper: Sec. IV-B, Table VII, Sec. V-B LongMemEval
The LongMemEval claim of 'same accuracy as full replay with 1% of its context' rests on a single run with a paired CI of -6.2 to +6.2 points. The mode choice used a 14-question pilot that shares 10 questions with the evaluation set. The judge audit covered only 3 of 7 configurations and excluded Mem0, LEDGER and fallback mode. Against Mem0 the CI is -3.8 to +11.4, so no ranking is established.
- Medium severityWhere in paper: Sec. IV-C Systems, Sec. VII External validity, Sec. VIII
The baseline set is thin. Only one memory product (Mem0) is evaluated. Zep, which the authors call closest to their design, and A-MEM are not evaluated. All extraction, routing and answering use the gpt-4.1 family.
- Low severityWhere in paper: Sec. IV-D, Sec. V-E, Sec. V-G (a'), Sec. V-H
There is no downstream agent task, so the paper shows the memory supplies the right facts, not that a coding agent works better with it. The coding histories are short (about 4.4k tokens per user at the end), so cost savings there are small: $0.23 against $0.28 for full replay. The LEDGER to LEDGER+EP comparison is also confounded, because it adds the episode store and also stops extracting low-trust events.
Questions for the authors
- How does LEDGER+EP compare with Zep/Graphiti's temporal knowledge graph on the same coding workload and on description-only deletions?
- What is the recall loss from episode scrubbing, which the paper says was not measured?
- If Mem0 were given a description-based delete integration, how much of the deletion gap would remain?
- How often do real coding-agent traces violate A3 (trust order) and A4 (valid time equals statement time)?
- Does the held-out lead survive a third-party workload with non-templated phrasings and fact kinds beyond the nine used?
- What is the leave-one-out effect of each safeguard on the final system?
Related in the directory
Open-source projects
Open source · Mem0
Open source · Zep
Open source · Letta
Commercial products
Product · San Francisco, CA
Product · San Francisco, CA
Product · Berkeley, CA
People
Person · Co-founder & CEO
Person · Founder & CEO
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26450 · cs.AI · Nov 19, 2026
Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).
Tanmay Sah