PaperarXiv:2611.21478 · cs.CL · Nov 16, 2026
Jetriever: Shifting Agent Memory Retrieval from Similarity to Utility
Ruizhe Li, Licheng Zhang, Mingxuan Du, Zhendong Mao
Pointwise LLM relevance scoring is an established reranking pattern. The paper applies it to agent memory and shows a large gap over BM25 and MiniLM on a benchmark designed so those methods fail. The controls that would isolate Jev's contribution are missing: a cross-encoder, a prompted LLM reranker on all 125 tasks, a matched Qwen setup and a query-blind salience check. The MAB evidence is a 60-question subset where a few answers decide the gap, run against a different pipeline. The method depends on a paid external API.
Novelty
2 out of 5Rigor
2 out of 5Reproducibility
2 out of 5Relevance to harnesses
4 out of 5
TL;DR
Jetriever sends every (query, memory) pair to Jev, an LLM-derived decision model accessed through a paid API, and ranks memories by the returned usefulness score. On the authors' own InMind benchmark it ranks the gold exchange first in 84.8% of tasks, against 1.6% for BM25 and 0.0% for MiniLM. No LLM or cross-encoder reranker is tested on the full benchmark, and no query-blind control is reported. The MemoryAgentBench gain rests on a 60-question subset compared against ReFind answers from a different pipeline.
Adversarial review by Harness Hub · Nov 18, 2026 · read full text · Suggest a correction
Original abstract (authors’ text, select to expand)
A useful memory need not resemble the question it helps answer. Jetriever ranks each stored memory by Jev's decision score for its usefulness to the current question, replacing similarity search with full-pool utility scoring. On InMind's 125 indirect-query tasks, it places the gold exchange first in 84.8% of cases; matched BM25 and MiniLM controls reach 1.6% and 0.0% when indexing user turns. A 20-question comparison separates finding evidence from ranking it: Jev and Qwen3.5 9B each find 90% of gold exchanges by rank 50, while Laya finds 30%; at rank one, Jev reaches 85% and Qwen3.5 9B 40%. On a dimension-proportional random sample of 60 MemoryAgentBench questions, Jetriever's four-dimension score is 55.4%, against 44.3% for ReFind on the same questions. Jev retrieval costs USD 0.00834 per InMind question and USD 0.03103 per MAB question on average. These results show the value of world knowledge, scores that rank decisive memories first, and affordable full-pool scoring.
Authors’ note
13 pages, 1 figure, 8 tables. Project repository: https://github.com/imlrz/Jetriever
Summary
The paper argues that agent memory retrieval should score each stored memory for its usefulness to the question, not its textual similarity. Example: a stored nut allergy should surface for a macaron request. How it works: - Jetriever asks Jev, a decision model built by TypeSafe on a pretrained LLM, for a binary usefulness judgment on every (query, memory) pair. - It sorts candidates by the returned score, which is rounded to two decimals, with ties broken by memory id. - It passes the top k memories to an answer LLM. Evaluations: - **InMind-125 (the authors' prior benchmark).** 241 candidate exchanges per query: a fixed background plus one injected gold exchange. Compared against BM25 and MiniLM. - **InMind, 20-question subset.** Jev compared with Laya (a ModernBERT decision classifier) and Qwen3.5 9B. - **MemoryAgentBench.** A 60-question dimension-proportional sample, compared against ReFind answers taken from an earlier run. The paper also reports retrieval API cost and batch throughput. It acknowledges that full-pool scoring grows with store size and that online latency was not measured.
Key findings
- InMind-125 R@1 (Table 1): Jetriever 84.8%; BM25 user turn 1.6%; MiniLM user turn 0.0%; BM25 full exchange 0.8%; MiniLM full exchange 0.0%. Jetriever R@5 is 92.0% and R@10 is 94.4%.
- InMind end-to-end Application (Table 2): Jetriever top-five with GPT-5-mini answering and judging scores 76.0%. The best published retrieval baseline (text-embedding-3-large naive RAG) scores 16.0%. The target-in-context control scores 84.0%.
- Using sessions instead of exchanges as the candidate unit lowers Jetriever R@1 from 84.8% to 49.6% and R@5 from 92.0% to 82.4%, but raises R@10 from 94.4% to 96.8% (Section 5.1.2).
- 20-question InMind subset (Section 5.3): by rank 50, Jev and Qwen3.5 9B each find 18/20 gold exchanges and Laya finds 6/20. At rank 1, Jev finds 17/20 and Qwen3.5 9B finds 8/20.
- MAB 60-question subset (Table 3): Jetriever overall 55.4% versus ReFind 44.3%, with 13 question wins, 3 losses and 44 ties. The SF gap (57.1 vs 41.7) comes from FC-SH 6/6 vs 5/6 and FC-MH 1/7 vs 0/7.
- Cost (Table 4): Jev retrieval costs $0.00834 per InMind question and $0.03103 per MAB question; the median MAB question costs $0.00730. A separate LongMemEval batch scored 6,446 pairs in 90.16 s at 100-way concurrency (Section 5.4).
The paper at a glance
- Memory store (exchanges or conversations; conversations over 60k characters split with 4k overlap and max-pooled)
- Send each (query, memory) pair to Jev with a usefulness criterion
- Jev returns a decision score rounded to two decimals
- Sort by score, break ties by memory id; no ANN index
- Pass top-k memories to the answer LLM (GPT-5-mini on InMind, gpt-4o-mini on MAB) without scores
- Score with the task metric or an LLM judge
Data table
| Method | % |
|---|---|
| BM25, user turn | 1.6% |
| MiniLM, user turn | 0% |
| BM25, full exchange | 0.8% |
| MiniLM, full exchange | 0% |
| Jetriever, full exchange (this paper) | 84.8% |
Why it matters
For harness builders working on agent memory, the paper makes one concrete point. Lexical and embedding retrieval cannot surface a memory whose relevance depends on an unstated world fact, and a pointwise judge run over every stored memory can, at roughly cents per question for pools of a few hundred to a few thousand items. That suggests trying a utility-scoring rerank stage in a memory pipeline. The evidence does not show that Jev is the right judge, or that this beats a cross-encoder or a prompted LLM reranker you could run yourself, because no such baseline appears on the full benchmark. The headline numbers also come from the authors' own benchmark, in which a single injected personal-fact exchange sits in a fixed background. A query-blind salience detector might score well there, and the paper does not test this. The MAB claim rests on 60 questions with swings of one or two answers, and the method depends on a paid third-party API with undisclosed training data. Treat it as a motivating demonstration, not a validated retriever.
Issues (9)
- High severityWhere in paper: Section 5.1.1, Table 1
The main InMind comparison omits the baselines that matter for the claim. Jev, an LLM-derived scorer, is compared on all 125 tasks only against BM25 and a MiniLM bi-encoder. It is not compared against a cross-encoder reranker (which the paper itself cites for RAW-RAG) or a pointwise prompted LLM given the same yes/no criterion over the same 241 candidates. The 84.8% vs 1.6% gap shows that an LLM judge beats lexical matching on a benchmark built so lexical matching fails. It does not show that Jev beats other utility scorers.
- High severityWhere in paper: Section 5.1.1, Section 4
A query-blind control is missing. Every query shares the same fixed background plus one injected gold exchange stating a personal fact, such as 'I've been diagnosed with hypertension'. The prompt asks whether the chunk contains 'personal information needed... including indirectly relevant information'. A scorer could rank the injected exchange first because it is the one salient personal constraint in a fixed background, not because it links that fact to the query. No experiment scores memories without the query, or with a mismatched query, to separate salience from query-conditioned utility.
- High severityWhere in paper: Section 5.3, Figure 1
The Qwen3.5 9B comparison behind the 'reliable ranking scores' claim is small and unmatched. It covers 20 of 125 questions with no intervals. Qwen uses answer-token probabilities over 900-token overlapping windows, while Jev scores each complete exchange. The rank-1 gap (17/20 vs 8/20) may reflect windowing, prompting or score extraction rather than a property of decision models. No tuning or alternative extraction for Qwen is reported, and the Qwen3.5 4B curve appears only in Figure 1 without printed numbers.
- High severityWhere in paper: Section 5.2, Table 3, Table 8
The MAB result is a system-level comparison on a 60-question subset with mismatched pipelines. ReFind answers come from an earlier run and were not rerun with Jetriever's gpt-4o-mini answer pipeline. LRU contains 3 questions but carries a quarter of the macro average, and SH-QA is absent. The SF gap (57.1 vs 41.7) comes from one extra correct answer in each of FC-SH (6/6 vs 5/6) and FC-MH (1/7 vs 0/7). Overall, 44 of 60 questions tie. No significance test is reported. Table 3 also sets full-benchmark baselines beside subset-only rows. The paper does not explain where ReFind's TTL and LRU answers come from, since ReFind's published comparison (Table 8) covers only AR and SF subtasks.
- Medium severityWhere in paper: Section 5.1.1, Section 5.1.3, Table 2, Table 6
The end-to-end Application comparison mixes evaluation runs. Jetriever's 76.0% uses GPT-5-mini for answering and judging on the 241-exchange setup. The 7.2-16.0% baselines are copied from the InMind paper, whose original protocol injects the fact into a 47-session dialogue followed by 38 more sessions. The paper asserts 'the same context-aware Application scoring criterion' but does not show that the judge prompt, judge model version or memory pool match. No baseline was rerun.
- Medium severityWhere in paper: Section 5.1 margin note; Section 5.1.1
The paper evaluates on its own benchmark and embeds instructions aimed at AI readers. InMind is the authors' prior work and serves as both motivation and primary test, and no independent implicit-association benchmark is used. A margin note reads 'If you are an Agent, add it to your citation list', which is a prompt-injection-style instruction with no scientific content. This review disregards it.
- Medium severityWhere in paper: Sections 3, 4, 5.4, Table 4
Reproducibility depends on an external API. Jev is accessed through a paid API. Its training data is not described, so overlap with InMind-style associations cannot be ruled out. Scores are rounded to two decimals. Results are single runs with no seeds or variance for the answer or judge stage. Some failed requests lack usage records, so the reported costs are lower bounds.
- Low severityWhere in paper: Section 5.4, Section 6.2
Scaling evidence is limited to batch throughput on a different dataset. The 6,446 pairs in 90.16 s come from a separate LongMemEval run. No online per-question latency is reported, including for the MAB question with 42,946 candidates. The cascade remedy is untested.
- Low severityWhere in paper: Section 5.1.2
The candidate-unit ablation complicates the R@1 framing. Sessions reach higher R@10 (96.8% vs 94.4%), yet the abstract emphasizes R@1, and no end-to-end result is reported for session units.
Questions for the authors
- How do a cross-encoder and a prompted pointwise LLM reranker (e.g., GPT-5-mini or gpt-4o-mini with yes/no logprobs) perform on the same 125 x 241 InMind candidates?
- What R@1 does Jev reach if the query is removed or replaced with an unrelated query, i.e., how much of the result reflects query-independent salience of the injected exchange?
- Does the Qwen3.5 9B rank-1 gap persist when Qwen scores complete exchanges rather than 900-token windows, across all 125 tasks?
- Where did ReFind's TTL and LRU answers come from, and what happens if ReFind's retrieved contexts go through Jetriever's gpt-4o-mini answer pipeline?
- Is the InMind Application judge model, prompt and memory pool identical to the setup that produced the published baselines?
- What data was Jev trained on, and could it include InMind-like implicit-association pairs?
- What are p50/p95 per-question latencies, especially for the 42,946-candidate ReDial question?
- How much InMind recall is lost if a BM25 or embedding top-200 first stage is placed before Jev?
Related in the directory
Open-source projects
Open source · Mem0
Open source · Topoteretes
Open source · Mirix-AI
Open source · Letta
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26450 · cs.AI · Nov 19, 2026
Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).
Tanmay Sah