Harness Hub
All papers

Lost in the Hallway: Benchmarking Persistent Spatial Memory Across Revisits

Henry Zheng, Weibo Gong, Yang Yue, Rui Huang, Yang Zhang, Gao Huang

Verdict: Incremental

Making place identity across separated visits an explicit test target, with same-type distractors, is a reasonable contribution. The headline conclusions follow a familiar benchmark pattern: models are weak, humans are strong, and in-domain fine-tuning helps. The fine-tuning gain lacks controls for domain and format adaptation. The agentic negative result is underspecified in the abstract, and relevance to harness builders is low.

  • Novelty

    3 out of 5
  • Rigor

    2 out of 5
  • Reproducibility

    2 out of 5
  • Relevance to harnesses

    1 out of 5

TL;DR

ReSpaceBench is a video QA benchmark of 3,244 questions over 590 synthetic and real-home scenes. It tests whether models can recognize a revisited room and tell it apart from similar-looking rooms. The best zero-shot model reaches 50.5% overall, and three agentic video frameworks give no overall gain over direct inference with the same base model. Fine-tuning Qwen3-VL-8B on a separate synthetic training set (RS-30K) raises it from 35.5% to 60.4%.

Adversarial review by Harness Hub · Nov 19, 2026 · abstract only · Suggest a correction

Original abstract (authors’ text, select to expand)

Persistent spatial understanding is essential for embodied agents to operate reliably in environments they repeatedly encounter. It requires recognizing revisited places despite viewpoint changes, distinguishing them from similar-looking rooms, and updating memories as environments change. Yet existing evaluations often leave physical place identity across separated visits implicit. We introduce ReSpaceBench, a benchmark comprising 3,244 questions across 590 synthetic and real-home scenes, organized into 9 tasks spanning room association, persistent state tracking, and downstream reasoning. Controlled walkthroughs define visits by physical room entry and deliberately include same-type room distractors and cross-session object changes. Our evaluation of 32 models and 3 agentic video-understanding frameworks reveals substantial limitations: the strongest evaluated zero-shot model achieves 50.5% overall accuracy and 52.2% on a 450-question subset, compared with 90.1% for human annotators. The evaluated agentic frameworks offer no overall improvement over direct inference with the same base model. Analysis of agent execution traces reveals evidence omissions and incorrect cross-visit room associations as major failure modes, suggesting that iterative evidence seeking alone does not ensure consistent spatial memory. Further analyses expose a gap between recognizing familiar rooms and identifying the correct visited room, while annotation-guided sampling highlights the importance of preserving intervening visits and transitions. We also provide RS-30K, a separate training dataset from synthetic sources with no template, scene, route, or video overlap with the benchmark. Fine-tuning Qwen3-VL-8B raises overall accuracy from 35.5% to 60.4%, with the largest gains in association and improvements extending to real-life scenes.

Filed undermemoryevaluationagents

Summary

The paper targets persistent spatial memory for embodied agents. That means recognizing a physical room across separated visits, rejecting same-type distractor rooms, and updating memory when objects change between sessions. ReSpaceBench contains 3,244 questions over 590 synthetic and real-home scenes, organized into 9 tasks in three groups: room association, persistent state tracking, and downstream reasoning. Controlled walkthroughs define visits by physical room entry and deliberately include same-type room distractors and cross-session object changes. The authors evaluate 32 models and 3 agentic video-understanding frameworks. The strongest zero-shot model reaches 50.5% overall accuracy, and 52.2% on a 450-question subset where human annotators reach 90.1%. The agentic frameworks give no overall improvement over direct inference with the same base model. Their execution traces show evidence omissions and incorrect cross-visit room associations as major failure modes. Further analyses report a gap between recognizing familiar rooms and identifying the correct visited room. Annotation-guided sampling suggests that preserving intervening visits and transitions matters. The authors also release RS-30K, a synthetic training set stated to have no template, scene, route, or video overlap with the benchmark. Fine-tuning Qwen3-VL-8B on RS-30K raises overall accuracy from 35.5% to 60.4%, with the largest gains in association and improvements extending to real-life scenes. This review is based on the abstract only.

Key findings

  • Strongest evaluated zero-shot model: 50.5% overall accuracy on ReSpaceBench (abstract).
  • On a 450-question subset, the strongest zero-shot model scores 52.2% and human annotators score 90.1% (abstract).
  • The 3 evaluated agentic video-understanding frameworks offer no overall improvement over direct inference with the same base model (abstract).
  • Agent execution traces show evidence omissions and incorrect cross-visit room associations as major failure modes (abstract).
  • Analyses show a gap between recognizing familiar rooms and identifying the correct visited room (abstract).
  • Annotation-guided sampling highlights the importance of preserving intervening visits and transitions (abstract).
  • Fine-tuning Qwen3-VL-8B on RS-30K raises overall accuracy from 35.5% to 60.4%. The largest gains are in association, and improvements extend to real-life scenes (abstract).

The paper at a glance

01Controlledwalkthroughs of 590synthetic andreal-home scenes02Visits defined byphysical roomentry, withsame-type roomdistractors and…033,244 questionsacross 9 tasks(room association,persistent statetracking,…04Evaluate 32 modelsand 3 agenticframeworks; humanannotators on a450-question subset05Agent traceanalysis andannotation-guidedsampling analysis06Fine-tuneQwen3-VL-8B onRS-30K (synthetic,no stated overlapwith the benchmark)
  1. Controlled walkthroughs of 590 synthetic and real-home scenes
  2. Visits defined by physical room entry, with same-type room distractors and cross-session object changes
  3. 3,244 questions across 9 tasks (room association, persistent state tracking, downstream reasoning)
  4. Evaluate 32 models and 3 agentic frameworks; human annotators on a 450-question subset
  5. Agent trace analysis and annotation-guided sampling analysis
  6. Fine-tune Qwen3-VL-8B on RS-30K (synthetic, no stated overlap with the benchmark)
Fig. 1 — How the paper works, step by step (arXiv:2611.21485).
Qwen3-VL-8B (zero-shot)35.5%Strongest evaluated zero-shot model50.5%Qwen3-VL-8B fine-tuned on RS-30K (this paper)60.4%
Fig. 2 — ReSpaceBench overall accuracy (zero-shot and fine-tuned conditions mixed; see labels), higher is better. Source: Abstract, arXiv:2611.21485.
Data table
ReSpaceBench overall accuracy (zero-shot and fine-tuned conditions mixed; see labels)
Method%
Qwen3-VL-8B (zero-shot)35.5%
Strongest evaluated zero-shot model50.5%
Qwen3-VL-8B fine-tuned on RS-30K (this paper)60.4%

Why it matters

For most harness builders this paper is peripheral. It is an embodied, video-based spatial benchmark, not a coding-agent, tool-loop, or text-memory evaluation, and it does not build on or evaluate any project in this directory. The one transferable point is a negative result. Wrapping a base model in an iterative evidence-seeking agent loop did not improve overall accuracy over a single direct call. The traces fail through omitted evidence and wrong identity association across visits. That parallels a known weakness in agent memory layers: retrieval loops that do not keep entity identity consistent across sessions. Builders of memory systems for multimodal or robotics agents may find the task split useful (association vs state tracking vs downstream reasoning). Builders of coding or web harnesses can skip it.

Issues (6)

  • High severity

    The fine-tuning result (35.5% to 60.4%) is supervised training on data built by the same authors for the same problem. The abstract rules out template, scene, route, and video overlap. It does not rule out distribution-level alignment such as synthetic rendering source, room taxonomy, question format, or answer-option structure. The abstract gives no comparison against fine-tuning on other generic spatial or video data. Without one, the gain cannot be attributed to persistent spatial memory rather than format and domain adaptation. Placing 60.4% alongside the 50.5% zero-shot best (as in the bars here) compares trained against untrained conditions.

  • High severity

    The claim that agentic frameworks give no overall improvement rests on 3 unnamed video-understanding frameworks. The abstract says they share a base model with the direct-inference baseline but does not name it. It also omits tool or frame budget, iteration limits, prompt adaptation, and seeds. A negative result about iterative evidence seeking is only as strong as the effort spent configuring those harnesses, and the abstract gives no evidence they were adapted to multi-visit walkthroughs.

  • High severity

    The abstract does not give the synthetic vs real-home split of the 590 scenes or the 3,244 questions. If the benchmark is mostly synthetic, the headline accuracies mainly measure performance on rendered scenes. The claim that fine-tuning improvements 'extend to real-life scenes' then rests on an unstated and possibly small real subset, with no confidence intervals or variance reported.

  • Medium severity

    The human baseline of 90.1% covers only a 450-question subset, so the comparable model number is 52.2%, not the full-benchmark 50.5%. The abstract does not state annotator count, how the subset was selected, or whether humans saw full video with free scrubbing while models saw sampled frames. Unequal input access can inflate the human-model gap.

  • Medium severity

    The paper's own annotation-guided sampling analysis shows that keeping intervening visits and transitions matters. If the 32 models were run under fixed frame budgets that drop those segments, part of the reported failure reflects the input pipeline rather than model memory. Per-model frame budgets are not given in the abstract.

  • Low severity

    The abstract does not mention a code, data, or evaluation-harness release URL for ReSpaceBench or RS-30K. That limits independent reproduction of the 32-model sweep.

Questions for the authors

  • Which 3 agentic video-understanding frameworks were evaluated, on which shared base model, and with what tool budgets, iteration limits, and prompt adaptation?
  • What is the synthetic vs real-home split in scenes and questions, and what are base vs fine-tuned accuracies on the real subset alone, with confidence intervals?
  • How was the 450-question human subset sampled, how many annotators took part, and did they view full video or the same sampled frames as models?
  • What frame sampling strategy and frame budget were used for each of the 32 models?
  • How does fine-tuning Qwen3-VL-8B on a comparable amount of generic spatial or video QA data compare with RS-30K?
  • Does the RS-30K fine-tuned model lose accuracy on general video or spatial benchmarks outside ReSpaceBench?
  • Are results averaged over multiple seeds or decoding runs, and what is the variance?

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).

Tanmay Sah

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.