Harness Hub
All papers

Mind the Gap: A Deployed Multi-Agent System for Documentation Maintenance

Lorenzo Masoero

Verdict: Incremental

This is a sensible, openly caveated deployment report. It turns agent-vs-human answer deltas into a documentation-repair backlog. The quantitative support is thin: - correctness is judged by LLMs only, on a minority of verifiable claims; - the before/after test re-asks the ticket whose human answer seeded the fix, with no variance control; - only 7 gaps have a gap-tagged commit; - the proactive component never fired. The paper offers a usable design pattern, not strong evidence.

  • Novelty

    3 out of 5
  • Rigor

    2 out of 5
  • Reproducibility

    1 out of 5
  • Relevance to harnesses

    3 out of 5

TL;DR

An Amazon multi-agent system drafts support answers from documentation and treats the difference from the human answer as a documentation gap to fix. Over 43 days in shadow mode it logged 92 gaps from 499 triaged tickets. Quality evidence is almost entirely LLM-judge output. The self-improvement result rests on 35 selected pairs that re-answer the same ticket whose human resolution was used to write the fix. Only 7 of the 92 gaps have a gap-tagged documentation commit.

Adversarial review by Harness Hub · Nov 20, 2026 · read full text · Suggest a correction

Original abstract (authors’ text, select to expand)

The "AI revolution" has enabled teams across the technology industry to deliver software features at unprecedented velocity. While code evolves faster than ever, customer-facing documentation often struggles to stay up to speed. As users increasingly rely on LLM-based agents that consume documentation as their primary interface, the impact of this documentation gap worsens: stale information propagates and is treated as authoritative. Signals for identifying where documentation has fallen behind already exist in abundance, both explicit (e.g., resolved support tickets revealing what documentation lacks) and implicit (e.g., code-documentation discrepancies detectable from automated loops). We deploy a multi-agent system, built around a triage agent we call "TriAgent," that continuously extracts these signals and repairs documentation drift. Over 43 days of shadow-mode operation, our system triaged 499 tickets and surfaced 92 gaps in the documentation. Using a panel of LLM judges from independent model families, 81.1% of TriAgent's verifiable assertions were correct (245 of the 302 assertions the human resolution addresses; majority of three judges); on length-comparable pairs the panel preferred TriAgent's response in 70.5% of pairs and the human engineer's in 23.2% (the rest had no judge majority). Importantly, agent- and human-sourced replies tended to be complementary, with the human often surfacing insights that were either buried within the existing documentation or missing from it entirely; our system turns each such insight into a documentation gap and, after human review, a fix. Repairing a gap measurably improved the agent's later answer: when the agent re-answered the same ticket after the repair, blinded judges preferred the post-repair answer over its own pre-repair answer 71.4% of the time (n=35). A blinded human audit confirms strong alignment on self-improvement.

Authors’ note

Accepted at IAAI-27 (Innovative Applications of AI, co-located with AAAI 2027), Emerging Applications track. 12 pages (6 pages + references and appendix), 2 figures, 6 tables

Filed undermulti-agentself-improvingagentsllm

Summary

The paper reports a shadow-mode deployment on the support queue of an internal A/B testing service. How the system works: - TriAgent drafts a documentation-grounded reply to each incoming ticket. - The Feedback Loop agent (live) and the Archivist (historical harvest) compare that draft with the human resolution and write gap entries to a shared improvement tracker. - A Student agent probes topics with synthetic questions. - A Code-Merge Watcher maps merged code changes to documents through a registry. - A Documentator drafts fixes, passes them through a self-review gate, and opens a code review that a human must approve. How it is evaluated: - A panel of three cross-family LLM judges scores 192 agent/human pairs. It checks assertion-level correctness against the human reply and gives a blinded pairwise preference. - 35 before/after re-answers test whether repairs improved later answers. - A BM25 retrieve-and-generate baseline is compared against the full pipeline. - A single-author blinded audit re-labels a subsample. The authors disclose several negatives: - The code watcher found no gaps. - Reviewer time was not measured. - The human audit agrees with the panel only at kappa about 0.30 on correctness and usefulness.

Key findings

  • Operational scale (Table 2): 499 tickets triaged and 92 documentation gaps discovered. Median response availability lead time is 4.6 h, and drafting time is under 5 min.
  • Correctness (Section 3.2): of 1121 extracted assertions, 245 are correct, 57 incorrect and 813 (72.5%) unverifiable. Accuracy over verifiable assertions is 81.1% (245/302), with Wilson CI [76.3, 85.1]. It is 21.9% if unverifiable claims count as wrong. The contradiction floor is 57/1121 (5.1%).
  • Pairwise preference (Table 3): across all 192 pairs, the agent is preferred in 78.1% and the human in 18.2%. On the 112 length-comparable pairs the split is 70.5% vs 23.2%, with 6.2% having no majority. Judge kappa is 0.41.
  • Coverage (Section 3.3): the agent raises a correct material point the human omits in 91.1% of pairs. The human raises one the agent omits in 83.9%.
  • Self-improvement (Section 3.4): on 35 selected pairs the post-repair answer is preferred in 71.4% (CI [54.9, 83.7], kappa 0.61). The single-author human audit gives 74.3%, with Cohen's kappa 0.68 against the panel. The 35 pairs come from the 40 of 231 matched pairs that were not high-confidence (Appendix C.5).
  • Retrieval baseline (Section 3.5): the full system is preferred 64.6% of the time and the BM25 baseline 31.2% (kappa 0.55). Both run over documentation the loop had already improved.
  • Gap sources (Appendix A): harvest 34, live triage 28, Student 6, Feedback Loop 2, no recorded source 21, manual 1. The Code-Merge Watcher contributed 0. More than 90% of gaps are of type missing.
  • Closure (Section 3.1): all 92 gaps are marked fixed, most in a single batch at the end of the window. Only 7 carry a gap-tagged commit.

The paper at a glance

01Ticket arrives;TriAgent drafts adocumentation-groundedreply (shadow mode)02On-call humanresolves the ticket03Feedback Loop /Archivist compareagent and humanreplies; Studentprobes; Code-Merge…04Gap entries arededuplicated andprioritized in theshared improvementtracker05Documentatorresearches, draftsand self-reviews afix06Fix goes to codereview; a humanapproves the merge
  1. Ticket arrives; TriAgent drafts a documentation-grounded reply (shadow mode)
  2. On-call human resolves the ticket
  3. Feedback Loop / Archivist compare agent and human replies; Student probes; Code-Merge Watcher checks the registry
  4. Gap entries are deduplicated and prioritized in the shared improvement tracker
  5. Documentator researches, drafts and self-reviews a fix
  6. Fix goes to code review; a human approves the merge
Fig. 1 — How the paper works, step by step (arXiv:2611.23142).
Full system (TriAgent pipeline) (this paper)64.6%BM25 + same generator baseline31.2%
Fig. 2 — Blinded pairwise preference, full pipeline vs BM25 retrieve-and-generate (192 pairs, cross-family judge majority), higher is better. Source: Section 3.5, arXiv:2611.23142.
Data table
Blinded pairwise preference, full pipeline vs BM25 retrieve-and-generate (192 pairs, cross-family judge majority)
Method%
Full system (TriAgent pipeline) (this paper)64.6%
BM25 + same generator baseline31.2%

Why it matters

For harness builders the reusable idea is architectural. The gap between an agent's grounded answer and the eventual human resolution is treated as a free supervision signal. That signal is routed into a shared backlog, which agents drain by editing the retrieval corpus, with a self-review gate and a mandatory human merge. This is a concrete pattern for teams running support or ops agents over a knowledge base, and it is a corpus-level alternative to agent memory. The paper does not show that the loop saves effort or improves answers on new tickets. Reviewer time is unmeasured, and the proactive code-drift component produced zero gaps. The before/after test re-asks the same ticket whose human answer seeded the fix, which makes the improvement close to built in. Most other evidence is LLM-judge output that one author agreed with only fairly. Builders of coding agents or tool loops will find little transferable measurement. Builders of RAG-backed support agents get a reasonable design sketch, not a validated result.

Issues (9)

  • High severity

    The self-improvement test is close to circular. The 35 pairs re-answer only the triggering ticket. The documentation fix for that gap was derived from that same ticket's human resolution, via the Feedback Loop or Archivist comparison. Preferring the post-repair answer therefore largely shows that putting the human's answer into the corpus lets the agent retrieve it. It does not show generalization; the authors concede that improvement on new related tickets is not measured. Further weaknesses: - There is no null control: the same ticket is never re-answered twice with unchanged docs. - The sample is selected: 35 of the 40 non-high-confidence pairs out of 231 matched pairs. - The CI spans [54.9, 83.7]. - The human audit is by one author of the system, not an independent annotator. The abstract's phrase 'measurably improved the agent's later answer' overstates this.

  • High severity

    The headline 81.1% correctness is computed on 302 of 1121 assertions. The other 813 (72.5%) cannot be checked against the terse human reply, so the agent's accuracy on most of its claims is unknown. Judges agree least where it matters: - only 39% of 'incorrect' labels are unanimous; - kappa on the verifiable subset is 0.50, below the overall 0.73 quoted in Section 3.2 and the Conclusion; - the single-author audit agrees with the panel on correctness only at kappa about 0.30. The authors call these rates 'panel-internal estimates' in the Limitations and Conclusion. The abstract still presents 81.1% without that caveat.

  • High severity

    Causal attribution for repairs is thin. The abstract says 'the documentation commit that resolved each tagged gap postdated its discovery'. However: - only 7 of the 92 gaps carry a gap-tagged commit; - all 92 are marked fixed, mostly in one end-of-window batch; - the paper does not say how many of the 35 before/after pairs fall among those 7.

  • Medium severity

    There is possible contamination in the scored set. The 192 pairs include retrospective re-triage of historically resolved tickets. If those re-answers ran against documentation that had already been updated from earlier harvests, the agent could retrieve content derived from the very human resolutions it is graded against. That would inflate correctness and preference. The paper does not state which documentation snapshot each scored response used.

  • Medium severity

    The retrieval-only baseline is not a clean ablation: - it runs over documentation the loop had already improved; - it is compared against the full agent's 'deployed response', so the two sides may have seen different corpus states; - the two differ in both retrieval and orchestration. The 64.6% vs 31.2% result does not isolate which component earns the complexity.

  • Medium severity

    The agent-vs-human preference is length-confounded. The agent is preferred in every pair where its response is more than twice as long. The 'comparable-length' control still allows a 2x ratio. Judge kappa is 0.41, and human audit agreement on usefulness is about 0.30. 'Preferred over the human engineer' means preferred by LLM judges, not by customers or engineers.

  • Medium severity

    Several components are claimed but not demonstrated: - The Code-Merge Watcher recorded no gaps, and its registry held only 21 documents and 18 packages. - Reviewer time was not measured, so the '<5 min' drafting time does not show that human effort is saved. - The self-review gate's catch rate is not reported. - 21 of 92 gaps have no recorded detection source.

  • Medium severity

    Reproducibility is minimal: - all models are anonymized; - the data are internal tickets; - no code or data URL is given; - the study covers a single service with a single annotator. Appendix E also says all evaluations use 'the judge model (Model B)', which conflicts with the stated three-judge panel of B, C and D.

  • Low severity

    The human resolution is the sole ground truth and is never audited. A wrong or incomplete human answer would penalize correct agent claims or inflate agreement.

Questions for the authors

  • How much of the before/after preference survives a control that re-answers the same ticket twice with unchanged documentation?
  • Does a repaired gap improve answers on held-out tickets that share the gap but did not seed the fix?
  • How many of the 35 before/after pairs correspond to the 7 gaps with gap-tagged commits?
  • Which documentation snapshot did the retrospective re-triage responses in the 192-pair set use, and could it contain content derived from those tickets' human resolutions?
  • How much reviewer time did the Documentator's drafts consume, and how many were rejected or substantially rewritten?
  • Did the BM25 baseline and the deployed agent response see the same documentation snapshot?

Related in the directory

Open-source projects

The agents coordinate following the multi-agent conversation pattern the paper cites as Wu et al. 2024 (AutoGen).
Cited as the oracle-backed agent repair setting the paper contrasts with documentation, which lacks test oracles.

Commercial products

Commercial documentation-maintenance agent in the same problem space; it is a competitor, though the paper does not cite it.

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).

Tanmay Sah

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.