Harness Hub
All papers

SWE-WhatIf: Simulating Real Users of Coding Agents from Recorded Session

Tae Soo Kim, Tongshuang Wu, Juho Kim

Verdict: Useful

The paper gives a well-described design for replaying real coding-agent sessions with a stateful, intent-based simulator, and harness builders can adapt it. However, the quantitative wins are small or depend on modified baselines and same-family LLM judges. The ranking and user-study claims are underpowered, and nothing is released.

  • Novelty

    3 out of 5
  • Rigor

    2 out of 5
  • Reproducibility

    2 out of 5
  • Relevance to harnesses

    4 out of 5

TL;DR

SWE-WhatIf turns a recorded user-agent coding session into a simulator of that specific user. The simulator holds an intent model (intents, acceptance criteria, raise/drop conditions) and a style profile, so the session can be replayed against new agents. On reply similarity it edges out a script-based simulator, it roughly ties that simulator on rewritten sessions, and it gives the clearest adaptation signal (66.3 win rate). All of this rests on 89 sessions, LLM judges, author-modified baselines and two replays per policy, and no code is released.

Adversarial review by Harness Hub · Nov 19, 2026 · read full text (truncated) · Suggest a correction

Original abstract (authors’ text, select to expand)

Recorded sessions between users and coding agents offer a way to evaluate how agents complete real-world tasks and interact with users. However, these sessions cannot simply be replayed with new agents, because recorded user replies may no longer apply when an agent takes a different trajectory. We introduce SWE-WhatIf, which replays recorded sessions with new agents by modeling the user's intents, the criteria for satisfying them, and the conditions under which they arise. Using this model, a user simulator generates replies that adapt to unseen agent turns while remaining grounded in the recording. Across 89 real-world sessions, SWE-WhatIf generated the replies most similar to those of real users and matched the strongest baseline when sessions were rewritten. To test adaptation, we replay sessions with agents instructed to avoid or repeat actions that the recorded user pushed back on. SWE-WhatIf produced the highest win rate for avoiding these actions (66.3%), significantly above chance. We further demonstrate its utility by ranking five LLMs as coding agents, where the highest-ranked model depended on how users specified and developed their intents, and a study with 16 users, where replaying their own sessions changed 14 participants' rankings and informed their future agent use.

Filed undercoding-agentsagents

Summary

The paper addresses a practical gap: recorded coding-agent sessions cannot be replayed with a new agent, because the user's recorded replies stop fitting once the agent's trajectory diverges. SWE-WhatIf builds an intent model from the recording using three Claude Code (Opus 5.5) agents that draft, review and revise it. The model contains: - intents, each with criteria; - conditions under which each intent is raised or dropped; - a style profile computed by deterministic code. During replay, a GPT-5.6 Luna simulator does three things after each agent turn: - assesses criteria and conditions with an LLM; - updates intent states with deterministic rules; - writes a reply in the user's style. Replays are scored by burden-adjusted success: the share of goals met, multiplied by (n_real+1)/(n_sim+1). Validation uses 89 sessions (65 from SWE-Together, 24 from SWE-chat) in three tests: - **Replication:** LLM-judged similarity to the real reply. - **Robustness:** the same comparison on paraphrased and merged sessions. - **Adaptation:** whether an agent told to avoid pushed-back actions beats one told to repeat them. Two extrinsic evaluations follow. One ranks five LLMs in opencode on 58 tasks, one replay each. The other is a 16-person Prolific study in which participants replayed their own sessions.

Key findings

  • Replication similarity (Table 1), single-turn and multi-turn: - SWE-WhatIf: 53.2 and 44.9 - SWE-Together: 51.3 and 40.4 - SWE-Interact: 35.0 and 31.4 - Goal-Prompted: 26.3 and 24.8
  • Robustness on rewritten sessions (Table 1): averages are 45.1 for SWE-WhatIf and 44.7 for SWE-Together. On multi-turn, SWE-Together is higher (41.3 vs 40.7).
  • Adaptation win rate of the proactive policy (Table 1, p before Holm correction): - SWE-WhatIf: 66.3 (p=.001) - SWE-WhatIf without style profile: 62.9 (p=.012) - SWE-Together: 55.6 (p=.314) - Goal-Prompted: 55.1 (p=.306) - SWE-Interact: 50.6 (p=1.00)
  • Task-level tallies (Section 3.3): with the style profile the proactive policy won 52 tasks and lost 23. Without it, it won 50 and lost 27.
  • Error analysis (Table 2): SWE-WhatIf replays contain extra-work turns in 52.2% of cases and premature turns in 33.1%. SWE-Together is passive in 96.1% of replays.
  • LLM ranking (Section 4): agreement with Agent Arena is Kendall tau_b .80 for SWE-WhatIf, versus .60 for SWE-Together and .53 for DeepSWE. Claude Opus 5.5 met 65.3% of goals and Sonnet 5.5 met 65.1%, with 6.4 vs 6.7 user messages on average.
  • User study (Section 5): - 14 of 16 participants changed their ranking, and 10 changed their top agent. - 12 of 16 concluded that a smaller model was enough. - Simulated replies were marked 49% 'Sounds like me', 38% 'Partly' and 13% 'Not like me'.

The paper at a glance

01Recorded session +startingenvironment02Draft / review /revise intent model(Claude Code, Opus5.5)03Deterministic styleprofile from usermessages04Replay: agent turn-> LLM assesscriteria/conditions-> rule-basedintent state update…05End when intentssatisfied/dropped/abandonedor 7 agent turnswithout change06Score: LLM-judgedgoal success x(n_real+1)/(n_sim+1)
  1. Recorded session + starting environment
  2. Draft / review / revise intent model (Claude Code, Opus 5.5)
  3. Deterministic style profile from user messages
  4. Replay: agent turn -> LLM assess criteria/conditions -> rule-based intent state update -> LLM respond in user style
  5. End when intents satisfied/dropped/abandoned or 7 agent turns without change
  6. Score: LLM-judged goal success x (n_real+1)/(n_sim+1)
Fig. 1 — How the paper works, step by step (arXiv:2611.21095).
Goal-Prompted55.1 0-100SWE-Interact50.6 0-100SWE-Together55.6 0-100SWE-WhatIf (this paper)66.3 0-100SWE-WhatIf without style profile62.9 0-100
Fig. 2 — Adaptation win rate of proactive policy over observed policy, higher is better. Source: Table 1, arXiv:2611.21095.
Data table
Adaptation win rate of proactive policy over observed policy
Method0-100
Goal-Prompted55.1 0-100
SWE-Interact50.6 0-100
SWE-Together55.6 0-100
SWE-WhatIf (this paper)66.3 0-100
SWE-WhatIf without style profile62.9 0-100

Why it matters

For people building coding-agent harnesses, the useful idea is turning real session logs into a regression test that tolerates trajectory divergence. The paper's design is concrete enough to copy: an intent/criteria/condition representation, with a rule-based state machine sitting between an LLM assess call and an LLM respond call. You could use it to evaluate a harness change against your own users' steering instead of against fully specified SWE-bench-style tasks. Burden-adjusted success is a simple way to penalize harnesses that need more user turns, though the simulator's own stopping rules feed into it. The evidence that the simulator is faithful is thin: - The replication margin over a simulator that has the recorded messages verbatim is small, and robustness is a tie. - Every judgment is made by an LLM, including models from the same families being ranked. - The model ranking uses five models with one replay per task. No code or artifacts are released, so you would have to reimplement it from the appendix prompts. Treat it as a design reference for user-simulator-based harness evals, not as a validated benchmark.

Issues (8)

  • High severity

    The baselines are author-modified, not faithful reimplementations. SWE-Interact and the goal-prompted simulator receive SWE-WhatIf's own intents and criteria in place of their original requirement sources, and SWE-Interact's tool access is removed. For the 24 SWE-chat tasks, the SWE-Together prompts were written by Claude Opus 5.5 rather than taken from the release. The comparison therefore tests prompt framings built around the proposed intent model, not the original systems.

  • High severity

    No artifacts are released: no code, intent models, goals or replays. Only task and session IDs are given. Every component runs on specific proprietary models (GPT-5.6 Luna, GPT-6.1 Sol, GPT-5.6 Terra, Claude Opus/Sonnet 5.5). Reproducing any number requires reimplementing the full pipeline from appendix prompts.

  • High severity

    The ranking is exposed to judge/model-family overlap. Intent models are built by Claude Code on Opus 5.5, and Opus 5.5 is one of the ranked models. Goals are written by GPT-5.6 Terra, and success is judged by GPT-6.1 Sol, while GPT-6 Sol and GPT-6 Luna are ranked. No cross-judge check is reported, so self-preference cannot be ruled out in the Section 4 ranking or the GPT-vs-Claude subgroup claims.

  • Medium severity

    Replication and robustness rest entirely on an LLM similarity judge (GPT-6.1 Sol, 1-5 scale, run twice). There is no human agreement check and no confidence intervals or significance tests. The margins are small: - Single-turn replication: 53.2 vs 51.3. - Robustness average: 45.1 vs 44.7. - Robustness multi-turn: SWE-Together is ahead, 41.3 vs 40.7. Calling this 'most similar' and 'matched' overstates differences that may be within judge noise.

  • Medium severity

    The adaptation test is constructed rather than natural. An LLM extracts pushed-back actions and writes 'repeat' and 'avoid' instructions. SWE-WhatIf's intent model is built from the same pushback evidence in the same recording. The test therefore mostly measures sensitivity to actions already encoded in the criteria, not open-ended adaptation. There are further concerns: - Each policy is replayed only twice per task. - Removing the style profile changes the win rate from 66.3 to 62.9, although style should not affect task success. This suggests run-to-run variance of that size. - Task selection requires at least one pushback, which biases the sample toward sessions where this test works.

  • Medium severity

    The model ranking claim is weak: - Kendall tau_b over five models, with ties allowed, so .80 vs .60 vs .53 differ by only one or two pairwise orderings. - Each task is replayed once. - Seven tasks where an agent could not finish within the time limit are dropped, which removes exactly the hard cases. - The subgroup conclusions rest on 26 and 32 tasks with no variance or tests. These include GPT models being more efficient with upfront requirements, and GPT-6 Sol ranking first without reactive intents and fourth with them.

  • Medium severity

    The user study cannot support the utility claim: - 16 Prolific participants, three author-chosen agents, up to three turns each, in partially reconstructed environments. - No control condition. - A ranking change after seeing any outputs, compared with a blind prior prediction, is expected and does not show the replays are faithful. - Only 49% of simulated replies were rated 'Sounds like me'.

  • Low severity

    The burden-adjusted metric multiplies success by (n_real+1)/(n_sim+1). It can exceed raw success, and it rewards replays where the simulator itself stops early through the 4-turn abandonment rule or 7-turn no-change termination. Simulator behavior therefore directly shapes agent scores. Message count is also a crude proxy for burden, which the authors acknowledge.

Questions for the authors

  • How well does the GPT-6.1 Sol similarity judge agree with human ratings of simulated versus real replies?
  • What are the per-task confidence intervals for replication and robustness, and is the 53.2 vs 51.3 gap significant?
  • Does the model ranking hold if success is judged by a model outside the OpenAI and Anthropic families being ranked, and if intent models are built by a non-ranked model?
  • How does the adaptation result change if SWE-Interact keeps its original tool access and requirement sources rather than SWE-WhatIf's intent model?
  • How stable are the adaptation win rates and the five-model ranking across more replays per task, and does including the seven timed-out tasks change the ranking?
  • What is the adaptation win rate on raw goal success alone, without the burden multiplier, and what are the Holm-corrected p-values?

Related in the directory

Open-source projects

Single agent harness used to rank the five LLMs in Section 4.
Agent replayed in the adaptation test and the agent that builds intent models.
SWE-chat sessions were filtered to Claude Code or Codex recordings; participants revised harness preferences between them.
DataClaw session recordings are one source of the SWE-Together tasks used.

Pi

Pi sessions (Zechner, 2026) supply many SWE-Together tasks, e.g. pi-mono task IDs.
Cited as the fully specified, no-user benchmark style this work argues is insufficient.

Commercial products

Cited as the author of opencode (Anomaly, 2025), the evaluation harness.

People

Cited author of Pi, a source of recorded sessions used as tasks.

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.

Yuzhi Huang, Jie Wu +11

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.