PaperarXiv:2611.23216 · cs.SE · Nov 17, 2026
Higher-Scoring Coding Models Did Not Get More Work Done
Yining Hua
The natural-experiment design and the within-developer framing are clearly reported, and Appendix E shows convincingly that pooled merge-rate comparisons mostly rank developers. The headline, however, is a wide-interval null on a saturated, largely self-merged metric. It is computed on a small, selected slice of PRs, with control sampling that the paper itself calls unreliable for workload. The title claims more than the data support.
Novelty
4 out of 5Rigor
3 out of 5Reproducibility
4 out of 5Relevance to harnesses
3 out of 5
TL;DR
The paper uses a 19-day export-control suspension of Claude Fable 5 as a natural experiment. In a difference-in-differences analysis of 297 users against 8,908 Opus 4.8 controls, access to Fable 5 changed the 7-day PR merge rate by +1.4 pp [-1.4; +4.3]. Five within-developer model upgrades pooled to -0.9 pp [-1.9; +0.2]. The outcome sits near a ceiling: the treated group's pre-release merge rate was 92.7%, and 95% of sampled PRs were merged by their own author. The title's 'did not get more work done' is a non-significant result on a saturated, mostly self-judged metric.
Adversarial review by Harness Hub · Nov 20, 2026 · read full text (truncated) · Suggest a correction
Original abstract (authors’ text, select to expand)
New coding models advertise themselves on score improvements of coding benchmarks. Yet no study has tested if a model that scores higher on these benchmarks gets more real work done. On June 12, 2026, a U.S. export control directive banned access to Claude Fable 5 for 19 days, which forced users to switch to other models. During this time, most former Fable 5 users switched to Opus 4.8 (11 points lower on SWE-Bench Pro compared to Fable 5). We treat this suspension as a natural experiment, where the people and projects stayed the same and only the model changed, so we could isolate the effect of Fable 5 on user productivity. We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the control group. We also repeated the comparison across five model upgrades before and after the suspension in 2026 (1,208 developers, 15,442 pull requests). Fitting a difference-in-differences model with developer and date fixed effects, we found that having access to Fable 5 increased the 7-day merge rate of pull requests by only 1.4 percentage points [-1.4; +4.3], which was not statistically significant. Similarly, no significant increases of merge rates were found with the upgrades (weighted mean: -0.9 percentage points [-1.9; +0.2]); all five estimates were negative (95% intervals included 0, did not differ from each other). We also did not observe a significant increase in the number of pull requests merged per week (+2.4% [-4.9%; +10.2%]).
Authors’ note
20 pages, 4 figures
Filed underevaluationcoding-agents
Summary
The paper identifies Claude Code PRs on GitHub whose description names the model. Fable 5 users' PRs are split into four phases: pre-release (P0), three days of availability (P1), the 19-day suspension (P2) and restoration (P3). A PR-level linear probability model with developer and calendar-day fixed effects is fitted, with developers who used Opus 4.8 but never Fable 5 as controls. The headline estimand is the reversal contrast (beta1+beta3)/2, measured against the suspension phase. A second, observational analysis compares the same developers' PRs in the week before and the week after five Opus/Sonnet upgrades, pooled by inverse-variance weighting. Weekly merged PRs are estimated only in this upgrade analysis, against developers who did not switch. A descriptive audit of 2,019 PRs found three things: - 95% were merged by their author. - 90% had no approval from another account. - The median time to merge was under half an hour. The paper concludes that benchmark gains of 5 to 18 SWE-Bench Pro points did not show up in merged work. Code and a hashed dataset are released.
Key findings
- Suspension reversal contrast: +1.4 pp [-1.4; +4.3] in 7-day merge rate (Table 2). Placebo (P0 vs suspension): -2.6 pp [-6.2; +1.1].
- Five upgrades pooled: -0.9 pp [-1.9; +0.2]. Within developer and repository the estimate is -1.2 pp [-2.3; -0.02], which is statistically significant and negative. The abstract does not report this figure (Table 2).
- Weekly merged PRs, upgrade analysis only, measured against non-switchers: +2.4% [-4.9%; +10.2%]. This outcome is not estimated for the suspension (Table 2, footnote b).
- Pre-release 7-day merge rate was 92.7% for the treated group and 91.5% for the sampled control group (Table 1).
- Of 2,019 sampled PRs, 95% were merged by the author and 90% had no approval from another account (Section 4.3).
- Only 1.0% (95% CI 0.6% to 1.7%) of sampled merged PRs whose commits named a Claude model also named it in the description (Appendix B).
- Scaling by actual use gives +2.7 pp [-2.6; +8.0]. A refit on all PRs with equal weight per developer gives +3.6 pp [+0.1; +7.1], the only significant suspension setting (Section 4.1, Appendix D).
- A naive pooled, leaderboard-style comparison has Opus 4.8 ahead by 2.6 pp. Restricted to the same developer and repository, the gap falls to -0.9 pp [-3.5; +1.7]. Developer-level ICC is 0.42 (Appendix E).
The paper at a glance
- GitHub Search for Claude Code PRs whose description names exactly one model
- Treated: developers naming Fable 5 in P1; control: developers naming Opus 4.8 but never Fable 5
- Split PRs into P0/P1/P2/P3 by creation time and exclude the days around each switch
- PR-level LPM with developer and calendar-day fixed effects; reversal contrast vs the suspension
- Upgrade analysis: same developers, week before vs week after, inverse-variance pooled
- Audit 2,019 PRs for merger identity, approvals, checks and reverts
Data table
| Method | percentage points |
|---|---|
| Pooled, unadjusted (leaderboard-style) | -2.6 percentage points |
| Pooled + calendar-day FE | -2.3 percentage points |
| Within developer | -2.1 percentage points |
| Within developer and repository | -0.9 percentage points |
| Suspension DiD reversal contrast (this paper) | 1.4 percentage points |
Why it matters
For harness builders the measurement lessons are worth more than the headline. First, the PR merge rate from public Claude Code traffic is a near-saturated signal that authors mostly decide for themselves. Appendix E shows that pooled merge-rate comparisons mostly rank developers, not models, with a developer-level ICC of 0.42. Anyone building agent dashboards, model routers or evals on merged-PR counts should take that seriously and compare within the same developer and repository. Second, the result suggests that for the self-merged work people currently delegate, swapping to a higher-scoring model produced no detectable change in merges, checks or reverts. The paper does not test why, and it does not show that harness changes would do better. Third, identifying the model from co-author lines, together with the released dataset, can be reused for field evaluation. The paper cannot speak to harder or externally reviewed work, long-horizon autonomous runs, or time saved, because the outcome does not observe those. It does not study harness design.
Issues (9)
- High severityWhere in paper: Table 1; Section 4.3; Appendix A ('Because almost all PRs were already merged')
The outcome is near a ceiling and is mostly the author's own decision. The pre-release 7-day merge rate was 92.7% (treated) and 91.5% (control). 95% of sampled PRs were merged by the author, 90% had no approval from another account, and the median time to merge was under half an hour. A merge here mostly records that a developer merged their own PR, so 'work done' rests on a metric with little headroom and no external quality check. The paper's own outcomes with more headroom do not settle this. Merged-within-1-hour for the suspension is 0.0 [-5.8; +5.8], and the subset of developers with unmerged PRs is -1.3 [-6.9; +4.3]; both are too wide to exclude meaningful effects.
- High severityWhere in paper: Title; Abstract; Section 4.1; Appendix A (weightings); Appendix D
The title and abstract turn a non-significant estimate into a negative claim. The suspension interval [-1.4; +4.3] does not rule out a gain of several points. Scaling by actual use gives +2.7 [-2.6; +8.0], P3 alone gives +8.5 [-6.3; +23.4], and one weighting is significantly positive at +3.6 [+0.1; +7.1]. The paper sets no smallest effect of interest and does not say what merge-rate change an 11-point SWE-Bench Pro gap should produce, so the null cannot be read as 'no effect'. The supportable conclusion is 'no detectable change in a saturated self-merge metric'.
- High severityWhere in paper: Appendix B; Appendix F ('Fewer PRs named the model'); Section 5 limitations
The sample is heavily selected. Only 1.0% of sampled merged PRs whose commits named a Claude model also named it in the description. Among treated developers, the share of PRs with a Claude line fell from 16.5% in P1 to 6.7% during the suspension, so the suspension estimate rests on about 7% of their PRs. Of 253 treated developers active during the suspension, only 84 opened a PR with a Claude line. If tasks moved to other tools during the suspension, the phase contrast mixes in task composition. The refit on all PRs (+0.5 [-2.3; +3.3]) is diluted, because Fable 5 was only 14% of P1 and 2% of P3 in that sample.
- Medium severityWhere in paper: Section 3.2; Section 3.3; Appendix F ('Workload during the suspension')
Treated and control PRs enter under different rules. Treated PRs include all PRs with a Claude co-author line, while control developers are defined by PRs naming Opus 4.8. The control sampling also behaves oddly: control PRs per developer per day fell from 6.5 to 4.1 to 3.4 across phases, which the paper attributes to how controls were sampled. Without a clear argument that this sampling does not also affect merge-rate trends, the parallel-trends assumption behind the DiD is weaker than presented.
- Medium severityWhere in paper: Section 3.1; Appendix A (upgrade inclusion rule); Section 5
The upgrade analysis is observational and leaves the analyst room for choices. The February Opus upgrade was dropped from the merge-rate pool because it had been seen in an earlier exploratory analysis. Windows are one week. The Opus 4.7 week overlaps a Claude Code quality problem (April 16-20). Sonnet 4.6 to 5, the one positive estimate (+1.8 [-0.6; +4.3]), was excluded by the outside-author rule. Alternative poolings range from -1.4 to 0.0. The abstract presents this analysis as corroboration, but it lacks the exogenous assignment of the suspension.
- Medium severityWhere in paper: Abstract; Table 2 footnote b; Appendix F
The abstract reports weekly merged PRs (+2.4%) as if it covered throughput broadly. The paper does not estimate it for the suspension because no reliable control exists for that outcome. The throughput claim therefore rests only on the observational upgrade analysis.
- Medium severityWhere in paper: Appendix A (phase asymmetry); Appendix D (P3 special case); Appendix F
The phases are asymmetric and include a launch effect. P1 lasts only three days right after release, when developers named the model more often and did unusual work. Fable 5 usage in P3 is only 26.9%. A further 33.7% of P3 PRs carry an unnamed Claude line, 82% of them from one developer's custom footer, and the reversal contrast counts these as treated.
- Low severityWhere in paper: Appendix A (check failures); Appendix F (reverts)
The quality-side evidence is thin and partly points the other way. Test and build failures were 6.4 pp higher when Fable 5 was available [+0.1; +12.8], although this has no comparison group and failure rates swing between 8% and 40% across phases. Reverts total 17 of 5,878 merged treated PRs, too few to detect anything. In the upgrade analysis, Opus 4.7 to 4.8 had significantly more check failures (+5.4 [+1.1; +9.7]).
- Low severityWhere in paper: Section 4.1; Appendix A (weightings); Appendix F (refit on all PRs); Table 1
Some related robustness estimates are hard to reconcile and are not explained. On all PRs, equal weight per developer gives +3.6 [+0.1; +7.1], while computing each developer's mean first gives +1.0 [-1.1; +3.3]. On named PRs, equal weight per developer gives +3.7 [-5.5; +12.9]. Table 1 balance uses 244 treated and 315 sampled control developers, not the 8,908 controls used in the DiD.
Questions for the authors
- What is the suspension estimate restricted to repositories where a non-author account merges or approves PRs, even if the sample is small?
- Why do 'equal weight per developer' (+3.6) and 'developer mean first' (+1.0) differ on the same all-PR refit, and which one is the intended estimand?
- Does the control-group sampling that produced falling PR counts (6.5, 4.1, 3.4 per day) also shift control merge rates over the window?
- What did the excluded February Opus upgrade show in the exploratory analysis, and how does including it change the five-upgrade merge-rate pool?
- What minimum merge-rate effect was the design powered to detect, and how does it compare with an effect one would expect from a 10.8-point SWE-Bench Pro gap?
- Do outcomes that are not bounded near 100%, such as follow-up fix PRs touching the same files within N days, show model differences?
Related in the directory
Open-source projects
Open source · Anthropic
Open source · SWE-bench
Open source · Harbor
People
Person · Official YouTube channel
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26444 · cs.RO · Nov 19, 2026
Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.
Yuzhi Huang, Jie Wu +11