Harness Hub
All papers

Beyond Leaderboard Ranks: Selecting Software Engineering Agents Under Budget Constraints

Srinath Srinivasan, Tim Menzies

Verdict: Useful

This is a careful, self-critical reanalysis of public leaderboard traces, and it gives harness builders concrete warnings. Small eval sets cannot resolve gaps of 12.5 to 20.7 pp. Cache accounting can inflate reported costs several times over. Difficulty-filtered subsets understate cost. Successive halving is a sensible budget-allocation baseline. The headline efficiency numbers, though, depend on oracle task-cost knowledge and a lenient selection metric. Policy differences are untested, and nothing is validated on fresh executions.

  • Novelty

    3 out of 5
  • Rigor

    3 out of 5
  • Reproducibility

    3 out of 5
  • Relevance to harnesses

    4 out of 5

TL;DR

A reanalysis of 58,197 HAL run-task records. On agent leaderboards, 27 of 35 accuracy-cost frontier steps are not statistically resolved, and published costs can be off by up to 11.7x. The headline result, that successive halving needs only 6.6% of exhaustive evaluation spend, assumes the evaluator knows each task's exact cost before running it. A live evaluation does not have that information.

Adversarial review by Harness Hub · Nov 18, 2026 · read full text · Suggest a correction

Original abstract (authors’ text, select to expand)

Teams that adopt LLM agents for issue resolution must decide which agent to pay for, and how much to spend finding out. Leaderboard ranks give weak evidence for this decision. Accuracy differences may be statistically unresolved, reported costs may be inconsistent, and evaluating candidate agents can itself be expensive. We investigate budget-constrained agent selection using 58,197 run--task records extracted from the Holistic Agent Leaderboard (HAL), focusing on SWE-bench and six additional benchmarks. Across seven benchmarks, 27 of 35 steps along the accuracy--cost Pareto frontiers are statistically unresolved. On SWE-bench, a run costing $34.16 trails the $170.78 leader by 14.6 percentage points, but the available tasks cannot statistically distinguish their accuracy. Reported costs also differ from standardized token-based estimates by up to 11.7x, while selective task evaluation can bias estimates of full-benchmark execution cost. We therefore compare evaluation strategies under explicit evaluation budgets and deployment-cost caps. Under advance knowledge of task costs, successive halving reaches our uncertainty-aware selection target on SWE-bench using 6.6% of exhaustive evaluation spend, achieving the lowest cost on six of seven benchmarks and tying on the seventh. Its advantage narrows when task costs are unknown before execution or exact-best selection is required. Our findings motivate decision-oriented agent evaluation that accounts for accuracy uncertainty, validates execution costs, and compares complex selection methods against simple adaptive allocation.

Filed underevaluationcoding-agentsagentsllm

Summary

The authors rebuild per-task outcome and cost matrices from public HAL traces. They keep seven benchmarks: 262 runs and 20,633 cells. SWE-bench Verified Mini is the primary case, with 34 runs on 48 shared tasks. They reprice the token logs with a standardized price sheet. Runs are compared pairwise with exact McNemar tests and Benjamini-Hochberg correction, and the authors count how many Pareto-frontier steps are statistically resolved. They also measure: - how far published costs diverge from reconstructed costs; - how much accuracy-oriented task sampling biases full-benchmark cost estimates. Finally, they replay five evaluation policies against the historical matrices under dollar budgets: successive halving, random tasks, mid-range filtering, cell-level D2H and row-level EZR. The success measure is a selection score Q. Q gives credit to any truly affordable run whose accuracy is statistically unresolved from the best affordable run. The paper proposes no new method. It is an empirical study of how much statistical evidence and evaluation spend agent selection needs.

Key findings

  • On SWE-bench Verified Mini (34 runs, 48 tasks), 225 of 561 pairwise comparisons (40.1%) are unresolved after BH correction. 3 of 4 frontier steps are unresolved (Section IV-A, Table I).
  • Across seven benchmarks, 27 of 35 accuracy-cost frontier steps are unresolved. USACO is the only benchmark with no unresolved step (Table I).
  • A $34.16 run trails the $170.78 leader by 14.6 percentage points but is not statistically distinguishable from it. The best-case resolution floor at n=48 is 12.5 pp, and the simulated 80%-power gap is 20.7 pp. Only 4.8% of pairs have 90% intervals within +/-10 pp (Section IV, Tables I and II).
  • Across 333 runs, the published-to-reconstructed cost ratio has a median of 1.00, a 95th percentile of 5.00 and a maximum of 11.7x. The 11.7x and 9.9x cases come from apparent double-counting of cached tokens in two Anthropic runs. Stale o3 pricing produces a 5.0x gap (Section V-A).
  • Oracle mid-range task filtering gives a median estimated-to-true cost ratio of 0.796 on SWE-bench, against 0.985 for random-task sampling. Tasks with pass rates below 0.30 cost $4.38 per cell, versus $2.59 for mid-range tasks (Table III, Section V-B).
  • With advance cost checks, SH reaches Q=0.80 on SWE-bench at $353.62 out of an exhaustive cost of $5,399.89 (6.55%). D2H reaches it at $379.19 and EZR at $942.20 (Table IV, Section VI-A).
  • Without advance cost checks, SH has the lowest spend on only four of seven benchmarks. Every SH replicate overruns a 10% SWE-bench budget, with a mean overrun of $3.04 (Section VI-C).
  • At a 25% budget on SWE-bench, every SH selection is affordable and mean regret is 1.4 pp. SH picks the exact-best affordable run in only 60.5% of cases (Table V).

The paper at a glance

01Extract 58,197run-task recordsfrom 379 HAL tracearchives02Filter to 7benchmarks withcomplete binaryoutcomes and tokenlogs (262 runs,…03Reconstructper-cell API costwith a standardizedprice sheet andcache allocation04Run paired exactMcNemar tests withBH correction;build accuracy-costPareto frontiers05Compare publishedvs reconstructedcosts; measurecost-estimate biasof task samplers06Replay SH, random,mid-range, D2H andEZR policies underdollar budgets anddeployment caps07Score selections byQ, affordability,regret andexact-best rate
  1. Extract 58,197 run-task records from 379 HAL trace archives
  2. Filter to 7 benchmarks with complete binary outcomes and token logs (262 runs, 20,633 cells)
  3. Reconstruct per-cell API cost with a standardized price sheet and cache allocation
  4. Run paired exact McNemar tests with BH correction; build accuracy-cost Pareto frontiers
  5. Compare published vs reconstructed costs; measure cost-estimate bias of task samplers
  6. Replay SH, random, mid-range, D2H and EZR policies under dollar budgets and deployment caps
  7. Score selections by Q, affordability, regret and exact-best rate
Fig. 1 — How the paper works, step by step (arXiv:2611.21515).

Why it matters

Anyone who runs a coding-agent harness and compares model or scaffold configurations faces the problem this paper quantifies. On the 48 shared SWE-bench tasks, gaps below 12.5 pp cannot be resolved even in the best case. Realistic disagreement patterns raise the gap needed for 80% power to 20.7 pp. Internal A/B tests of harness changes on small task sets are therefore mostly uninformative unless the effect is large. The cost findings are directly actionable: - Log cache-read and cache-write tokens separately. - Reprice all runs at one fixed rate card. - Do not estimate deployment cost from a difficulty-filtered subset, because the hard tasks are the expensive ones. Successive halving is a reasonable default for spreading an eval budget across configurations. Its measured advantage, however, depends on knowing task costs in advance and on a lenient selection target. Treat it as a baseline to beat, not a settled recipe. The paper says nothing about harness design itself: scaffolds, tool loops or memory.

Issues (8)

  • High severity

    The headline efficiency claims (6.6% of exhaustive spend; lowest cost on six of seven benchmarks plus one tie) hold only under 'advance cost checks'. In that setting the policy sees each cell's exact historical cost before buying it, which the paper itself calls idealized. Without it, SH leads on only four of seven benchmarks, and every SH replicate exceeds a 10% SWE-bench budget. The abstract's wording, 'advantage narrows', understates how much of the result rests on oracle cost information.

  • High severity

    The selection score Q gives full credit to any affordable run that is statistically unresolved from the best one. It also drops caps where a policy makes no recommendation. On low-resolution benchmarks the target is close to free: on SciCode every pair is unresolved, so policies tie at the first budget level. Even on SWE-bench, SH at a 25% budget picks the exact-best affordable run only 60.5% of the time. Q=0.80 therefore measures 'not provably worse on 48 tasks' more than 'a good choice'.

  • Medium severity

    The abstract says SH's advantage narrows when exact-best selection is required. The text reports exact-best rates only for SH (Table V) and gives no exact-best comparison against the other policies. That part of the abstract claim is not supported by numbers in the paper.

  • Medium severity

    Policy comparisons are crossings of interpolated mean curves, not tested differences. The paper says so itself: 'rather than statistically established differences'. On SWE-bench, SH vs D2H is $353.62 vs $379.19. Under median-based cost estimation the order reverses: D2H needs $449.04 and SH $477.87. The claim that SH 'leads' rests on small margins that change with the estimator.

  • Medium severity

    All policies, heuristics, thresholds and ablations were developed and evaluated on the same historical HAL corpus. There is no held-out benchmark and no fresh execution. The replay fixes outcomes, which ignores agent stochasticity, and only four repeated-configuration groups exist to estimate run-to-run variance. At tight budgets, the SH implementation can eliminate runs that have no observation at all, which may disadvantage expensive configurations.

  • Medium severity

    The software-maintenance conclusions rest on a single benchmark: 48 tasks of SWE-bench Verified Mini and 34 runs drawn from a convenience sample. Tasks are treated as independent despite clustering by repository. Test-based 'resolved' labels are taken as ground truth. The other six benchmarks use simplified binary outcomes, for example exact match for AssistantBench and full task success for SciCode and CORE-Bench, so they measure something different from their native grading.

  • Medium severity

    The $34.16 vs $170.78 example is framed as a case where a fivefold cost premium may not be justified. The paper's own numbers point the other way: the MDE is 20.7 pp, and only 4.8% of SWE-bench pairs have intervals within +/-10 pp. A 14.6 pp gap could well be real. Non-rejection here mostly reflects low power, not evidence that the cheaper run is competitive. The abstract leans on this example.

  • Low severity

    The abstract leads with the 11.7x cost discrepancy. That figure is the maximum, from apparent cache double-counting in two Anthropic runs; across 333 runs the median ratio is 1.00 and the 95th percentile is 5.00. The reconstruction also depends on fitted price periods and cache-allocation assumptions. These are documented only in a replication artifact, and the text gives no URL for it.

Questions for the authors

  • In the prospective setting (no advance cost checks) on SWE-bench, how does SH compare with random-task sampling in dollars needed to reach exact-best selection rather than Q?
  • How do the other policies' exact-best rates compare with SH's 60.5% on SWE-bench at a 25% budget?
  • If every candidate run is guaranteed at least one observation before elimination, does SH's ranking against D2H change?
  • Given the observed disagreement patterns, what fraction of SWE-bench pairwise non-resolutions would persist on the full 500-task Verified set?
  • Were policy hyperparameters (SH halving schedule, D2H weight 0.15, MR band 0.30-0.70) fixed before the results were seen, and how do results change on a held-out benchmark?
  • Will the replication artifact, including the price-sheet and cache-allocation assumptions, be released at a stable URL?

Related in the directory

Open-source projects

Primary benchmark; the analysis uses 48 shared tasks of HAL's SWE-bench Verified Mini subset.
Cited as evidence that the scaffold and its interface, not only the model, drive repair success and cost.
TAU-bench Airline is one of the general tool-use benchmarks used to test whether the findings generalize.

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).

Tanmay Sah

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.