PaperarXiv:2611.20882 · cs.AI · Nov 16, 2026
Accelerate Bench: Evaluating Agents for Application-Level Program Optimization
Bowen Liu, Yuhan Li, Nuo Chen, Weile Luo, Miao Peng, Qintong Zhang, Qifan Zhang, Jia Li, Ziqi Gao, Lydell Huang
The benchmark is carefully built: tasks are workload-defined, final scoring uses private verification, timing noise is calibrated, and variance is reported honestly. Several process observations are actionable for harness builders. However, the leaderboard is a single-run, two-harness comparison in which only the top separation is resolved. The abstract's interaction numbers are single-case replays from v1.0, verifier hardening is tested on two tasks, and nothing is released yet.
Novelty
3 out of 5Rigor
3 out of 5Reproducibility
2 out of 5Relevance to harnesses
4 out of 5
TL;DR
Accelerate Bench is a benchmark of 32 CPU application-optimization tasks adapted from supercomputing competitions. Each agent gets an 8-hour budget, public tests during development and a private verifier at the end. On v1.1, 16 model-plus-harness agents score 25.5% to 67.4% Score@1, one run each. Among neighbors in the ranking, only the two Claude agents in Claude Code are separated by the paper's bootstrap intervals. Process findings come from a separate v1.0 sample of five weaker agents.
Adversarial review by Harness Hub · Nov 19, 2026 · read full text (truncated) · Suggest a correction
Original abstract (authors’ text, select to expand)
Application optimization requires coding agents to discover useful changes and coordinate them across a complete program while preserving the required computation. We introduce Accelerate Bench, a benchmark of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task provides a working program and specifies the workload, correctness requirements, timing boundaries, and permitted edits, leaving agents to choose where and how to optimize. Public testing supports iterative development, and a separate verifier scores the final submission. On benchmark version 1.1, we evaluate 16 coding agents, each pairing a model with a harness. Their Score@1 ranges from 25.5% to 67.4%. Most of their programs are correct, and the size of the speedup separates them: the two leading agents are at least 2x faster on 22 or more of the 32 tasks, while the other fourteen reach 2x on 8 to 17 and leave 125 of their 412 correct outcomes below 1.25x. Controlled replays show why coordination across an application matters: identical source code runs between 0.29x and 2.12x depending only on build and runtime settings, and one code change is worth 1.01x or 6.4x depending on another. Accelerate Bench pairs executable workloads with agents' programs and trajectories to evaluate both their gains and the process behind them.
Authors’ note
49 pages, 8 figures. Work by Bowen Liu was done during an internship at Tencent Hy Team. Yuhan Li is Project Lead; Nuo Chen is corresponding author (chennuo26@gmail.com). Project page: https://acceleratebench.com/
Filed underharnessevaluationcoding-agentsruntimeagents
Summary
Accelerate Bench defines each task by a fixed workload and its required outputs, not by a reference human patch. Each task ships a working baseline, build and run scripts, and a public test. Agents may edit source, compiler flags, dependencies and runtime settings inside a permitted directory. A private verifier rebuilds the submission and times it against the unmodified program on held-out inputs (typically four), which are often larger or freshly randomized. The reward is the fraction of baseline runtime saved, C * max(0, 1 - T/B). Score@1 is the mean reward over the 32 tasks. On v1.1, each of 16 agents runs once per task. Three GPT models run in Codex and all other models run in Claude Code. Claude Sonnet 5.5 (67.4) and Claude Opus 5.5 (59.4) lead. Below them, paired task-bootstrap intervals separate no two neighbors. Repeated runs exist only for GPT-6 Astra and Hy4-Preview. A separate v1.0 analysis covers 480 runs of five agents. It uses LLM-annotated trajectories plus controlled replays of individual submitted programs. It reports three things: - Agents mostly change computation and rarely the build. - Build and runtime settings move one unchanged GROMACS program between 0.289x and 2.123x. - In one QiboTN program, deferred loading is worth 1.01x alone and 6.3-6.4x once broader simulator dispatch is enabled.
Key findings
- Table 1 (v1.1, one run per agent): Score@1 ranges from 25.5 (Hy3) to 67.4 (Claude Sonnet 5.5). Claude Opus 5.5 scores 59.4, or 65.6 on the 29 tasks the provider's safety system did not stop.
- Section 4: 472 of 512 agent-task outcomes pass correctness. Each agent is correct on 26 to 32 tasks; GPT-6.1 Terra is correct on all 32 yet scores 37.9.
- Appendix A.4: Claude Sonnet 5.5 leads every non-Claude agent by at least 13.4 points at the lower interval end. Claude Opus 5.5 leads GPT-6.1 Sol by 10.2 (0.1-18.2). No two neighbors below them are separated, and 64 of 120 pairs have intervals excluding zero.
- Appendix A.3 Table 4: re-judging fixed submissions gives a Score@1 SD of 1.45 (GPT-6 Astra) and 1.57 (Hy4-Preview). The SD across three independent runs is 2.48 and 4.42. Correctness flipped for 3/96 and 7/77 re-judged submissions.
- Appendix A.2: a configuration-only baseline (no source edits) scores 3.85%. Only GROMACS exceeds 1.25x under it (2.225x in Table 3). Using it as the reference lowers each agent's Score@1 by 0.5 to 2.4 points.
- Section 5.1 (v1.0, 476 labeled runs): computation or reuse changes appear in 225 runs and compiler or build tuning in 40.
- Section 5.3: one unchanged GROMACS source runs from 0.289x to 2.123x across four build and runtime settings. In one DeepSeek-V4-Flash QiboTN program, deferred loading is worth 1.01x alone and 6.3-6.4x with broader simulator dispatch.
- Section 5.2: every agent's share of test calls rises late in the run. Hy3 has the highest late share (21.7%) but reaches 1.25x in only 16 of 96 runs.
The paper at a glance
- Working baseline program + workload contract + permitted edit scope
- Agent loop (8h): inspect, edit source/flags/runtime config, call public test
- Public test: rebuild, run ~2 public inputs, report correctness and speedup
- Final submission frozen
- Private verifier: rebuild, alternate timing vs unmodified program on ~4 held-out inputs, correctness and stateful checks
- Reward = C * max(0, 1 - T/B); Score@1 = mean over 32 tasks
Data table
| Method | % |
|---|---|
| Claude Sonnet 5.5 (Claude Code) | 67.4% |
| Claude Opus 5.5 (Claude Code) | 59.4% |
| GPT-6.1 Sol (Codex) | 49.2% |
| GPT-6 Astra (Codex) | 47.2% |
| Hy4-Preview (Claude Code) | 45.2% |
| Hy4 (Claude Code) | 45% |
| Kimi K3 (Claude Code) | 42.5% |
| DeepSeek-V4.1-Flash-1017 (Claude Code) | 40.1% |
| GLM-5.3 (Claude Code) | 39.7% |
| GLM-5.3-Flash (Claude Code) | 39.3% |
| DeepSeek-V4-Pro (Claude Code) | 37.9% |
| GPT-6.1 Terra (Codex) | 37.9% |
| DeepSeek-V4-Flash (Claude Code) | 34.1% |
| GLM-5.2 (Claude Code) | 34% |
| Grok 4.6 (Claude Code) | 33.9% |
| Hy3 (Claude Code) | 25.5% |
Why it matters
For coding-agent harness builders, this is one of few long-horizon evaluations that targets end-to-end runtime of a whole application. There is no reference patch, and a hidden verifier runs on unseen inputs. The process findings point at harness features rather than model choice. Agents rarely touch build and runtime configuration, even though a single GROMACS source moves from 0.289x to 2.123x on settings alone. In one HPL case an agent optimized a function the binary never called. A Palabos run saw all 192 host CPUs and ran 11 minutes instead of 1.5 s. Restore or select operations appear in 206 of 480 trajectories. Together these suggest profiler and binary-diff tools, explicit core-count and pinning awareness, and best-candidate tracking as harness responsibilities. The caveat is that most of these are single-case observations, and the analysis uses five weaker agents on an older benchmark version. The v1.1 leaderboard says little about harnesses: only Claude Code and Codex are used, non-OpenAI models are all placed in Claude Code, and each agent runs once. The table cannot tell you whether a harness helps or hurts a given model.
Issues (10)
- High severityWhere in paper: Section 4 Experimental setup; Appendix A 'Agents and run validity'; Section 7
Harness and model are confounded, and the design may favor the leaders. Claude models run in their native harness. Grok, DeepSeek, GLM, Kimi and Hy models are also placed in Claude Code, and v1.1 moved Grok 4.6 off Grok Build (and Gemini-3.8-Flash off Gemini CLI). Only GPT models get their vendor harness. No harness ablation is run, so the Claude lead may partly reflect harness-model fit. The abstract says each agent pairs a model with a harness, but it then reports the ranking as a ranking of 'coding agents'.
- High severityWhere in paper: Table 1 caption; Appendix A.3 Table 4; Appendix A.4
Each agent has one scored run per task. Independent-run SD is 2.48 to 4.42 points, and re-judging the same programs adds about 1.5 points. Repeated runs exist for only two agents. Below the two Claude agents no neighbors are separated, yet the abstract groups 'the other fourteen' and quotes a 25.5-67.4 spread, which reads as a resolved ordering.
- Medium severityWhere in paper: Abstract; Section 5.3; Appendix L.2
The abstract's mechanistic numbers come from single replayed artifacts. The 0.29x-2.12x figure is one GROMACS run and the 1.01x vs 6.4x figure is one DeepSeek-V4-Flash QiboTN program, both from the v1.0 analysis sample, and the paper does not state how they were chosen. They are existence proofs presented as showing 'why coordination across an application matters'. The paper itself says the analysis does not explain the v1.1 ranking. The text gives 6.3-6.4x; the abstract gives only the upper value.
- Medium severityWhere in paper: Section 2.3; Appendix E; Appendix A.3
Verifier robustness against shortcut submissions is validated on only 2 of 32 tasks (14 planned controls on HPCG-derived and paged generation). For the remaining 30 tasks, the claim that invalid optimizations fail rests on design and student-finalist review, not on tested adversarial variants. Fresh private inputs also flip correctness on near-tolerance programs: 3/96 and 7/77 re-judged submissions changed correctness.
- Medium severityWhere in paper: Section 7 'Task coverage and baselines'; Appendix A.2 Table 3
Baselines are unevenly tuned, so part of each speedup measures slack in the starting program. GROMACS ships with its SIMD option set to None, and a config-only recipe gives 2.225x. HPCG-derived admits a direct solver, and GPT-6 Astra reaches 28.9x-36.7x there. The configuration reference changes the reference on only nine tasks and is floored at 1x.
- Medium severityWhere in paper: Section 2.2; Appendix D.1
Contamination is not assessed. All tasks derive from public supercomputing competitions (ASC, PAC, IPCC, SCC, IndySCC) whose applications and optimization write-ups are plausibly in pretraining data. Large wins on known structure, such as the HPCG-derived direct solver, may reflect recall rather than discovery.
- Medium severityWhere in paper: Appendix B.3; Table 1; Table 5
Compute is uncontrolled. In v1.0, continuation runs inherit session history but get a fresh 8-hour timeout. In an earlier 842-run snapshot, the median parent-plus-continuation time was 2.92 hours, and 10 of the 91 chains exceeded eight hours. Kimi K3's scored record is a continuation on 16 tasks. Table 1 reports no token or step usage for v1.1, and v1.0 usage is missing for three agents.
- Medium severityWhere in paper: Reproducibility Statement; Section 1
Nothing is released yet. Task packages and verifiers are promised 'with the published version'. Only a project page (https://acceleratebench.com/) is printed, so results cannot currently be reproduced.
- Low severityWhere in paper: Table 1 footnote; Appendix A 'Runs stopped by the provider's safety system'
Claude Opus 5.5 loses three tasks to provider safety stops (59.4 vs 65.6). The evaluation gateway routes every request to the evaluated model, which prevented Claude Code's documented fallback model from taking over. The losses are scored as agent failures.
- Low severityWhere in paper: Section 5; Appendix I; Appendix K.2
The trajectory annotator, DeepSeek-V4.1-Flash, is also one of the five analyzed agents. Technique labels come from GPT-5.6 Luna. In the reviewed sample, 15 of 40 trajectories without a flagged dependency still showed one on review, which indicates a high annotator miss rate.
Questions for the authors
- How do Grok 4.6, DeepSeek, GLM or Kimi models score in their native or a neutral harness compared with Claude Code on v1.1?
- What are token, step and wall-clock usage per agent on v1.1, and does Score@1 track compute consumed?
- Can at least three runs be reported for every agent, or at least the top four, to test whether the Claude lead and mid-table order hold?
- How were the GROMACS and QiboTN artifacts chosen for the replays, and how often do comparable interactions appear across tasks?
- Were shortcut-style invalid variants tested on any task beyond HPCG-derived and paged generation?
- What fraction of top speedups match optimizations documented in public competition reports for the same applications?
Related in the directory
Open-source projects
Open source · Anthropic
Open source · OpenAI
Open source · xAI
Open source · Google
Open source · Harbor
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26450 · cs.AI · Nov 19, 2026
Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).
Tanmay Sah