Harness Hub
All papers

Scientific Coding with Large Language Models: Evidence from Finite Element Analysis

Erfan Hamdi, Joel Shor, Emma Lejeune

Verdict: Useful

This is a careful benchmark extension that reports a reliability metric (Pass^3) and a concrete self-verification failure taxonomy that harness designers can act on. Its strongest claim is that a coder/tester split yields the best reliability, but that rests on an uncontrolled exploratory comparison using different models and a larger iteration allowance. No result carries uncertainty estimates, and the agentic study omits the strongest single-pass models.

  • Novelty

    3 out of 5
  • Rigor

    2 out of 5
  • Reproducibility

    3 out of 5
  • Relevance to harnesses

    4 out of 5

TL;DR

FEM-Bench gains 17 harder finite element tasks: nonlinear, history-dependent and dynamics. The authors test 15 models single-pass and 6 models in two agentic harnesses. Agents raise Pass@3, but no agentic run exceeds 78.8% Pass^3, which is below the 79.4% Pass^3 of the best single-pass model (Fable 5.1). An LLM-judged audit of 249 failed agentic runs finds narrow tests and generic sanity checks dominate. An exploratory coder/tester split posts the top numbers (97.6% Pass@3, 88.8% Pass^3). It was run only with Fable 5.1 and GPT-6 Astra, and there is no matched single-agent control for those models.

Adversarial review by Harness Hub · Nov 19, 2026 · read full text (truncated) · Suggest a correction

Original abstract (authors’ text, select to expand)

Large language models (LLMs) offer a potential route to automating scientific programming, but their usefulness depends on consistently producing numerically correct code. We extend FEM-Bench to 17 finite element tasks involving nonlinear and history-dependent mechanics and structural dynamics, and evaluate 15 models on implementation and test generation without execution feedback. The best model succeeds in 87.1% of single-pass implementation trials, but rankings differ between tracks. The leading implementer achieves 49.9% joint test success, while the leading tester reaches 82.6%. For six models, we compare single-pass generation with two agentic harnesses that allow code execution and revision. Agentic execution solves some tasks that never succeeded in single-pass trials and improves consistency on others, but gains vary across models and harnesses, and regressions also occur. Claude Opus 5 with SimpleAgent reaches 96.5% Pass@3, the probability of at least one success in three attempts, but 77.1% Pass$^3$, the probability that all three succeed. No model-harness combination exceeds 78.8% Pass$^3$. To understand why agents submit incorrect code after passing their own checks, we analyze 249 failed agentic trajectories and identify six categories of self-verification failure. The most common are inadequate test coverage (73.1%) and reliance on generic consistency checks (70.3%), and in 28.5% of cases the tests share an error with the implementation. Motivated by these findings, an exploratory configuration with separate coder and tester agents achieves the highest observed scores, with 97.6% Pass@3 and 88.8% Pass$^3$. These results identify consistency across attempts and independent verification as the main obstacles to trusting LLM-generated scientific code.

Authors’ note

28 pages, 7 figures, 14 tables

Filed underharnessagentsllm

Summary

The authors extend FEM-Bench with 17 finite element tasks: elastoplastic updates, hyperelasticity, cohesive zones, Kirchhoff plates, mesh generation and Newmark dynamics. Several tasks come in helper-availability tiers (H/T notation). There are two single-pass tracks: - **Implementation**, scored as function correctness (empirical Pass@1) against reference outputs on prescribed inputs. - **Test generation**, scored as joint test success. A test case must accept the reference and reject the designated faulty implementations, with partial credit per trial. All 15 models run both tracks with 5 trials per task. Six models (Claude Opus 5, GPT-5.6 Sol, GPT-5.6 Luna, Kimi K3, GLM-5.2, DeepSeek V4 Flash 0731) also run the implementation track in two harnesses: - **SimpleAgent**: the authors' own design. Python execution runs in a fresh, stateless sandbox, there is no internet, and submission is allowed only after at least one execution whose latest exit code is 0. - **OpenHands**: persistent workspace, internet enabled, run inside a Singularity container. Both harnesses get 2,000,000 tokens, 200 iterations and 2 hours per trial. Results are reported as Pass@3 and Pass^3 estimated from 5 trials. The 249 completed but incorrect agentic submissions are labeled with a six-category self-verification codebook. Three LLM judges label each case, and GPT-6 Astra adjudicates disagreements. Finally, an exploratory coder/tester extension of SimpleAgent runs with Claude Fable 5.1 and GPT-6 Astra in both role assignments. The tester: - sees only compiled .pyc code, - keeps a persistent pytest suite, - sends written feedback plus failing test names, - alone approves submission.

Key findings

  • Single-pass function correctness ranges from 32.9% (DeepSeek V4 Flash 0731) to 87.1% (Claude Fable 5.1). GPT-6 Astra and Gemini 4 Argon each reach 82.4% (Section 5.1, Table 4).
  • Test generation ranks models differently. GPT-6 Astra leads joint test success at 82.6%, while Fable 5.1, the top implementer, gets 49.9%. GPT-5.6 Luna is third in test generation (74.3%) but has 54.1% function correctness (Section 5.1).
  • Per harness, across 102 model-task combinations: SimpleAgent newly solves 10, improves 31, degrades 11 and loses 1. OpenHands newly solves 9, improves 22, degrades 13 and loses 5 (Section 5.2, Figure 4).
  • Claude Opus 5 with SimpleAgent rises from 90.6% to 96.5% Pass@3 and from 65.9% to 77.1% Pass^3. With OpenHands its Pass^3 is 78.8%, the best agentic Pass^3. Recorded cost is $38.54 for SimpleAgent versus $74.15 for OpenHands (Sections 1, 5.2).
  • GPT-5.6 Luna with SimpleAgent: Pass@3 rises from 70.6% to 80.6%, but Pass^3 moves from 38.8% to 38.2%. GPT-5.6 Sol does better with OpenHands (88.2% Pass@3, 72.4% Pass^3) than with SimpleAgent (85.3%, 64.1%) (Section 5.2).
  • Runs terminated at a token or wall-time limit: 9 of 510 SimpleAgent trials and 83 of 510 OpenHands trials (Section 5.2).
  • Among 249 failed completed submissions (Section 5.3): VF-01 inadequate coverage 73.1%, VF-02 generic consistency checks 70.3%, VF-04 non-independent verification 28.5%, VF-05 test accommodation 12.0%, VF-06 unverified post-test modification 6.0%, VF-03 ineffective assertions 4.4%.
  • Coder/tester (Section 5.2.1): Astra coder with Fable tester reaches 97.6% Pass@3 and 88.8% Pass^3 at $65.72. The reversed pairing reaches 91.8% and 84.7%. Single-pass Pass^3 is 68.8% for Astra and 79.4% for Fable (Table 7).

The paper at a glance

0117 FEM tasks (H/Thelper tiers)02Single-passimplementation andtest tracks, 15models x 5 trials03Agenticimplementationtrack: SimpleAgentand OpenHands, 6models x 5 trials04FEM-Benchevaluation againstreference outputsandexpected-failure…05Pass@1 / Pass@3 /Pass^3 estimation06249 failedtrajectories -> 3LLM judges -> GPT-6Astra adjudicator-> 6 failure…07Exploratorycoder/testerSimpleAgent withFable 5.1 and GPT-6Astra
  1. 17 FEM tasks (H/T helper tiers)
  2. Single-pass implementation and test tracks, 15 models x 5 trials
  3. Agentic implementation track: SimpleAgent and OpenHands, 6 models x 5 trials
  4. FEM-Bench evaluation against reference outputs and expected-failure implementations
  5. Pass@1 / Pass@3 / Pass^3 estimation
  6. 249 failed trajectories -> 3 LLM judges -> GPT-6 Astra adjudicator -> 6 failure categories
  7. Exploratory coder/tester SimpleAgent with Fable 5.1 and GPT-6 Astra
Fig. 1 — How the paper works, step by step (arXiv:2611.21176).
Claude Opus 5 single-pass65.9%Claude Opus 5 + SimpleAgent77.1%Claude Opus 5 + OpenHands78.8%GPT-6 Astra single-pass68.8%Claude Fable 5.1 single-pass79.4%Coder-tester: Fable coder / Astra tester (this paper)84.7%Coder-tester: Astra coder / Fable tester (this paper)88.8%
Fig. 2 — Pass^3 (all three of three attempts succeed), higher is better. Source: Sections 1, 5.2, 5.2.1; Table 7, arXiv:2611.21176.
Data table
Pass^3 (all three of three attempts succeed)
Method%
Claude Opus 5 single-pass65.9%
Claude Opus 5 + SimpleAgent77.1%
Claude Opus 5 + OpenHands78.8%
GPT-6 Astra single-pass68.8%
Claude Fable 5.1 single-pass79.4%
Coder-tester: Fable coder / Astra tester (this paper)84.7%
Coder-tester: Astra coder / Fable tester (this paper)88.8%

Why it matters

For harness builders, the leaderboard matters less than two other contributions: the Pass@3 versus Pass^3 framing and the self-verification failure taxonomy. In a domain where correctness can be checked numerically, the paper shows that letting an agent execute code and gating submission on a green run mostly raises the chance of occasionally being right, not of being reliably right. Luna's Pass^3 does not move, and the best single-pass model already matches the best agentic Pass^3. When agents ship wrong code, it is usually after passing tests that were too narrow, too generic, or built on the same misconception as the implementation. The coder/tester design is concrete and reusable. Its tester sees only compiled code, keeps a persistent suite, and holds sole approval rights, which points toward separating verification authority from the coder. However, the evidence that role separation itself causes the gain is weak. The design was tested only with Fable 5.1 and GPT-6 Astra, it had a somewhat larger iteration allowance, and neither model was run in the plain harnesses. The domain is narrow: Python FEM kernels with reference outputs. Transfer to general software-engineering harnesses is plausible but not shown.

Issues (7)

  • High severity

    The headline coder/tester result has no matched control. It is run only with Claude Fable 5.1 and GPT-6 Astra, neither of which is among the six models run in SimpleAgent or OpenHands. Fable 5.1 single-pass already reaches 79.4% Pass^3, above every agentic model-harness result (max 78.8%). The coder/tester setup also differs in resources: a 32,000 per-call output cap versus 30,000, 200 iterations per agent rather than 200 total, and up to 50 candidate versions. Without plain SimpleAgent runs for Fable and Astra, or a same-model coder/tester pair, the 88.8% cannot be separated from three other explanations: model strength, extra compute, and the added review loop. The abstract's wording ('Motivated by these findings ... achieves the highest observed scores') invites a causal reading that the design does not support.

  • Medium severity

    The agentic-versus-single-pass conclusions are drawn on six models that exclude the strongest single-pass implementers (Fable 5.1, Astra, Gemini 4 Argon, Gemini 3.1 Pro). The abstract reports that agentic execution helps and that no model-harness combination exceeds 78.8% Pass^3. It does not say that a single-pass model (Fable 5.1, 79.4%) already exceeds that ceiling. Whether harnesses help strong models remains untested.

  • Medium severity

    No confidence intervals or significance tests are reported. Every metric comes from 5 trials on 17 tasks. Several differences are small enough to be noise: Opus 5 OpenHands 78.8% versus SimpleAgent 77.1% Pass^3, Luna 38.8% versus 38.2% Pass^3, and Sol 88.2% versus 85.3% Pass@3. The text nonetheless draws harness-ranking conclusions from them.

  • Medium severity

    The harness comparison is confounded: - SimpleAgent was built by the authors for this benchmark. OpenHands is a general framework run with a different toolset. - OpenHands had internet access. Its agents downloaded from the public FEM-Bench repository, and preliminary runs tried to locate the reference code. - OpenHands hit token or wall-time limits in 83 of 510 runs versus 9 for SimpleAgent. Some losses (e.g., Kimi K3 cohesive-zone T2) are budget exhaustion, not wrong code. - Only one fixed budget is tested. The resulting gap mixes harness design, tool access, budget fit and possible leakage.

  • Medium severity

    The failure taxonomy is assigned entirely by LLMs, and the judges see the reference implementation and the evaluator results. This invites hindsight labeling: knowing the code failed makes 'inadequate coverage' close to automatic, which may inflate VF-01 (73.1%) and VF-02 (70.3%). A single adjudicator, GPT-6 Astra, decides 145 of 249 cases. The judges include GPT-6.1 Sol and Claude Opus 5.5, which share families with the coding models being judged. Agreement statistics are deferred to Table 14. Human involvement is limited to codebook development on more than 20 trajectories, with no reported human-versus-judge agreement on the final labels.

  • Low severity

    The self-verification percentages are conditional on failed, completed submissions only. Successful runs, which may share the same testing habits, are never coded, so the analysis cannot show that these behaviors distinguish failures from successes. The 94 excluded runs are also mostly OpenHands budget exhaustions.

  • Low severity

    Single-pass settings are uneven across models. Output caps range from 32,768 tokens (Gemma 4 31B) to 1,048,576 (Gemini 4 Argon). DeepSeek V4 Flash and Kimi K3 runs are split across providers, including after deprecation. This weakens cross-model ranking claims.

Questions for the authors

  • What Pass@3 and Pass^3 do Claude Fable 5.1 and GPT-6 Astra reach in plain SimpleAgent and OpenHands?
  • Would a same-model coder/tester pair, at a matched iteration and token budget, show the same gain?
  • What are the bootstrap confidence intervals on Pass^3 per configuration, given 5 trials on 17 tasks?
  • How often did OpenHands agents retrieve FEM-Bench materials? Do the OpenHands results change if those runs are excluded or internet access is disabled?
  • What is human-versus-adjudicator agreement on a sample of the 249 labeled trajectories? Do label rates change if judges are not shown the reference implementation?
  • Do VF-01 and VF-02 behaviors occur at similar rates in successful trajectories?

Related in the directory

Open-source projects

One of the two agentic harnesses evaluated, compared directly against the authors' SimpleAgent.
Source of the Pass^k reliability metric the paper adopts as its main consistency measure.

Commercial products

Commercial steward of the OpenHands framework used as a baseline harness.

People

Lead author of the OpenHands paper the study cites for its general-purpose harness.

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).

Tanmay Sah

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.