PaperarXiv:2611.21541 · cs.AI · Nov 16, 2026
SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents
Brian T. Weston, Alondra Torres Contreras, Aldair E. Gongora
The design is careful: exact ground truth, a proper scoring rule, a frozen held-out split, pre-registered ablations and paired CIs. It isolates a concrete failure, confident wrong physical parameters under a misspecified simulator. The abstract overstates in three ways: - It presents generality from a single domain. - It reads parity from non-significance. - Max-effort and Fable conclusions are entangled with the token budget and a modified final-turn harness. For harness builders the value lies mainly in the effort and budget lessons, not in the domain results.
Novelty
3 out of 5Rigor
4 out of 5Reproducibility
3 out of 5Relevance to harnesses
3 out of 5
TL;DR
An interactive benchmark in which an agent measures a plate's thermal properties with a heat-conduction simulator that is sometimes under-resolved or missing physics, and reports 95% intervals scored by a proper interval score. On 38 physics-wrong held-out tasks, only GPT-6 Astra and Claude Fable 5.2 stay calibrated at default effort. Claude 5.5 models recover only at much higher reasoning effort. Older and open-weight models do not recover.
Adversarial review by Harness Hub · Nov 18, 2026 · read full text (truncated) · Suggest a correction
Original abstract (authors’ text, select to expand)
We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. At default effort the failing Claude 5.5 models rarely diagnose the missing physics (Opus in 35% of episodes, against 66% for Astra), and even when they do, their ranges often miss; Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause. Asking for wider intervals does not help; stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves. A pre-registered effort ablation shows that the correction is latent in the Claude 5.5 models: at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short. We release the environment, graders and harness.
Authors’ note
28 pages, 7 figures. Code and leaderboard: https://github.com/Btwestyo/sciresearch-eval
Filed underharnessevaluationagents
Summary
Each task gives the agent: - one heating experiment and up to four more experiments it designs; - a 1D finite-volume simulator with free parameters k, rho*c and h; - fit and diagnostic tools under a compute budget. The hidden 'real' sample is simulated four times finer. In physics-wrong tasks it has richer physics: k(T), a glued joint, radiation, or side loss (2D axisymmetric). Six task families separate identifiability, solver under-resolution, missing physics and candidate selection. About 30% of tasks are controls. Scoring uses the Gneiting-Raftery interval score on a log10 scale. No LLM judge is used for scoring. Two LLMs judge only a secondary 'names the mechanism' measure. The held-out split has 93 tasks. Nine models share one harness with these limits: 60 tool calls, 80 turns, 32k output tokens per turn, and 600k cumulative input tokens. Each model runs three times per task at the provider's default effort. Additional experiments: - Hint ablations, on the Claude 5.5 models only. - A pre-registered reasoning-effort ablation. - A later Fable 5.2 addendum. The paper reports four main findings: - Procedural traps are handled. - On missing-physics tasks, most models report narrow, wrong intervals. - Astra's advantage is attributed to experiment design: low-power, short experiments in which the flawed simulator is valid, and estimates based on a single well-fit dataset. - Opus 5.5 at xhigh or max effort recovers by a different route: selecting which existing data to trust.
Key findings
- Physics-wrong default coverage (Section 4): 62% Opus, 56% Sonnet 5.5, 49% Haiku and 20%-38% open-weight, against 94% Astra and 99% Fable 5.2.
- Table 3 geometric-mean interval score on 38 physics-wrong tasks at default: Fable 5.2 0.020, Astra 0.037, Sonnet 5.5 0.75, Opus 5.5 0.77, Haiku 5.5 1.35, Sonnet 4.5 1.39, Qwen3.8-27B 1.43, Muse Glimmer 30B 1.76, Gemma 4 31B 2.07.
- Controls (Section 4): every model's median interval score is at most 0.010, against at least 0.97 on physics-wrong tasks for all models except Astra and Fable.
- Diagnosis rate at default (Section 5): Claude 5.5 models 14-35%, Astra 66%, Fable 5.2 89%. When the effect is named, Opus's ranges contain the truth in 77% of cases against 100% for Astra. Astra stays at 82% when it does not name the cause.
- Experiment design (Section 5): Astra uses at most half the initial power in 70% of episodes, against 8% for Opus. Astra's final dataset fits to 1.4x noise, against Opus's median 21x.
- Hint ablations (Section 6): hint H ('widen your ranges') changes nothing. Hint S (Astra's strategy) improves interval scores 2.5x for Opus, 1.6x for Sonnet and 1.8x for Haiku, while coverage changes by +4, +0 and +7 points. The fresh-split replication gives 2.2x, 1.7x and 1.7x.
- Effort for Opus (Table 4): max is 23x better [16-33x] than default at a median 75.7k output tokens, against 3.5k at default. xhigh is 28x better [17-46x] at 47.2k tokens. Opus max against Astra default shows no detectable change [0.78-1.65x]; the paper states this is not an equivalence test.
- Fable 5.2 (Section 7): default is 1.9x better [1.3-2.7x] than Astra default, at 20.8k against 3.3k median output tokens. At low effort (2.8k tokens) Fable is 6.5x worse [4.5-9.6x] than Astra default.
- Reasoning length is not sufficient (Section 7, Table 4): Qwen3.8 writes a median 70,000 tokens per episode yet covers 38%. Sonnet 4.5, Gemma and Muse show no detectable change at their highest settings.
The paper at a glance
- Generate task: hidden sample with a flaw (k(T), joint, radiation, side loss, coarse solver, uninformative data) or a control
- Agent sees one experiment plus budgets
- Agent designs up to 4 experiments via request_measurement
- Agent runs simulator, fits and diagnostics within compute budget
- Agent submits k, rho*c, h with 95% intervals and a hidden-target prediction
- Interval score (log10 scale) and coverage against exact truth
Data table
| Method | interval score (log10 scale) |
|---|---|
| Claude Fable 5.2 | 0.02 interval score (log10 scale) |
| GPT-6 Astra | 0.037 interval score (log10 scale) |
| Claude Sonnet 5.5 | 0.75 interval score (log10 scale) |
| Claude Opus 5.5 | 0.77 interval score (log10 scale) |
| Claude Haiku 5.5 | 1.35 interval score (log10 scale) |
| Claude Sonnet 4.5 | 1.39 interval score (log10 scale) |
| Qwen3.8-27B | 1.43 interval score (log10 scale) |
| Muse Glimmer 30B | 1.76 interval score (log10 scale) |
| Gemma 4 31B | 2.07 interval score (log10 scale) |
Why it matters
This is a domain benchmark for research and simulation agents. It is not a harness method, and its core thermal-physics results matter mainly to people building agents that call simulators or fit models. For general harness builders the useful parts are its documented harness effects. First, the default reasoning setting can hide a capability: Opus coverage moves from 0.62 to 0.99 on submitted answers when effort is raised, so evaluations should report effort explicitly. Second, a cumulative input-token budget interacts badly with long reasoning that is re-sent every turn. 33 of 198 Opus max episodes and 60 of 66 Fable max episodes hit the budget, and 16 of those 33 Opus episodes claimed to have submitted without calling submit. A loop should verify that the terminal tool was actually called. Third, one model rejected a mid-conversation tool-list change, which forced a refusal-based final turn. Fourth, telling the model to widen its intervals did not produce calibration; stating a concrete action helped only where the action mapped to one knob. All of this comes from a single domain, which limits how far it transfers.
Issues (9)
- High severityWhere in paper: Section 1 'Why one domain'; Section 4; Limitations (i)
Generality is asserted, not tested. Section 1 says the skill 'is not specific to heat conduction'. Yet all results come from one simulator family with 38 physics-wrong held-out tasks. The T6-none cell, called 'the cleanest contrast', has only 3 tasks. The model ranking and the 'Astra designs, Opus selects' explanation may be domain-specific.
- Medium severityWhere in paper: Limitations (v); Section 5 Figure 4a; Table 2
The grading target shapes the k(T) result. For temperature-dependent conductivity the benchmark grades the room-temperature value. The paper itself calls a temperature average 'a defensible alternative'. Part of the Claude 5.5 'confidently wrong' result on k(T) tasks, and part of the reported advantage of Astra's gentle-heating strategy, therefore depends on this target choice rather than on a clear error. The Opus 'compromise' in Figure 4a is one such case.
- Medium severityWhere in paper: Section 3 footnote 1; Limitations (x); Section 4; Table 9; Appendix Table 6
The Fable 5.2 headline is not run under identical conditions. Fable ran on 2026-11-10/11, after the other models' 2026-10-21 to 2026-11-03 window. Its final-turn rule is enforced differently: the full tool list is kept and non-submit calls are refused. The abstract's '1.9 times better than Astra's' rests on this comparison. The CI is also inconsistent: [1.3-2.7x] in Sections 4 and 7, but [1.3-2.8x] in Table 9.
- Medium severityWhere in paper: Section 7 'A new failure at high effort' and 'Claude Fable 5.2'; Table 4
Max-effort conclusions are partly measurements of the harness's 600k cumulative input-token budget. Fable at max had 60 of 66 episodes reach the budget, yet 'at max it does not improve' is reported as a finding. Opus at max submitted only 0.83 of the time. Only the 33 affected Opus episodes received a rerun at twice the budget.
- Medium severityWhere in paper: Abstract; Section 7; Section 8 item 5; Table 4; Limitations (ix)
The effort comparisons are weaker than the abstract implies: - Secondary rows use 1 run per task and are compared against 3-run defaults. - Reasoning controls are not equivalent across providers. - H2 is met by a CI that includes no difference, not by an equivalence test; the paper notes Astra could be up to about 1.3x better. 'No longer detectably behind Astra' is absence of evidence, not parity. - The abstract and Lessons give '13-31 times more reasoning', while Section 7 states Opus writes '14-23 times as many' tokens as Astra. The abstract range is never defined as a ratio against a stated baseline.
- Medium severityWhere in paper: Limitations (iv); Section 9 'Stopping and evidence sufficiency'
The fixed budget of four experiments may favour models that exhaust it. Astra always does and Fable does in 96% of episodes, while most other models stop early. The design-based advantage credited to Astra is partly entangled with stopping behaviour under one untested budget.
- Medium severityWhere in paper: Section 5 'Diagnosis helps but is not enough'; Appendix C; Limitations (ii)
The diagnosis rates quoted in the abstract come from two LLM judges, Claude Opus 5.5 and GPT-6 Astra. Both are also models under test. Their labels are not validated against human labels. Blinding reduces but does not remove the risk of self-recognition bias.
- Low severityWhere in paper: Section 6; Limitations (iii); Appendix Table 14
Hint S was written after inspecting Astra on the held-out split. The pre-registered fresh-split replication mitigates this, but that split has 63 tasks (34 physics-wrong). Hints were run only on the Claude 5.5 models, not on open-weight models or Fable, so 'cannot vs does not think to' is not tested for the weakest models.
- Low severityWhere in paper: Code and data availability; Limitations (vi), (viii)
Headline numbers cannot be independently reproduced. The held-out split and its traces are withheld, and evaluation on it is available only on request. All proprietary models were accessed through Bedrock and may change. Open-weight models ran on different hardware, and 11 Qwen episodes ended in endpoint timeouts that were counted as misses.
Questions for the authors
- How do the max-effort results for Fable, Sonnet 5.5 and Haiku 5.5 change if the input-token budget is raised, or reasoning is stripped from re-sent context, for all models rather than only the 33 Opus episodes?
- How does the ranking on k(T) tasks change if the temperature-averaged conductivity is accepted as correct?
- Does Astra's advantage persist with an experiment budget of 2 or 8?
- How well do the LLM judges' mechanism-naming labels agree with human labels on a sample?
- Was Fable 5.2 run under the original final-turn mechanism on any subset?
- Does the Astra vs Claude 5.5 ordering hold on a second domain with a different misspecification structure?
- Would hint S help the open-weight models, or is their failure upstream of strategy?
Related in the directory
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26444 · cs.RO · Nov 19, 2026
Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.
Yuzhi Huang, Jie Wu +11