PaperarXiv:2611.21056 · cs.CL · Nov 16, 2026
Evaluation-Free Recursive Self-Improvement for Coding Agents
Zhe Ren, Guowei Rong, Yimeng Chen, Nanbo Li, Piotr Piękos, Mateusz Ostaszewski, Dandan Guo, Jürgen Schmidhuber
The recipe is practical and low-cost, the gate ablation is informative, and per-run DeepSeek results are reported for all methods. What it does not settle is evaluation-free versus evaluate-and-select in general. The baselines were tuned and scored on one 60-task subset, transfer sets are about 50 tasks, the Qwen baselines are single runs, and the starting scaffold is weak.
Novelty
3 out of 5Rigor
2 out of 5Reproducibility
2 out of 5Relevance to harnesses
4 out of 5
TL;DR
ELITE evolves a coding-agent scaffold without running candidates on benchmark tasks. Each generation, an LLM proposes K=7 focused changes, each change is implemented separately from the same parent, and the compatible ones are reimplemented as one descendant. The only gates are deterministic compile, import, interface and smoke checks. Under DeepSeek-V4-Flash, ELITE is slightly below DGM and HGM on a 60-task SWE-bench Pro subset (85.6% vs 86.7% and 88.3%), costs far less, and scores higher on Verified and Polyglot subsets. The baselines were given feedback only on the same 60 Pro tasks they are scored on, so the transfer comparison partly measures that setup choice.
Adversarial review by Harness Hub · Nov 19, 2026 · read full text (truncated) · Suggest a correction
Original abstract (authors’ text, select to expand)
Recent self-improving coding agents have achieved strong benchmark performance by iteratively modifying their own executable implementations. A common pattern uses downstream task-derived evidence to decide which candidates or lineages continue evolving, which we term evaluate-and-select. Obtaining this signal requires repeatedly executing evolved agents on downstream tasks, so broader exploration incurs increasing evaluation cost, while repeated optimization against a fixed evaluation distribution may encourage benchmark-specific adaptation. We ask: can coding agents self-improve without downstream task evaluation inside the evolution loop? We introduce Explore-then-Integrate (ELITE), which integrates implementation mechanisms rather than selecting a single exploratory child to continue the lineage. At each generation, ELITE proposes diverse focused improvements, implements them independently from the same parent, and inspects the resulting programs to identify complementary, redundant, and conflicting mechanisms before reimplementing a compatible subset as one coherent descendant. No downstream task outcomes or task-solving trajectories of evolved candidates are used to guide evolution; only deterministic checks of structural and operational validity are used inside the loop. On SWE-bench Pro, ELITE improves the initial agent from 60.0% to 85.6%, compared with 86.7% for the Darwin Godel Machine (DGM) and 88.3% for the Huxley-Godel Machine (HGM), while using zero candidate task evaluations and reducing observed wall-clock evolution time by 8.8x and 3.2x, respectively, under the evaluated configurations. In cross-benchmark transfer without further adaptation, ELITE averages 73.3% on SWE-bench Verified and 69.3% on Polyglot. Together, these results demonstrate the feasibility of coding-agent self-improvement without downstream candidate task evaluation in our setting.
Authors’ note
32 pages, 14 figures
Filed underevaluationcoding-agentsself-improvingagents
Summary
The paper names the pattern used by DGM, HGM and similar systems 'evaluate-and-select': downstream task outcomes decide which agent variants or lineages keep evolving. It asks whether a coding agent can improve its own scaffold without that signal inside the loop. ELITE runs T=10 generations. In each generation: - An LLM reads the current agent source and a compact evolution history, then proposes K=7 limitation-grounded hypotheses. - Each hypothesis is implemented independently against the same parent. - A deterministic exploration gate drops invalid children (compilation, imports, predefined assertions, modification scope). - An LLM integrator inspects the valid patches, labels mechanisms as complementary, redundant, conflicting, overly complex or incomplete, and reimplements a compatible subset as one patch on the parent. - A stricter acceptance gate adds smoke tests of the tool-call protocol and main execution flow. If it fails, the parent is kept. No task outcomes or trajectories of evolved candidates are used in the loop. Setup: - Backbones: DeepSeek-V4-Flash and Qwen-3.7-Plus. - Starting point: the DGM scaffold (bash plus a whole-file editor). - Evolution target: a fixed 60-task SWE-bench Pro subset. DGM and HGM receive outcome feedback from this subset. - Transfer benchmarks: SWE-bench Verified and Polyglot subsets, used only for post-evolution evaluation. - Replication: three runs per method under DeepSeek. Under Qwen, three ELITE runs and one run per baseline. Results: - DeepSeek (Table 1, Pro / Verified / Polyglot): ELITE 85.6 / 73.3 / 69.3; DGM 86.7 / 63.3 / 62.7; HGM 88.3 / 65.3 / 64.7. - Cost (Table 2): ELITE uses 0 candidate task evaluations versus 810 (DGM) and 800 (HGM). Evolution takes 4.90 h versus 43.15 h and 15.66 h. - Further experiments: design ablations, K sensitivity, one mini-swe-agent run with Claude-opus-5.5, and cross-backbone scaffold transfer.
Key findings
- DeepSeek-V4-Flash, Table 1: ELITE raises Pro from 60.0 to 85.6. DGM reaches 86.7 and HGM 88.3; both use Pro-subset outcomes as evolutionary feedback.
- Transfer, Table 1 (DeepSeek): ELITE scores 73.3 on Verified and 69.3 on Polyglot. DGM scores 63.3 and 62.7, and HGM 65.3 and 64.7. Both baselines sit near the initial agent's 64.0 and 64.0.
- Qwen-3.7-Plus, Table 1: ELITE is lowest of the three methods on Pro (45.6 vs 50.0 DGM and 48.3 HGM). It is highest on Verified (56.7 vs 54.0 for both) and Polyglot (61.3 vs 50.0 DGM and 58.0 HGM). The Qwen baselines are single runs.
- Cost, Table 2 (DeepSeek): ELITE uses 0 task evaluations, 80M estimated tokens, 4.90 h and $8. DGM uses 810 evaluations, 2900M tokens, 43.15 h and $45. HGM uses 800 evaluations, 2100M tokens, 15.66 h and $30.
- Gate ablation, Table 4 (three-run mean): removing the deterministic gates drops the average from 76.1 to 52.0, and Pro to 59.4, below the initial 60.0.
- Evaluation-based Select-one, Table 4 (single run, children scored on 30 Pro tasks per round): reaches 91.7 on Pro versus ELITE's 85.6. Its transfer scores are lower (Verified 70.0 vs 73.3, Polyglot 62.0 vs 69.3).
- Post-hoc trace, Fig. 3a and Sec. 5.3: at the first accepted step, the integrated descendant scores 81.7, below the best exploratory child at 83.3. Across five accepted updates, Pro goes from 60.0 to 86.7 with two intermediate regressions.
- Mini-swe-agent with Claude-opus-5.5, Table 3: Pro goes from 71.7 to 83.3, Verified from 70.0 to 78.0 and Polyglot from 72.0 to 78.0. This is a single run with no DGM or HGM comparison.
The paper at a glance
- Current agent source + evolution history
- LLM proposes K=7 limitation-grounded hypotheses
- Implement each hypothesis independently from same parent
- Deterministic exploration gate (compile, imports, assertions, scope)
- LLM integrator classifies mechanisms and reimplements compatible subset
- Acceptance gate with tool-protocol smoke tests; else keep parent
- Repeat for T=10 generations
Data table
| Method | % |
|---|---|
| Initial Agent | 62.7% |
| DGM | 70.9% |
| HGM | 72.8% |
| ELITE (this paper) | 76.1% |
Why it matters
For harness builders, the usable idea is a low-cost loop for improving a scaffold. Have the model read the agent code, propose several narrow fixes, implement each in isolation, then reimplement the compatible ones behind compile and smoke-test gates. This avoids the hundreds of benchmark rollouts that DGM-style loops spend per run. The case studies show recognizable harness hygiene: - returning exit codes alongside full command output; - quote-aware tool-call parsing; - deterministic test discovery; - stripping non-source artifacts from patches; - change-aware test selection. The starting scaffold is weak: bash plus a whole-file editor with truncated views. Much of the gain may therefore be low-hanging fruit that a careful manual review of the harness would also find. Evidence on a mature scaffold is one unbaselined mini-swe-agent run. Two lessons carry over to anyone building self-editing or auto-tuned harnesses. First, structural validity gates are what keep unsupervised self-modification from collapsing: without them, Pro falls below the starting agent (Table 4). Second, selecting on a small fixed task subset can raise in-sample scores without transfer, though here that rests on single runs and subsets of about 50 to 60 tasks.
Issues (9)
- High severityWhere in paper: Sec. 5.1 (Benchmarks; Compared methods and evolution protocol), Table 1
The transfer advantage may come from how the baselines were configured rather than from the evaluate-and-select paradigm. DGM and HGM get outcome feedback only from the same fixed 60-task Pro subset that their Pro scores are reported on. Their 86.7 and 88.3 are therefore in-sample numbers, and their flat Verified and Polyglot scores are what heavy selection on 60 tasks would be expected to produce. There is no held-out Pro split, and no baseline configuration with broader or rotating evaluation tasks. Both the 'comparable on Pro' claim and the 'better transfer' claim depend on this one setup.
- High severityWhere in paper: Table 1, Tables 6-7 (Appendix A.1.2), Sec. 5.1
The evaluation sets are small and the Qwen comparison is thin. Pro has 60 tasks, so one task is about 1.7 points. Verified and Polyglot per-run scores move in 2-point steps (for example 72.0, 74.0, 64.0 in Tables 6-7), which suggests subsets of about 50 tasks; the sizes are deferred to Appendix A.4. The Qwen Verified gap (56.7 vs 54.0) is roughly one task, and the Qwen DGM and HGM numbers are single runs. Only the DeepSeek comparison has three runs per method.
- Medium severityWhere in paper: Abstract; Sec. 5.2; Table 1
The abstract reports only the DeepSeek backbone, where ELITE's Pro gap to the baselines is smallest (1.1 and 2.7 points). Under Qwen-3.7-Plus, ELITE is the lowest of the three on Pro (45.6 vs 50.0 and 48.3), and the paper itself calls that trade-off 'more pronounced'.
- Medium severityWhere in paper: Table 5, Sec. 5.4; Appendix A.1.1 (gate outcomes); Sec. 4.2
'Evaluation-free' holds only inside the loop. The K sweep (Table 5) reports the same Verified and Polyglot subsets used as transfer results, and its K=7 row reproduces the main-table numbers. K=7 is the default, and it is the best width on both transfer benchmarks. The gate assertions (including the tool-order and call-limit checks), the prompts and T=10 are human-designed, and there is no in-loop stopping rule even though the trace shows regressions. Benchmark knowledge can enter through these design choices.
- Medium severityWhere in paper: Table 2, Sec. 5.2 (Evolution efficiency)
The cost comparison depends on configuration. Baseline resource numbers are 'from the original runs' (Table 2 caption). The 8.8x and 3.2x wall-clock speedups depend on parallelism and on the evaluation budgets given to DGM and HGM (810 and 800). The visible text does not justify those budgets or compare them with the original DGM and HGM settings. Token figures are estimates.
- Medium severityWhere in paper: Table 4, Sec. 5.4
Most ablations are single runs. The strongest alternative, Evaluation-based Select-one, beats ELITE on Pro (91.7 vs 85.6) while scoring children on only 30 tasks per round. The case against it rests on one run with a 3.3-point Verified gap and a 7.3-point Polyglot gap on small subsets. Single exploration (K=1) is not compute-matched, which the paper acknowledges.
- Medium severityWhere in paper: Appendix A.2, Appendix A.1.4, Table 3
The weak starting scaffold inflates the apparent self-improvement. The initial agent has a whole-file-only editor, 10,000-character view truncation and a bash tool without exit-status reporting. The case studies show ELITE mainly repairs these gaps. The only test on an established scaffold (mini-swe-agent with Claude-opus-5.5) is a single run, has no DGM or HGM comparison, and leaves some intermediate scores unreported.
- Low severityWhere in paper: Appendix A.1.3, Fig. 7; Appendix A.1.1, Fig. 5
The evidence that integration helps comes from one lineage and small samples. The sampled diagnostic (68.0 integrated vs 48.0 exploratory) uses 10 tasks from one DeepSeek run, and the paper states it does 'not establish a general advantage'. The 25-of-41 mechanism-incorporation count comes from manual inspection.
- Low severityWhere in paper: Sec. 5.1, Sec. 5.3
Contamination is not discussed. Public SWE-bench repositories may appear in backbone training data, and nothing checks whether the transfer gains interact with memorized repositories.
Questions for the authors
- How do DGM, HGM and ELITE compare on held-out SWE-bench Pro tasks that no method used as evolution feedback?
- Does the transfer advantage persist if DGM and HGM evaluate on a larger or rotating task pool?
- What are the exact task counts of the Verified and Polyglot subsets, and how were they sampled?
- Were the 810 and 800 evaluation budgets matched to the original DGM and HGM settings?
- Does ELITE beat DGM and HGM when all three start from mini-swe-agent with repeated runs?
- Were K=7, T=10 or the gate assertions chosen with any Verified or Polyglot results in view?
- Will the evolution code, prompts, gate scripts and evolved scaffolds be released?
Related in the directory
Open-source projects
Open source · SWE-bench
Open source · SWE-agent (Princeton & Stanford)
Open source · Princeton NLP / SWE-agent
Open source · Aider AI
People
Person · Creator of Aider
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26444 · cs.RO · Nov 19, 2026
Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.
Yuzhi Huang, Jie Wu +11