PaperarXiv:2611.23219 · cs.LG · Nov 17, 2026
Do LLM Agents Still Need to Be Told How to Self-Improve?
Shobhita Sundaram, Cansu Sancaktar, Phillip Isola
The paper is a clearly instrumented demonstration that a capable model in a minimal harness can design its own context-optimization loop, and its ablations on evaluation-information leakage are informative. Several weaknesses keep it from being stronger: - Benchmarks are filtered on Opus-5's failures. - Transfer evidence is single-model. - Samples are small, with no significance tests. - Harness and prompt differ across families. - Behaviour analysis is judged by a model that is also under test. - Concurrent work (ASPIRE, Open-Ended Optimization) narrows the novelty.
Novelty
3 out of 5Rigor
2 out of 5Reproducibility
2 out of 5Relevance to harnesses
3 out of 5
TL;DR
A frozen LLM runs as a teacher inside a stripped-down Claude Code or Pi session with Read/Write, Bash and a Gameplay tool. It writes its own practice games and edits a player's prompt and skill files, with no prescribed loop. Newer models gain on held-out PuzzleScript levels. For Opus-5, the only model whose transfer is reported, some of the gain carries to ARC-AGI-3 L1 and MazeBench. Samples are small (16 to 20 items per benchmark, 3 teacher seeds). Test sets are filtered on Opus-5's failures, and model families run in different harnesses.
Adversarial review by Harness Hub · Nov 20, 2026 · read full text (truncated) · Suggest a correction
Original abstract (authors’ text, select to expand)
LLM agents increasingly drive their own learning. Self-improvement methods delegate various components of model optimization to the models themselves. As models become more capable, this motivates a basic question: do these increasingly capable models still need us to specify how they should self-improve? In this paper we study an autonomous paradigm in which, given basic primitives, resources, and a general goal for self-improvement, agents construct their own self-learning algorithm for context-based optimization. Using the setting of gameplay as a testbed, a frozen LLM agent is given a compute budget and tools for collecting experience and updating text documents, and allowed to freely experiment without an externally imposed loop or access to real-world data. We find that sufficiently capable models do self-improve under these conditions, demonstrating strategies that align with classic cognitive frameworks for self-regulated learning and constructs from machine learning literature on self-generated curricula. Furthermore, we study the extent to which the learned text artifacts transfer to external benchmarks, observing improvement across different game horizons, mechanics, and 3D navigation.
Authors’ note
Project page: https://ssundaram21.github.io/self-org/
Filed underevaluationself-improvingagentsllm
Summary
The setup pairs a teacher agent with a player, which is another instance of the same frozen model. The teacher's goal is to improve the player at unfamiliar single-player games by writing text artifacts. It has a budget of 20 context edits and 150 observed gameplays, and the tools {Read/Write, Bash, Gameplay}. The Gameplay tool runs a teacher-written PuzzleScript (or Python) game in a fresh player session and returns the score and transcript. No real games or validation scores are provided. Six models are tested: Opus-4.6/5, GPT 5.6-Sol/6-Astra and GLM-4.7/5.3. Each model runs 3 teacher sessions. Final contexts are evaluated on four benchmarks: - PuzzleScript-t20 (20 levels) - PuzzleScript-t30 (20 levels) - ARC-AGI-3 Level 1 (17 games) - MazeBench (16 rooms) The paper reports that newer models improve and older ones do not. It tags teacher behaviours such as A/B testing, BFS-filtered game generation, and more replays on medium-difficulty games. An LLM judge scores these behaviours against Zimmerman's self-regulated learning rubric. The ablations are run with Opus-5 only. They vary: - the toolset: Read-Write only, then adding Bash, then adding Gameplay; - the game language: Python instead of PuzzleScript; - how much the teacher is told about the evaluation setting. Learned artifacts are also swapped between Opus 5 and GPT-6 Astra.
Key findings
- Section 4.1 / Figure 4: on PuzzleScript-t20, newer generations improve over the default-context baseline: '+20% for Claude Opus-5, +10% for GPT 6-Astra, +6% for GLM 5.3'. Older generations show 'little or no improvement'. The text does not say whether these are absolute points or relative gains.
- Section 5.1 / Figure 9 (Opus-5 only): context optimized under 20-turn PuzzleScript gameplay improves PuzzleScript-t20 by 20%, PuzzleScript-t30 by 9%, ARC-AGI-L1 by 21%, and MazeBench SPL by 0.15.
- Section 5.1: a Read-Write-only teacher, with no gameplay tool and no Bash, outperforms the baseline. On all benchmarks besides MazeBench it recovers less than 50% of the Gameplay teachers' gain. The paper does not quantify the MazeBench case in text.
- Section 5.1 / Figure 10: Python gameplay 'recovers nearly all the performance gain of PuzzleScript gameplay in 3 out of 4 cases'.
- Section 5.2 / Figure 11: imposing t20-matched turn and token budgets raises PuzzleScript-t20 but lowers PuzzleScript-t30. Richer descriptions of the evaluation games improve PuzzleScript-Narrow while ARC-AGI-3 plateaus and slightly declines.
- Table 1, PuzzleScript-t20, mean ± std over three seeds: the Opus 5 player scores 25.4 ± 3.6 with no artifacts and 47.5 ± 4.5 with GPT-6 Astra artifacts. The GPT-6 Astra player scores 67.1 ± 3.8 with no artifacts and 72.1 ± 4.0 with Opus 5 artifacts.
- Figure 3: for Opus-5, self-organized learning achieves 'similar performance' to GEPA optimizing over 20 in-distribution real games under matched game-rollout budgets (3 seeds). This is shown for one model only.
The paper at a glance
- Teacher session (Claude Code or Pi, defaults disabled) receives a goal prompt and a budget of 20 updates and 150 gameplays
- Teacher writes PuzzleScript/Python games via Write/Bash
- Gameplay tool runs a fresh player session with the chosen artifacts and returns score and transcript
- Teacher edits player_folder (prompt, skill files); each edit consumes one update
- Loop until either budget is exhausted
- Evaluate the final context on PS-t20, PS-t30, ARC-AGI-3 L1 and MazeBench with fresh player sessions
Data table
| Method | % |
|---|---|
| Opus 5, no artifacts | 25.4% |
| Opus 5 with GPT-6 Astra self-optimized artifacts (this paper) | 47.5% |
Why it matters
For harness builders, the practical claim is that a capable model running inside a minimal coding-agent harness can run its own prompt and skill-file optimization. It needs only an execution or eval tool and an edit budget, and no hand-curated training tasks. For one model (Opus-5) and one benchmark, the result is reported as similar to GEPA hill-climbing on real games. That bears on how much scaffolding to build around skill-file or memory self-editing loops. The ablations give some operational guidance: - A Read-Write-only teacher already gains something. - An execution tool adds more. - Telling the optimizer the exact eval budgets causes specialization to them. The evidence base is thin. Benchmarks have 16 to 20 items, with three teacher seeds. Test sets are filtered against Opus-5's own baseline failures. All transfer and ablation results use Opus-5 only. Claude runs in Claude Code while the other families run in Pi with extra prompt fixes. The domain is puzzle games, not coding or tool-use. Teacher compute cost is not reported, which matters if you are deciding whether to run such a loop. Treat it as a demonstration that the pattern can work, not as a measured recipe.
Issues (9)
- High severityWhere in paper: Appendix B (PuzzleScript-t20, PuzzleScript-t30, ARC-AGI-3 L1 construction); Table 1
Both PuzzleScript sets and ARC-AGI-3 L1 are filtered by removing every level that Claude Opus-5 solves in 8/8 rollouts; ARC-AGI-3 L1 drops from 25 to 17 games. Selecting on one model's baseline failures favours apparent improvement for that model, and Opus-5 is the model used for every transfer and ablation result. The baseline is re-run, but selecting on low baseline success still invites regression-to-the-mean gains. It also makes the cross-family comparison non-neutral. Table 1 shows the asymmetry: on PuzzleScript-t20 the Opus 5 baseline is 25.4 while the GPT-6 Astra baseline is 67.1. Opus's +20% headline is measured on a set built from its own failures.
- High severityWhere in paper: Section 3.4, Table 2, Section 4.1 / Figure 4, Section 7
Statistical power is low for the size of the claims: - Each benchmark has 16 to 20 items. - Each game gets 4 eval rollouts (2 on MazeBench). - Each condition has 3 teacher seeds. The Opus 5 baseline alone has ±3.6 std. Gains such as +6% for GLM 5.3 and +9% on PS-t30 are reported without significance tests. The 'newer generations self-improve' trend rests on two models per family, so 'meta-capability improves across generations' is extrapolated from three pairs.
- Medium severityWhere in paper: Section 5 opening ('using Claude Opus-5'), Figure 9, Figure 11, Abstract
The abstract's generalization claim ('learned text artifacts transfer... across different game horizons, mechanics, and 3D navigation') is supported only for Claude Opus-5. All of Sections 5.1 and 5.2 use Opus-5, and the only other-model evidence is the PuzzleScript-t20 swap in Table 1. Whether GPT 6-Astra or GLM 5.3 artifacts transfer to ARC-AGI-3 or MazeBench is not shown.
- Medium severityWhere in paper: Appendix C.1 (Harness details), Appendix C.5 (Pi-specific prompt adjustments)
The harness differs across model families. Claude models run in Claude Code v2.1.232; GPT and GLM run in Pi v0.85.1 via OpenRouter. GPT and GLM teachers also received extra Gameplay tool wording and a budget statement in the system prompt, added after malformed tool calls and misread budgets. Cross-family differences in self-improvement and in the behaviour metrics are therefore confounded with harness and prompt.
- Medium severityWhere in paper: Section 4.2 (Alignment to Self-Regulated Learning), Figure 6
The SRL alignment scores are produced by GPT 6-Astra as an LLM judge. GPT 6-Astra is also one of the evaluated teacher models. No human annotation or inter-rater check is reported, only consistency across seeds. The paper says the link is not causal, but Figure 6 is presented as evidence that better self-learners use recognizable strategies.
- Medium severityWhere in paper: Section 5.1 (Read-Write comparison), Appendix C.5 (default player prompt)
Part of the gain does not come from self-generated experience. A Read-Write teacher with no gameplay and no Bash outperforms the baseline, and the under-50% recovery statement explicitly excludes MazeBench. The default player prompt is a few lines of output-format instruction, so some improvement is ordinary prompt-writing from prior knowledge against a weak baseline, not a self-organized learning loop. No stronger hand-written prompt baseline is reported.
- Medium severityWhere in paper: Figure 3, Section 4 opening paragraph, Section 3.4
The GEPA comparison is described only as 'under matched game-rollout budgets'. The visible text does not report GEPA's configuration, reflection model, tuning, or whether teacher token costs were matched. 'Comparable' is a visual judgment over 3 seeds for one model. Overall teacher token or dollar cost per session is not reported for any condition.
- Low severityWhere in paper: Section 3.2 (Context), Appendix B, Section 7 (Limitations)
The 'data-free' framing is weaker than it sounds. The teacher gets the official PuzzleScript documentation, and the evaluation games come from a public community PuzzleScript pool and ARC-AGI-3, both possibly in pretraining data. The teacher could recall human games or design priors. The paper acknowledges model priors about curricula but does not test for contamination.
- Low severityWhere in paper: Section 4.1, Section 5.1
The headline improvements '+20%', '+10%' and '+6%' do not say whether they are absolute win-rate points or relative gains. Except for Table 1, the underlying per-condition numbers appear only in figures.
Questions for the authors
- What are the absolute win rates and per-seed numbers behind '+20% / +10% / +6%', and do the gains survive a paired significance test across the 20 games?
- How do results change if PuzzleScript-t20 and ARC-AGI-3 L1 are filtered against each evaluated model's own baseline, or not filtered at all?
- Do GPT 6-Astra and GLM 5.3 artifacts transfer to PuzzleScript-t30, ARC-AGI-3 L1 and MazeBench as Opus-5's do?
- Would running GPT and GLM teachers inside Claude Code, or Claude inside Pi, separate model generation from harness effects?
- What budget, model and configuration was GEPA run with in Figure 3, and what is the total teacher token cost per session?
- How does a stronger hand-written player prompt compare with the self-optimized context?
- Does a human-annotated subset agree with the GPT 6-Astra SRL judge scores?
Related in the directory
Open-source projects
Open source · Anthropic
Open source · Earendil Works
Open source · gepa-ai
Commercial products
Product · Remote
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26450 · cs.AI · Nov 19, 2026
Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).
Tanmay Sah