PaperarXiv:2611.21152 · cs.CL · Nov 16, 2026 · revised Nov 17, 2026
From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop
Yang Tian, Tianyu Zheng, Xiyuan Yang, Yaxin Du, Chengran Yang, Bo Zhao, Jian Yang, Bryan Dai
The environment construction pipeline and its two-sided validity check are practical and reusable for repository-level evaluation or training environments. The same-harness data-source comparison is a reasonable design. However, the headline gains largely reflect the match between training task format and benchmark format. The evolution ablation is confounded by data quantity, there is no variance reporting, and no code or data URL is printed.
Novelty
3 out of 5Rigor
2 out of 5Reproducibility
2 out of 5Relevance to harnesses
3 out of 5
TL;DR
RepoFactory turns real Python repositories into 'rebuild this repo from a spec' environments and revises them with cheap proxies: binary validity, call-graph coupling and LOC/dependency scope. Supervised fine-tuning (SFT) of Qwen3.5-35B-A3B on trajectories from the Qwen3.8-Max teacher raises NL2Repo from 23.8 to 46.4. The benchmark has the same task format as the training data, and the evolution ablations are small and reported without variance.
Adversarial review by Harness Hub · Nov 19, 2026 · read full text · Suggest a correction
Original abstract (authors’ text, select to expand)
Training coding agents for long-horizon tasks requires environments with verifiable feedback. Existing pipelines often follow a linear process that ends with validation, establishing usability but providing little guidance on improving training value. Recursive self-improvement (RSI) often requires model rollouts or model training followed by downstream evaluation, with costs and feedback delays amplified on long-horizon tasks. How can we use timely, reliable proxy signals to improve training value and synthesize environments at scale? We present RepoFactory, an automated pipeline that converts real repositories into verifiable environments where agents build complete projects from scratch given a language specification. Its Environment Evolution Loop optimizes a multiplicative objective combining validity, structural coupling, and implementation scope through proxy feedback. RepoFactory produces 10,159 environments across 17 software domains. Training Qwen3.5-35B-A3B on these environments yields RepoHorizen-35B-A3B, which achieves gains of 22.6 percentage points on NL2Repo-Bench and 8.5 points on average across five benchmarks covering long-horizon coding and agentic tasks in real-world production settings, compared with 17.5 and 6.4 points without environment evolution. These results highlight the training value of RepoFactory environments and the additional gains from the Environment Evolution Loop.
Filed underevaluationcoding-agentsself-improvingagents
Summary
The pipeline has four stages. 1. **Find repositories.** It searches for Python repositories with a local test suite, at least five implementation files and fewer than 40,000 lines. 2. **Build a runtime.** It builds a Docker image with no network access and keeps a repository only if all tests pass. 3. **Write the task.** An agent writes a natural-language specification and an API manifest. These are checked against the symbols the tests import. 4. **Evolve the environment.** The Environment Evolution Loop jointly revises the oracle code, the specification and the tests for up to 12 rounds. It maximizes Q = A·C·S: - A is a binary validity check: the oracle passes and the unchanged workspace fails. - C is normalized propagation cost over the static call graph. - S is log10(LOC) + log10(1+D) - 2. All construction agents use Qwen3.8-Max. The result is 10,159 environments across 17 domains. Training data comes from running Qwen3.8-Max five times per environment and filtering to reward above 0.5 plus quality checks. That leaves 10,947 trajectories from 5,577 environments, used for 3-epoch SFT of Qwen3.5-35B-A3B. There is no reinforcement learning (RL) stage; training is offline distillation from a stronger teacher. Evaluation uses the Claude Code 2.1.140 scaffold on five benchmarks: NL2Repo, SWE-bench Verified, SWE-bench Pro, SWE-bench Multilingual and Terminal-Bench 2.1.
Key findings
- Table 1: RepoHorizen-35B-A3B scores 46.4 on NL2Repo (base 23.8, +22.6) and 53.5 average (base 45.1, +8.5). The no-evolution variant scores 41.3 and 51.5.
- Table 1: gains outside NL2Repo are smaller: SWE-bench Verified 72.2 (+6.4), SWE-bench Pro 47.6 (+5.2), SWE-bench Multilingual 61.0 (+1.3), Terminal-Bench 2.1 40.5 (+6.8).
- Table 2: under the same Claude Code harness, with 11,000 trajectories per baseline, the six open SWE datasets reach 22.6 to 25.2 on NL2Repo against 46.4 for RepoFactory. Margins on SWE-bench are smaller; for example, SWE-rebench reaches 68.8 on Verified against 72.2.
- Section 5.6 / Figure 6: on 500 sampled repositories, repair raises validated environments from 169 to 347 (yield 33.8% to 69.4%).
- Section 5.6 / Figure 5: optimizing S moves NL2Repo from 30.2% (Level 1) to 32.9% (Level 4). Optimizing C moves it from 30.2% to 32.5%. Each level uses 2,000 trajectories.
- Section 4: teacher success falls from 23.0% in the smallest-scope group to 13.4% in the largest.
- Section 5.3: NL2Repo goes from 36.5% to 46.4% when the remaining 2,577 environments are added to reach 5,577.
- Section 5.4: NL2Repo gains of 12.2 to 22.6 points across the 9B, 27B and 35B-A3B models.
The paper at a glance
- Repository search and filtering (Python, tests, 5+ files, <40k LOC, no external services)
- Docker runtime construction; keep only if all tests pass offline
- AST-based repository analysis (public APIs, test-imported symbols)
- Spec + API manifest synthesis, checked against test imports
- Environment Evolution Loop: repair validity A, then raise scope S or coupling C (up to 12 rounds, revert if no strict gain)
- Teacher (Qwen3.8-Max) rollouts x5, reward > 0.5 filter + quality checks
- SFT of Qwen3.5-35B-A3B, evaluated in Claude Code scaffold
Data table
| Method | % |
|---|---|
| OpenSWE | 25.2% |
| R2E | 22.6% |
| ScaleSWE | 24% |
| SWE-smith | 22.6% |
| SWE-rebench | 23.1% |
| SWE-rebench-v2 | 24% |
| RepoFactory (this paper) | 46.4% |
Why it matters
For people building coding-agent harnesses, the reusable part of this paper is the environment construction recipe, not the model. That recipe includes Docker runtimes that must pass all original tests offline, and a two-sided validity check: the oracle gets full reward and the unchanged workspace gets zero. It also includes a consistency check between the spec and API manifest and the symbols the tests import, plus cheap static metrics used as accept/revert signals in an agentic revision loop. These ideas carry over to anyone building internal eval suites or training environments for long-horizon repository tasks. All evaluation runs inside Claude Code rather than a custom harness, so the paper says nothing about scaffold design itself. The headline NL2Repo jump mostly shows that distilling a teacher that scores 55.9 on NL2Repo, using tasks shaped like that benchmark, helps on that benchmark. Transfer to SWE-bench and Terminal-Bench is real but modest. The paper names GitHub and Hugging Face releases but prints no URLs, so the pipeline cannot be reused from the text alone.
Issues (11)
- High severityWhere in paper: Section 5.5, Table 2; Section 5.2
The headline NL2Repo gain is largely in-distribution. RepoFactory tasks are full-repository generation from a natural-language spec, which is exactly the NL2Repo format, while all six Table 2 baselines are issue-resolution datasets. The 21.2-point lead over OpenSWE on NL2Repo therefore mainly reflects task-format match, not 'training value' per trajectory. On benchmarks outside that format the margins shrink: Verified 72.2 vs 68.8 (SWE-rebench), Multilingual 61.0 vs 59.7 (R2E). No other full-repository-generation dataset is trained under the same setup.
- High severityWhere in paper: Table 1 'w/o environment evolution' row; Section 5.2; Section 5.6
The evolution ablation confounds the proxy objective with data quantity and composition. Evolution recovers environments (169 vs 347 valid out of 500, Figure 6), so the evolved pool is larger and distributed differently. The paper does not report how many environments, trajectories or tokens the 'w/o environment evolution' model was trained on, or whether these were matched. The +5.1 NL2Repo and +2.1 average gains credited to evolution may come from more or different data rather than from optimizing S and C.
- Medium severityWhere in paper: Section 5.1 Evaluation Settings; Section 5.6 / Figure 5; Table 1
No seeds, variance or confidence intervals are reported anywhere, and decoding uses temperature 1.0. Several claimed improvements are small: - Multilingual is 61.0 vs 60.8 with and without evolution. - The S and C ablations move from 30.2% to 32.9% and 32.5%, each on 2,000 trajectories. - Intermediate C levels are described as fluctuating. None of these can be distinguished from run-to-run noise on the reported evidence.
- Medium severityWhere in paper: Section 3; Section 4; Appendix A.1; Table 1; Section 5.5
The gains come from distillation from a much stronger teacher. Qwen3.8-Max scores 55.9 on NL2Repo (Table 1), and the student's 46.4 stays below it. The 'teacher is the same' argument in Section 5.5 controls teacher identity. It does not control how teacher success on each data source shapes which trajectories survive the reward > 0.5 filter (Appendix A.1). The same model also builds the environments, so teacher and environment are coupled.
- Medium severityWhere in paper: Section 5.5; Section 4 (Figure 2c); Section 5.6
Matching trajectory counts does not match compute. Baselines use 11,000 trajectories against RepoFactory's 10,947, but RepoFactory trajectories are long: the median is 64 assistant turns (Figure 2c), and Section 5.6 states that optimizing S increases supervised tokens. Supervised token counts per data source are not reported, so RepoFactory may have received substantially more training tokens.
- Medium severityWhere in paper: Section 3.2 Environment Evolution Loop; Section 3.1 Eq. 2
The loop rewrites the oracle implementation and the tests to raise S and C. The only constraint is that the oracle passes and the empty workspace fails. Nothing checks that revised environments still reflect the original repository's behavior, or that tests were not weakened or made trivially satisfiable. The 'real repositories' grounding therefore partly turns into agent-authored code and tests after evolution.
- Medium severityWhere in paper: Section 3.2 Repository Search and Filtering
Contamination control is a single sentence: repositories 'included in established coding benchmarks' are excluded. The paper gives no detail on checks against NL2Repo source repositories, forks or near-duplicates. The teacher may also have memorized popular public repositories whose upstream tests become the verifier. Given that the gain is specific to NL2Repo, this needs a concrete deduplication audit.
- Medium severityWhere in paper: Table 1; Section 5.2
Table 1 reports an 'Avg.' for rows with missing cells, so those averages are not comparable to 53.5. GPT-5.6-sol, Qwen3.8-Max and MiniMax-M3 lack SWE-bench Multilingual, and Kimi-K2.6 lacks NL2Repo. The paper also does not say whether frontier-model numbers were run under the same Claude Code scaffold, so the claim of surpassing GPT-5.5 may compare across harnesses.
- Low severityWhere in paper: Abstract; Section 4; Section 5.3
The abstract cites 10,159 environments, but training used trajectories from 5,577 of them. The abstract's phrase 'trained on these environments' overstates the scale actually used.
- Low severityWhere in paper: Section 3.2; Section 5.6
The ablations do not test the loop as deployed. The recovery ablation allows 'up to three repair steps', while the deployed loop runs up to 12 rounds. The S/C ablations optimize one term at a time and never test the multiplicative Q, and Q is not compared with simpler alternatives such as additive weighting or S alone.
- Low severityWhere in paper: Section 4 Implementation Scope; Figure 2(d); Section 5.6
Evidence that S tracks training value rests partly on a difficulty correlation: teacher success falls from 23.0% to 13.4% across scope groups. Harder tasks are not necessarily more valuable for training, and the downstream S ablation gain (30.2% to 32.9%) is a single unreplicated run.
Questions for the authors
- How many environments, trajectories and supervised tokens were used for the 'w/o environment evolution' model, and were they matched to the evolved set?
- How many supervised tokens does each Table 2 data source contribute, and does RepoFactory still lead when tokens rather than trajectories are matched?
- What deduplication procedure was applied against NL2Repo's source repositories, including forks and near-duplicates?
- Were the Table 1 frontier and open-weight numbers produced with Claude Code 2.1.140 under the same settings, or taken from external reports?
- What is the run-to-run variance on NL2Repo at temperature 1.0, across multiple training and evaluation seeds?
- How does a model trained on another full-repository-generation dataset compare, to separate task format from environment quality?
- When the loop revises the oracle and tests to raise S or C, how often do revised tests weaken or diverge from the original repository's behavior, and is that audited?
Related in the directory
Open-source projects
Open source · Anthropic
Open source · SWE-bench
Open source · Harbor
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26444 · cs.RO · Nov 19, 2026
Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.
Yuzhi Huang, Jie Wu +11