PaperarXiv:2611.21407 · cs.AI · Nov 16, 2026
Harness Leverage Across Model Size: A Ladder of Five Local Agent Harnesses from 0.75B to 4.7B
Hao Wang, Ting Huang
The paper audits itself carefully and yields reusable findings. Exit-status scoring overcounts on small models, and the completion gate is not the carrier of the lead. The headline comparison, however, is an author-built harness fit to this regime against mismatched baselines. It has confirmatory support at 2 of 4 rungs, with registration fixed after the first rung was run.
Novelty
3 out of 5Rigor
3 out of 5Reproducibility
2 out of 5Relevance to harnesses
4 out of 5
TL;DR
The authors run their own small-model harness, Mingbird, against pi, OpenCode, goose and mini-swe-agent on 18 artifact-producing tasks with four local models from 0.75B to 4.7B. Mingbird has the highest mean at every rung, but only 2 of 4 pre-registered contrasts survive Holm correction. Its completion gate is not what carries the lead. A repeat-blocking rule is implicated at 0.75B only. The lead over pi at 4.7B is a depth difference the design does not explain. The paper concedes it cannot say whether harness leverage grows or shrinks with size.
Adversarial review by Harness Hub · Nov 18, 2026 · read full text (truncated) · Suggest a correction
Original abstract (authors’ text, select to expand)
A harness turns a model into a system that completes tasks: it builds the prompt, exposes tools and decides when work is done. Harness comparisons hold the model fixed. Model comparisons hold the harness fixed. We vary both: five shipped harnesses, 18 artifact-producing tasks, four locally deployable open-weight models from 0.75B to 4.7B and a 35.5B anchor, with one deterministic checker scoring all 360 ladder cells. The completion-controlling harness, Mingbird, has the highest mean at every size. Of four pre-registered paired contrasts, two survive Holm correction, at 0.75B and 4.7B. Thirteen of sixteen paired intervals exclude zero under the deployment reading, with leads from 0.075 over the strongest rival to 0.819 over the weakest. The completion gate is not the carrier: disabling all six executable checks moves the mean by +0.080, +0.117, +0.003 and -0.004, every interval including zero, and only at 1.9B does disabling it create empty runs. Removing one repeat-blocking rule alone costs 0.411 at 0.75B and leaves eight of eighteen tasks with no artifact. The harness effect is a progress effect: it decides whether a run leaves anything on disk. Of 357 usable cells, 127 leave none of their declared artifacts and 85 of those exit normally, so a scorer that reads exit status credits work that was never written. Whether the effect grows or shrinks with size, four rungs cannot settle: a floor arm produces a rise, a ceiling a decline, and a cross-family rung a peak.
Authors’ note
56 pages, 12 tables, 5 figures
Summary
Five shipped harnesses run on one Ollama 0.40.6 server with temperature 0, thinking off and a 32K context. Each cell is scored by a deterministic checker that reads only files, on 18 held-out tasks (HWF-01..15, HLH-01..03). The model ladder: - Qwen3.5 at 0.75B, 1.9B and 4.7B. - Granite 4.2 at 3.7B, the only cross-family rung. - A 35.5B MoE anchor, which runs mostly on the CPU and is reported on a six-task subset. The ladder has 360 cells, of which 357 are usable. The authors add: - a mechanism gradient that switches off Mingbird's six executable completion checks at each rung; - a streak-rule (anti_loop) ablation at 0.75B; - repeat collections at 0.75B and 4.7B; - a quantization ladder. Mingbird leads at every rung under the deployment reading (Table 4). Disabling the completion gate moves its score by +0.080, +0.117, +0.003 and -0.004, and every interval includes zero (Section 4.4). The abstract reports that removing the streak rule alone costs 0.411 at 0.75B. The provided text truncates before that result's details. Of 357 usable cells, 127 leave no declared artifact, and 85 of those exit normally. The slope analyses in Table 6 are inconclusive by the authors' own reading.
Key findings
- Mingbird ranks first at every rung under the deployment reading (Table 4): 0.664, 0.593, 0.762 and 0.968 at 0.75B, 1.9B, 3.7B and 4.7B.
- Two of the four pre-registered Mingbird-vs-goose contrasts survive Holm correction: 0.75B (adjusted p=0.0067) and 4.7B (adjusted 0.0410). The 3.7B contrast has raw p=0.016 and adjusted 0.080, and the 1.9B contrast does not approach the threshold (Section 4.2).
- At 3.7B, Mingbird vs pi is +0.005 [-0.230, +0.241] under the deployment reading and -0.020 under the rescored reading, effectively a tie (Table 5).
- Disabling all six completion checks changes the score by +0.080 [-0.183, +0.320] at 0.75B, +0.117 [-0.047, +0.302] at 1.9B, +0.003 at 3.7B and -0.004 at 4.7B (Section 4.4).
- The gate changes empty-run counts only at 1.9B: 0 of 18 empty with the gate on, 3 of 18 with it off.
- At 0.75B on HWF-02, pi issues a failing directory-listing bash call 274 times, 273 of them byte-identical, until the request reaches 32,767 tokens. The same model in Mingbird scores 0.786 on that task after the repeat intercept and a context reset (Section 4.4).
- 127 of 357 usable cells leave none of their declared artifacts, and 85 of those 127 exit normally. The rate ranges from 6% for Mingbird to 69% for mini-swe-agent (Section 1).
- Repeating the same configuration moves an arm mean by up to 0.233 (Section 3.8). Mingbird at 0.75B reads 0.664, 0.503 and 0.431 across three collections (Section 4.1).
- On log10 size, the five-arm span slope is +0.198 [-0.078, +0.476] and the gap-to-best slope is -0.591 [-0.836, -0.251]. The authors attribute these trends to floor, ceiling and cross-family artifacts (Table 6, Section 4.3).
The paper at a glance
- Five shipped harnesses x four local models (0.75B-4.7B) on one Ollama server, temp 0, 32K context
- 18 frozen artifact-producing tasks with per-(task, model) wall budgets
- Deterministic file-only checker (milestones 4/7, final artifacts 3/7)
- Deployment reading (timeout = 0) and rescored reading (archived dir scored)
- Paired task-level bootstrap and Wilcoxon contrasts; 4 pre-registered vs goose with Holm
- Mechanism gradient: completion gate on/off at each rung; streak-rule ablation at 0.75B
- Failure coding: empty-artifact cells, normal exits, repeated identical tool calls
Data table
| Method | score (0-1) |
|---|---|
| Mingbird (this paper) | 0.664 score (0-1) |
| goose | 0.186 score (0-1) |
| pi | 0.172 score (0-1) |
| OpenCode | 0.112 score (0-1) |
| mini-swe-agent | 0.012 score (0-1) |
Why it matters
For people building harnesses that drive small local models, the useful content is the failure analysis rather than the leaderboard. Below about 5B parameters, many runs end with nothing on disk while the process exits normally. Any eval that trusts exit status or the model's finish claim will therefore overcount. The paper rules out the completion gate as the main carrier of Mingbird's lead at three of four rungs. The trace evidence and the streak-rule ablation point to repeat-call blocking plus context reset at 0.75B, but that mechanism is shown only at the smallest rung. At 4.7B the lead over pi is a depth difference on tasks both arms finish, and the paper does not decompose it. The headline that Mingbird wins at every size should be read with three facts in mind. The authors built Mingbird for this regime. The baselines are tools built around frontier models, run through a rewriting proxy on a Windows host. Run-to-run swings are as large as several of the gaps. The negative result on scaling is useful in its own right: floor arms, ceilings and a cross-family rung can each manufacture a leverage-vs-size trend. Builders who target only frontier API models will find little here.
Issues (9)
- High severityWhere in paper: Sections 1, 3.3, 3.4, 3.8; Table 2
The comparison pits an author-built harness against third-party tools. Mingbird is the authors' own system, designed for 1-9B models on a Windows host. The held-out tasks mirror the development families 'task for task'. The comparator goose was fixed from the authors' earlier matrix. The baselines were designed around frontier models and Unix shells. The 'as shipped' framing therefore makes the comparison partly a test of fit to this regime and host. The abstract nonetheless presents a clean win at every size.
- High severityWhere in paper: Section 4.4 (HWF-02 trace); Table 3; Section 4.1
Host and protocol confounds act against specific baselines. pi's 0.75B failures are traced to 'cd: too many arguments' from an unquoted path containing a space: eleven of eighteen cells contain the error and nine end empty. The workspace path is an environment choice the authors control, and it interacts with pi's bash-only listing. mini-swe-agent's floor is attributed to its own reply contract, not to graded capability. The baselines go through a proxy that rewrites temperature and reasoning_effort and rely on the server's context default. Mingbird calls /api/chat directly, with its own compaction set to 8000.
- Medium severityWhere in paper: Section 3.8
The pre-registration came after the first rung. Hypotheses and estimators were fixed 'before the ladder's second rung started', so the 0.75B results, including the strongest surviving contrast, had already been collected. The goose comparator came from earlier data, which limits selection. Still, the 'pre-registered' label is weaker for the 0.75B contrast than the abstract implies.
- Medium severityWhere in paper: Section 3.8; Section 4.1; Section 4.2 repeat paragraph
Variance is on the same scale as the effects. Each configuration gets one temperature-0 run per cell, yet repeat collections move an arm mean by up to 0.233. Mingbird at 0.75B reads 0.664, 0.503 and 0.431 across three collections, and pi at 3.7B moves from 0.714 to 0.493 on a repeat. The minimum detectable effect is 0.22-0.39 at 80% power. Descriptive contrasts such as +0.075 vs pi at 4.7B and +0.184 vs pi at 1.9B sit below that.
- Medium severityWhere in paper: Abstract; Section 4.2; Table 5 caption
The abstract's '13 of 16 intervals exclude zero' mixes confirmatory and descriptive results. Only 4 contrasts were registered, and 2 survive Holm. The other 12 are uncorrected task-bootstrap intervals over 15-18 tasks.
- Medium severityWhere in paper: Abstract; Section 4.4
The abstract overstates the 'progress effect'. It says the harness decides whether a run leaves anything on disk. At 4.7B, pi leaves no empty cell, and the +0.075 lead is a difference in how far completed runs get. The paper says this part of the machinery is not decomposed. The loop-breaking explanation is supported by a trace and an ablation at 0.75B only.
- Medium severityWhere in paper: Section 3.5; Section 4.2; Table 4 caption
Exclusions and wall budgets interact with the results. Three infrastructure cells are excluded, two of them Mingbird's. Restoring the excluded cells at 0.75B shrinks the repeat gap from +0.476 to +0.315 and adds losses. Wall budgets are 1.3x the maxima from an earlier collection. Under the deployment reading a timeout scores zero, which hits slower arms such as pi at 3.7B hardest; 36 of the 58 rescored cells are at that rung.
- Low severityWhere in paper: Table 1; Section 4.7 (not in provided text)
Size, family and bit width are confounded on a four-point ladder. Quantization varies (Q8_0 for the two smallest Qwen tags, Q4_K_M for the others), and the 3.7B rung is a different family. The precision ladder's results fall outside the provided text.
- Low severityWhere in paper: Section 3.4; Section 3.8; Section 4.3
The scorer has limits at the top and on content. Mingbird and goose both reach 1.000 on 8 of 18 tasks at 4.7B. Milestones carry 4/7 of the score. The checker cannot detect fabricated content, and the fabricated-identifier audit does not enter the score.
Questions for the authors
- How do the baselines score if fixtures sit at paths without spaces, or on a Linux host, so the 'cd: too many arguments' failure is removed?
- What happens if a minimal repeat-call intercept plus context reset is added to pi or goose? That would test the proposed carrier outside Mingbird.
- Which Mingbird design choices or prompts were tuned on the development families that the held-out tasks mirror?
- Can the 0.75B rung be rerun with several seeds or nonzero temperature, to characterize the 0.233 collection swing?
- What are the full streak-off results at 0.75B, and does the 0.411 cost replicate at 1.9B and 4.7B?
- What explains the 4.7B depth lead over pi, given that neither the completion gate nor empty runs account for it?
- What are the per-arm fabricated-identifier rates, and do they change the ranking on retrieval and synthesis tasks?
Related in the directory
Open-source projects
Open source · Earendil Works
Open source · opencode (anomalyco)
Open source · goose (Block-initiated)
Open source · SWE-agent (Princeton & Stanford)
Open source · OpenHands (formerly All Hands AI)
Open source · Princeton NLP / SWE-agent
Commercial products
Product · Remote
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26444 · cs.RO · Nov 19, 2026
Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.
Yuzhi Huang, Jie Wu +11