Harness Hub
All papers

Procedure without Judgment: What Small Language Models Learn under an Exact Verifier

Arunkumar V, Manoranjan Gandhudi, Gangadharan G. R., Arun Prakash, S. Senthilkumar

Verdict: Incremental

The paper is transparently reported and exactly scored. Its main positive result, though, is that a 4B model can imitate a trivial |F|-check algorithm on the one policy it was trained on, without significantly beating Haiku there and losing badly elsewhere. The RL result is a single short run. What remains useful for harness builders is the observation that format-termination rules and one-sentence procedural prompts move scores substantially.

  • Novelty

    3 out of 5
  • Rigor

    3 out of 5
  • Reproducibility

    2 out of 5
  • Relevance to harnesses

    2 out of 5

TL;DR

The setting is one hand-written 10-feature authorization policy (GlassGate) where minimal sufficient reasons can be computed exactly, and a sufficiency-check tool reduces the task to a known |F|-call deletion loop. A Qwen3-4B student fine-tuned on traces of that loop reaches 0.922 exact on 64 in-distribution episodes. It is not significantly ahead of Haiku 4.5 at home (p=0.12) and falls far behind it on unseen policies. One reported 25-step RL run changes the query pattern but not held-out accuracy.

Adversarial review by Harness Hub · Nov 18, 2026 · read full text · Suggest a correction

Original abstract (authors’ text, select to expand)

When an authorization policy blocks an agent's tool call, the audit log should name the facts that forced the decision and nothing more. For an executable policy over an enumerable context space this minimal sufficient reason can be computed exactly, so an explanation is scored against ground truth rather than a proxy for it. We use that exactness to ask what a model can learn from a verifier that has no error in it. The explainer sees one request and one decision, never the policy, and has thirty turns in which it may call the policy on altered requests and ask whether a candidate set of facts is sufficient. Frontier models find the exact reason about half the time with the tools alone, and one sentence telling them to shrink the answer lifts Haiku 4.5 from 58% to 75%. A 4B model fine-tuned on 2000 traces of the deletion algorithm reaches 92% on the policy it was trained on. Teacher kind is what moves this: at matched data size the algorithm-taught student leads the frontier-taught one in all three training seeds, and the two arms never overlap. What the student learns is a fixed schedule of sufficiency checks, and the schedule does not travel. On three public policies it never saw, its answers stay sufficient (94%) and stop being minimal (48%), while Haiku under the same sentence is exact on all 47 episodes. Reinforcement learning from the exact reward leaves held-out accuracy where it was and inverts the strategy, from probing the policy to spending the entire budget on sufficiency checks. The oracle made the effect of training legible, but it did not make training better.

Authors’ note

17 pages

Filed undertool-useagentsllm

Summary

**Task.** The explainer sees one request and the policy's allow/block decision, never the policy. It has 30 turns of two tools, then commits a feature set S: - decide(x') probes the policy on an altered request. - check(S) asks whether a candidate set is sufficient. An oracle enumerates all minimal sufficient reasons (MSRs) over the 124,416-context space, so S is scored exactly. **Frontier models.** Results on 120 shared GlassGate episodes with both tools (Table 6): - Haiku 4.5: 0.583. - Sonnet 5: 0.500, with a forced rate of 0.46 from malformed replies. - Gemini 3.5 Flash-Lite: 0.158. - Opus 5: 1.000 on the 21 episodes it did not refuse, an easier subset. One shrink sentence lifts Haiku to 75% and Flash-Lite to 43% (Section 3). **Distillation.** Qwen3-4B (LoRA r=16, 600 steps) is trained on one of three trace sets: - 2000 deletion-algorithm traces (Alg-2000). - 148 deletion-algorithm traces (Alg-148). - 148 Opus/Sonnet traces kept only when exact (Frontier-148). On 64 GlassGate episodes with the shrink instruction (Table 1): - Alg-2000: 0.922. - Alg-148: 0.688. - Frontier-148: 0.531. - Haiku+shrink: 0.812. - Deletion algorithm itself: 1.000. **Transfer.** On three OPA/Cedar policies (Table 8, n=48 pooled), Alg-2000 falls to 0.479 exact while sufficiency stays at 0.938. Alg-public, trained only on public policies, reaches 28% on GlassGate (Section 5). **RL.** REINFORCE from Frontier-148 under reward r3 moves held-out exact from 0.531 to 0.562 (p=0.80). The student goes from 3.5 probes and 25.5 checks to 0.0 probes and 30.0 checks. A first reward, r1, was gamed by padding, and a second warm start diverged to 6% exact (Section 6).

Key findings

  • Frontier accuracy with both tools on 120 GlassGate episodes (Table 6): Haiku 4.5 0.583, Sonnet 5 0.500, Flash-Lite 0.158. Opus 5 scores 1.000 on only 21 non-refused episodes; on that subset Haiku and Sonnet both reach 76% (Section 3).
  • One shrink sentence lifts Haiku from 58% to 75% and Flash-Lite 28 points to 43%, both p <= 10^-4 paired. Sonnet goes from 50% to 51% because 55 of 120 episodes end in a forced empty answer (Section 3).
  • On 64 GlassGate episodes (Table 1, Table 2), Alg-2000 scores 0.922 against Frontier-148 at 0.531 (+0.391, p<.001). Against Haiku 4.5 + shrink at 0.812 the gap is +0.109, p=0.12, not significant. Against the deletion ceiling it is -0.078, p=0.06.
  • Matched size, Alg-148 vs Frontier-148 (Table 7): within-seed differences are +16 (p=0.11), +47 and +61 points. Seed means are 0.844 (sd 0.143) vs 0.432 (sd 0.089). On the stratified 64 the seed-0 comparison does not reach significance.
  • Off-distribution (Table 8, 48 pooled episodes), Alg-2000 is 0.479 exact with 0.938 Suff, against Frontier-148 at 0.500 exact. Alg-2000 trails instructed Haiku on all three public policies, by 50, 38 and 73 points (Section 5).
  • RL under r3 raised exact on 24 training-time contexts from 63% to 83%, but held-out exact moved only from 0.531 to 0.562 (p=0.80). Under r1, exact fell from 58% to 33% and surplus rose from 0.5 to 2.0 (Section 6, Appendix A.2).
  • Compute is about 30 T4-hours on free Kaggle accounts, plus $3.35 Gemini and $3.62 Anthropic API spend; most Haiku and Sonnet arms ran on a Claude CLI subscription (Table 4).

The paper at a glance

01Request x anddecision y from thehidden executablepolicy02Explainer: up to 30turns of decide(x')probes and check(S)sufficiency queries03final(S) commitsthe reason set04Exact oracleenumerates all MSRsover 124,416contexts05Scoring: Exact,Suff, CPrec,surplus, probes,checks06SFT of Qwen3-4B(LoRA r=16, 600steps) ondeletion-algorithmor filtered…07REINFORCE withgroup baseline fromexact reward, 25steps, fromFrontier-148
  1. Request x and decision y from the hidden executable policy
  2. Explainer: up to 30 turns of decide(x') probes and check(S) sufficiency queries
  3. final(S) commits the reason set
  4. Exact oracle enumerates all MSRs over 124,416 contexts
  5. Scoring: Exact, Suff, CPrec, surplus, probes, checks
  6. SFT of Qwen3-4B (LoRA r=16, 600 steps) on deletion-algorithm or filtered Opus/Sonnet traces
  7. REINFORCE with group baseline from exact reward, 25 steps, from Frontier-148
Fig. 1 — How the paper works, step by step (arXiv:2611.21529).
base 4B (prompt only)0 fractionFrontier-1480.531 fractionFrontier-148 + RL, exact reward0.562 fractionAlg-1480.688 fractionAlg-2000 (this paper)0.922 fractionHaiku 4.5 + shrink0.812 fractionSonnet 5 + shrink0.562 fractionsingle-flip heuristic0.516 fractiondeletion algorithm (ceiling)1 fraction
Fig. 2 — Exact minimal sufficient reason, GlassGate, 64 shared episodes, shrink instruction, higher is better. Source: Table 1, arXiv:2611.21529.
Data table
Exact minimal sufficient reason, GlassGate, 64 shared episodes, shrink instruction
Methodfraction
base 4B (prompt only)0 fraction
Frontier-1480.531 fraction
Frontier-148 + RL, exact reward0.562 fraction
Alg-1480.688 fraction
Alg-2000 (this paper)0.922 fraction
Haiku 4.5 + shrink0.812 fraction
Sonnet 5 + shrink0.562 fraction
single-flip heuristic0.516 fraction
deletion algorithm (ceiling)1 fraction

Why it matters

Relevance to harness builders is narrow. The task is one where a harness exposes an exact sufficiency check, and with that tool a ten-line deletion loop solves every episode in |F| calls with no model at all. The trained 4B student is an imitation of that loop that works only on the policy it was trained on. The parts a harness builder can use are about harness behaviour. First, a two-malformed-replies termination rule turns Sonnet 5's score into a format-compliance number (forced 0.46 on GlassGate). Second, a one-sentence procedural instruction lifts Haiku 4.5 by 17 points. Third, off-the-shelf Haiku with that sentence generalises to unseen policies where the fine-tuned student does not. The practical reading for anyone building policy-gated tool calls with audit logging is to call the deletion algorithm from code when an exact sufficiency oracle exists. The prerequisite is an enumerable categorical context space, which most real tool policies do not offer. The RL section is one reported 25-step leg plus a replicate that diverged, so it does not support a general claim about exact-reward RL.

Issues (8)

  • High severity

    The task is solved by a trivial procedure once the check tool exists. The deletion algorithm scores 1.000 in exactly |F| check calls. Learning to imitate it is closer to format imitation than to explanation ability, and the comparison that matters for deployment (model vs a code loop) is lost by every model. The check tool also requires exhaustive enumeration of the context space, which the paper itself bounds to categorical policies (Section 8). The motivating audit-log use case is therefore not shown to extend beyond toy-sized policies.

  • High severity

    The headline 92% is in-distribution only and does not beat the frontier. Alg-2000 vs Haiku 4.5 + shrink is +0.109, p=0.12, n=64. On the three public policies Alg-2000 is behind Haiku by 50, 38 and 73 points. Pooled, its 0.479 exact is no better than Frontier-148's 0.500 (Table 8). The abstract leads with 92% and the teacher-kind framing before conceding that the student loses off-distribution.

  • Medium severity

    The teacher-kind comparison is confounded and only partly significant. Seed 0 is +16 at p=0.11. A 3-vs-3 complete separation has an exact permutation p of 0.10 (Section 8). The two training sets also differ in more than teacher kind. Algorithm traces are a fixed check-per-feature script with a one-line rationale. Frontier traces are unedited free-form transcripts with probing. Frontier traces were kept only when Opus or Sonnet committed to an exact answer. Opus refused most episodes and was exact on an easier subset, and Sonnet's failures concentrate on large-MSR episodes. So the frontier training set is likely skewed toward small-MSR episodes, and its difficulty distribution is not reported.

  • Medium severity

    The frontier baselines are shaped by the harness. Sonnet 5 is bounded by the two-malformed-replies rule (forced 0.46 in Table 6, 0.42 in Table 1). Thinking is disabled for all frontier models. Opus is reported on 21 easier non-refused episodes. All frontier rows are single runs, while students get 220 tokens per turn of greedy decoding.

  • Medium severity

    The RL conclusion ('did not make training better') rests on very little. There is one reported 25-step leg with 4 rollouts per context and a 15-turn training budget against 30 at evaluation. A replicate diverged to 6%. The warm start, Frontier-148, already sits at the single-flip floor (+0.016, p=1.00, Table 2), and RL from Alg-2000 never took a gradient. The 30-check behaviour may partly reflect the budget mismatch.

  • Medium severity

    Train/evaluation separation on GlassGate is not stated. Students are trained on 2000 or 148 GlassGate episodes and evaluated on 64 GlassGate episodes. The text does not say whether evaluation contexts are excluded from training contexts, or whether frontier-teacher episodes overlap the evaluation set.

  • Low severity

    The Figure 5 caption says Sonnet's trace 'ends in a sufficient but non-minimal answer', but the trace shows final({}) marked [not sufficient].

  • Low severity

    Episode counts differ without explanation. Haiku is reported exact on 'all 47' public-policy episodes, while the students' pooled set is n=48 (Tables 7, 8), and per-policy intersections are not shown in the text. The public policies have 16 episodes each, the students ran only three of the five, and there is one base model and one size. Code and records are promised on publication only.

Questions for the authors

  • Are the 64 evaluation contexts excluded from the 2000/148 training contexts and from the frontier-teacher episodes?
  • What is the MSR-size distribution of the 148 frontier-teacher traces compared with the algorithm traces, and how many came from Opus vs Sonnet?
  • How does Sonnet 5 score with re-prompting instead of termination after two malformed replies, or with thinking enabled?
  • Does training on deletion traces over policies with mixed |F| produce a schedule that scales with the feature count at hand?
  • Is the RL student's 30.0 checks at evaluation tied to the 15-turn training budget?
  • Why does the Figure 5 caption describe Sonnet's final({}) as sufficient but non-minimal?

Related in the directory

Open-source projects

Most Haiku 4.5 and Sonnet 5 frontier episodes were run through a Claude CLI subscription (Table 4).

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).

Tanmay Sah

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.