Harness Hub
All papers

Turing Agent: Agentic Formal Verification for Mathematical Research

Yang Chen, Qianyi Wang, Ronghui Yang, Zhanhao Li, Yu Cong, Chen Zhang, Hangfan Zhang, Shao Zhang, Shuai Li, Toru Takisaka, Chao Xu, Yi Zhou, + 4 more

Verdict: Useful

The harness patterns are reusable: checker-gated decomposition, experience records that survive compaction, and selective review. The fixed-model comparison against Claude Code and Codex CLI is a reasonable frame. The quantitative case is weaker than the abstract suggests. Proof construction loses to Claude Code with one model, the formalization metric depends on an unvalidated LLM judge, and the paper has no cost parity, no ablations and no Lean-specific baselines. The case studies extend existing developments, and one of them is mostly glue around an existing theorem.

  • Novelty

    2 out of 5
  • Rigor

    2 out of 5
  • Reproducibility

    2 out of 5
  • Relevance to harnesses

    3 out of 5

TL;DR

A Lean formalization harness that separates theorem formalization from proof formalization. It uses a Lean-checked proof DAG, parallel subgoal agents, an experience pool that survives context compaction, and LLM semantic review on selected nodes. It beats Claude Code and Codex CLI on a statement-formalization metric scored by an LLM judge. On RLMEval proof construction with DeepSeek-V4-Pro, it loses to Claude Code. The open-problem case studies extend existing OpenAI Lean developments rather than starting from scratch.

Adversarial review by Harness Hub · Nov 19, 2026 · read full text (truncated) · Suggest a correction

Original abstract (authors’ text, select to expand)

Large language models (LLMs) are rapidly advancing mathematical discovery, moving beyond benchmark-level problem solving to tackle frontier research problems. This progress is ushering mathematics into an era of proof abundance, raising a fundamental challenge: how can we ensure the correctness of the ever-growing volume of AI-generated proofs and transform them into trustworthy, machine-verified mathematical knowledge? We introduce Turing Agent, an agent harness designed to orchestrate end-to-end formal verification of research-level mathematics. Its wo-scale architecture integrates theorem formalization and proof formalization agents, empowered by structured proof decomposition, parallel verification, accumulated proof experience, and selective semantic review. Leveraging our collaborations with theoretical computer science (TCS) researchers, we focus our evaluation on TCS, a particularly demanding domain characterized by intricate quantification, complex dependencies, and stringent semantic requirements. Against direct generation and general-purpose coding harnesses, including Codex CLI and Claude Code, Turing Agent achieves approximately 57-62% faithful formalization success on Formal TCS Bench and 83% overall pass@1 on a TCS-focused subset of RLMEval, using GLM-5.3 and DeepSeek-V4-Pro. Notably, Turing Agent, built on DeepSeek, successfully completes the formal verification of three previously unresolved TCS tasks from OpenAI's October 2026 collection. One particularly challenging TCS problem required about a hundred formally verified auxiliary claims. Moreover, by actively exploring and refining the formalization process, Turing Agent can uncover simpler alternative proofs, expose unnecessary dependencies in original proofs, and generate reusable mathematical knowledge as a byproduct of verification.

Filed underharnessorchestrationevaluationagentsllm

Summary

Turing Agent is a two-scale harness with two kinds of agent. Theorem formalization (TF) agents translate informal statements into Lean, using Mathlib and CSLib. They then run a Lean check and an LLM semantic review. Proof formalization (PF) agents decompose goals into child claims. A decomposition is accepted only after Lean verifies that the child statements imply the parent. Independent children go to parallel agents. Experience records from successful and failed attempts are saved before context compaction and supplied again afterwards. PF agents can request TF-style semantic review for intermediate claims they judge risky. A final pass rechecks the whole proof with an independent Lean kernel. An LLM review then compares the proof against the informal argument. Evaluation holds the model fixed (GLM-5.3 or DeepSeek-V4-Pro) and compares against direct generation, Codex CLI and Claude Code. Two benchmarks are used: - Formal TCS Bench (143 problems), scored by compilation plus GPT-6.1 Sol semantic review instead of the benchmark's BEq+ metric. - A 248-theorem PFR+TLB subset of RLMEval in easy mode. Section 5 describes three problems from OpenAI's October 2026 manuscript collection. In each, Turing Agent completed Lean developments that OpenAI's existing formalization had left incomplete.

Key findings

  • Formal TCS Bench success (compiles and passes GPT-6.1 Sol semantic review): 57.34% with GLM-5.3 and 61.54% with DeepSeek-V4-Pro. Section 4.2 reports these as 16.78 and 32.17 points above Claude Code, the strongest baseline.
  • Formal TCS Bench compilation: 97.20% (GLM-5.3) and 99.30% (DeepSeek-V4-Pro), the highest among evaluated methods (Section 4.2).
  • RLMEval PFR+TLB overall pass@1 with DeepSeek-V4-Pro: Turing Agent 82.66% versus Claude Code 86.69%. That is 4.03 points behind Claude Code and 8.06 points above Codex CLI (Section 4.2).
  • RLMEval with GLM-5.3: Turing Agent is reported as 'above 82%' and 2.42 points above Claude Code. The exact value is not printed in the text.
  • Case studies (Section 5): the hypergraph influence bound completed 27 auxiliary proof nodes and the random 3-SAT variance bound 108 verified auxiliary nodes. The matrix-hitting case targets the characteristic-zero companion result. It mostly connects existing library theorems (universal_hitting, completeClaim_of_hitting) through a representation translation and dependent-type rewriting.
  • Qualitative observations: agents change existential witnesses (e.g. C=A+B+1 instead of max(A,B)) so that linear arithmetic can discharge the goals. They also produce verified DAG nodes that the final proof never uses.

The paper at a glance

01Informal theorem +context02TF agent: retrieveMathlib/CSLib defs,write Leanstatement03Lean check +semantic reviewloop; mainstatement frozen04PF agent: decomposeinto child claims;Lean verifieschildren implyparent05Parallel PF agentson independent DAGnodes, withexperience poolacross compaction06Selective TFsemantic review onflaggedintermediate claims07Assemble branches;independent Leankernel check + LLMargument review
  1. Informal theorem + context
  2. TF agent: retrieve Mathlib/CSLib defs, write Lean statement
  3. Lean check + semantic review loop; main statement frozen
  4. PF agent: decompose into child claims; Lean verifies children imply parent
  5. Parallel PF agents on independent DAG nodes, with experience pool across compaction
  6. Selective TF semantic review on flagged intermediate claims
  7. Assemble branches; independent Lean kernel check + LLM argument review
Fig. 1 — How the paper works, step by step (arXiv:2611.21299).
Turing Agent (this paper)82.66%Claude Code86.69%
Fig. 2 — RLMEval (PFR+TLB, 248 theorems, easy mode) overall pass@1, DeepSeek-V4-Pro, higher is better. Source: Section 4.2 / Figure 6, arXiv:2611.21299.
Data table
RLMEval (PFR+TLB, 248 theorems, easy mode) overall pass@1, DeepSeek-V4-Pro
Method%
Turing Agent (this paper)82.66%
Claude Code86.69%

Why it matters

For harness builders, the reusable part is the design, not the headline numbers. Three patterns transfer to any long-horizon agent that has a hard checker. First, accept a task decomposition only when the checker proves the subgoals imply the parent. Second, keep an experience pool so findings survive context compaction. Third, spend semantic-review calls only on nodes the worker flags as risky. Comparing against Claude Code and Codex CLI with the model held fixed is also the right experimental frame. However, the evidence does not show that the specialized harness beats a general coding harness on proof construction: with DeepSeek-V4-Pro, Claude Code scores higher. The statement-formalization gain rests on a single LLM judge whose criterion overlaps the harness's own internal review. The paper does not compare against existing Lean-specific agents, and it gives no ablations, cost figures or code, so builders cannot tell which component matters. Outside formal verification, the main takeaways are checker-gated decomposition and memory that survives compaction.

Issues (10)

  • High severity

    The abstract presents '83% overall pass@1' on RLMEval as a result 'against ... Claude Code'. With DeepSeek-V4-Pro, Turing Agent scores 82.66% and Claude Code 86.69%, so the specialized harness loses to the general coding harness on proof construction for one of the two models. With GLM-5.3 the margin is only 2.42 points, and the exact score is not printed.

  • High severity

    Formal TCS Bench success is scored by a single LLM judge (GPT-6.1 Sol), which replaces the benchmark's own BEq+ metric. No human agreement study, second-judge check or error analysis is reported, even though TCS collaborators are said to provide expertise for assessing faithfulness. Turing Agent internally runs LLM semantic review on the same properties (domains, assumptions, quantifiers, conclusion). The 16.78 and 32.17 point margins over Claude Code may therefore partly reflect alignment with the judge rather than faithfulness.

  • High severity

    No compute or budget parity is reported. The paper gives no token counts, wall-clock limits, number of parallel PF agents, turn limits, or Lean tooling configuration for Codex CLI and Claude Code. Turing Agent runs several concurrent agents plus extra review calls, so higher scores may reflect more inference spent. Baseline prompts and setup are not described.

  • High severity

    The open-problem results are framed as completing 'previously unresolved TCS tasks'. All three build on OpenAI's existing partial Lean developments. The matrix-hitting case targets the characteristic-zero companion result, not the positive-characteristic theorem in the problem title. Its final proof reuses the existing universal_hitting theorem, and none of the agent's eight planned auxiliary goals entered the final proof. The agent's work was mainly representation translation and dependent-type plumbing.

  • Medium severity

    Only general-purpose coding harnesses and direct generation are baselines. Lean-specific agent systems that the related work names as doing similar decomposition, retrieval and formalization (Hilbert, Numina-Lean-Agent, Aristotle, Archon, MMAT) are not compared. The claimed novelty of DAG decomposition and semantic checking is therefore not tested against the closest prior work.

  • Medium severity

    Selection is unclear. Of 22 TCS problems not fully verified in Lean, three are highlighted. The paper does not say how many were attempted, how many failed, or at what cost. The paper notes that verified DAG nodes can go unused in final proofs (Figure 7 shows gray unused nodes for the hypergraph case). It does not report how many of the 108 3-SAT nodes the final proof actually uses, so 'about a hundred formally verified auxiliary claims' does not measure what the proof requires.

  • Medium severity

    There are no ablations of DAG decomposition, parallelism, the experience pool or selective review. The gains cannot be attributed to any of the advertised components.

  • Medium severity

    Pass@1 comes from one final submission per problem, with no repeated runs, variance or confidence intervals. Gaps of 2.42 and 4.03 points on 248 theorems are small enough that run-to-run variance could change them.

  • Medium severity

    RLMEval PFR and TLB are public Lean projects, and easy mode exposes project-specific auxiliary lemmas. Training-data contamination is not discussed for either model. The same concern applies to Formal TCS Bench, whose statements come from recent papers and have reference formalizations.

  • Low severity

    No code or data URL is printed, only a project website. Prompts, review rubrics and the trigger policy for selective review are described only in prose. Direct-generation baseline numbers are not printed in the text.

Questions for the authors

  • What were the token, wall-clock and parallel-agent budgets for Turing Agent versus Codex CLI and Claude Code on each benchmark?
  • How well does GPT-6.1 Sol semantic review agree with the TCS researchers' judgments, and how do the methods rank under BEq+?
  • Which components (DAG gating, experience pool, selective review, parallelism) account for the Formal TCS Bench gains?
  • How does Turing Agent compare with Lean-specific agents such as Hilbert or Numina-Lean-Agent under the same model?
  • Of the 22 TCS problems not fully verified, how many were attempted, how many completed, and at what cost?
  • How many of the 108 verified nodes in the 3-SAT case are used in the final proof?
  • What is the exact GLM-5.3 RLMEval score, and how much does it vary across repeated runs?

Related in the directory

Open-source projects

General-purpose coding harness used as the strongest baseline. It beats Turing Agent on RLMEval with DeepSeek-V4-Pro.
Codex CLI is the other general-purpose coding harness baseline, run with the same underlying models.

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).

Tanmay Sah

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.