PaperarXiv:2611.21299 · cs.AI · Nov 16, 2026
Turing Agent: Agentic Formal Verification for Mathematical Research
Yang Chen, Qianyi Wang, Ronghui Yang, Zhanhao Li, Yu Cong, Chen Zhang, Hangfan Zhang, Shao Zhang, Shuai Li, Toru Takisaka, Chao Xu, Yi Zhou, + 4 more
The harness patterns are reusable: checker-gated decomposition, experience records that survive compaction, and selective review. The fixed-model comparison against Claude Code and Codex CLI is a reasonable frame. The quantitative case is weaker than the abstract suggests. Proof construction loses to Claude Code with one model, the formalization metric depends on an unvalidated LLM judge, and the paper has no cost parity, no ablations and no Lean-specific baselines. The case studies extend existing developments, and one of them is mostly glue around an existing theorem.
Novelty
2 out of 5Rigor
2 out of 5Reproducibility
2 out of 5Relevance to harnesses
3 out of 5
TL;DR
A Lean formalization harness that separates theorem formalization from proof formalization. It uses a Lean-checked proof DAG, parallel subgoal agents, an experience pool that survives context compaction, and LLM semantic review on selected nodes. It beats Claude Code and Codex CLI on a statement-formalization metric scored by an LLM judge. On RLMEval proof construction with DeepSeek-V4-Pro, it loses to Claude Code. The open-problem case studies extend existing OpenAI Lean developments rather than starting from scratch.
Adversarial review by Harness Hub · Nov 19, 2026 · read full text (truncated) · Suggest a correction
Original abstract (authors’ text, select to expand)
Large language models (LLMs) are rapidly advancing mathematical discovery, moving beyond benchmark-level problem solving to tackle frontier research problems. This progress is ushering mathematics into an era of proof abundance, raising a fundamental challenge: how can we ensure the correctness of the ever-growing volume of AI-generated proofs and transform them into trustworthy, machine-verified mathematical knowledge? We introduce Turing Agent, an agent harness designed to orchestrate end-to-end formal verification of research-level mathematics. Its wo-scale architecture integrates theorem formalization and proof formalization agents, empowered by structured proof decomposition, parallel verification, accumulated proof experience, and selective semantic review. Leveraging our collaborations with theoretical computer science (TCS) researchers, we focus our evaluation on TCS, a particularly demanding domain characterized by intricate quantification, complex dependencies, and stringent semantic requirements. Against direct generation and general-purpose coding harnesses, including Codex CLI and Claude Code, Turing Agent achieves approximately 57-62% faithful formalization success on Formal TCS Bench and 83% overall pass@1 on a TCS-focused subset of RLMEval, using GLM-5.3 and DeepSeek-V4-Pro. Notably, Turing Agent, built on DeepSeek, successfully completes the formal verification of three previously unresolved TCS tasks from OpenAI's October 2026 collection. One particularly challenging TCS problem required about a hundred formally verified auxiliary claims. Moreover, by actively exploring and refining the formalization process, Turing Agent can uncover simpler alternative proofs, expose unnecessary dependencies in original proofs, and generate reusable mathematical knowledge as a byproduct of verification.
Filed underharnessorchestrationevaluationagentsllm
Summary
Turing Agent is a two-scale harness with two kinds of agent. Theorem formalization (TF) agents translate informal statements into Lean, using Mathlib and CSLib. They then run a Lean check and an LLM semantic review. Proof formalization (PF) agents decompose goals into child claims. A decomposition is accepted only after Lean verifies that the child statements imply the parent. Independent children go to parallel agents. Experience records from successful and failed attempts are saved before context compaction and supplied again afterwards. PF agents can request TF-style semantic review for intermediate claims they judge risky. A final pass rechecks the whole proof with an independent Lean kernel. An LLM review then compares the proof against the informal argument. Evaluation holds the model fixed (GLM-5.3 or DeepSeek-V4-Pro) and compares against direct generation, Codex CLI and Claude Code. Two benchmarks are used: - Formal TCS Bench (143 problems), scored by compilation plus GPT-6.1 Sol semantic review instead of the benchmark's BEq+ metric. - A 248-theorem PFR+TLB subset of RLMEval in easy mode. Section 5 describes three problems from OpenAI's October 2026 manuscript collection. In each, Turing Agent completed Lean developments that OpenAI's existing formalization had left incomplete.
Key findings
- Formal TCS Bench success (compiles and passes GPT-6.1 Sol semantic review): 57.34% with GLM-5.3 and 61.54% with DeepSeek-V4-Pro. Section 4.2 reports these as 16.78 and 32.17 points above Claude Code, the strongest baseline.
- Formal TCS Bench compilation: 97.20% (GLM-5.3) and 99.30% (DeepSeek-V4-Pro), the highest among evaluated methods (Section 4.2).
- RLMEval PFR+TLB overall pass@1 with DeepSeek-V4-Pro: Turing Agent 82.66% versus Claude Code 86.69%. That is 4.03 points behind Claude Code and 8.06 points above Codex CLI (Section 4.2).
- RLMEval with GLM-5.3: Turing Agent is reported as 'above 82%' and 2.42 points above Claude Code. The exact value is not printed in the text.
- Case studies (Section 5): the hypergraph influence bound completed 27 auxiliary proof nodes and the random 3-SAT variance bound 108 verified auxiliary nodes. The matrix-hitting case targets the characteristic-zero companion result. It mostly connects existing library theorems (universal_hitting, completeClaim_of_hitting) through a representation translation and dependent-type rewriting.
- Qualitative observations: agents change existential witnesses (e.g. C=A+B+1 instead of max(A,B)) so that linear arithmetic can discharge the goals. They also produce verified DAG nodes that the final proof never uses.
The paper at a glance
- Informal theorem + context
- TF agent: retrieve Mathlib/CSLib defs, write Lean statement
- Lean check + semantic review loop; main statement frozen
- PF agent: decompose into child claims; Lean verifies children imply parent
- Parallel PF agents on independent DAG nodes, with experience pool across compaction
- Selective TF semantic review on flagged intermediate claims
- Assemble branches; independent Lean kernel check + LLM argument review
Data table
| Method | % |
|---|---|
| Turing Agent (this paper) | 82.66% |
| Claude Code | 86.69% |
Why it matters
For harness builders, the reusable part is the design, not the headline numbers. Three patterns transfer to any long-horizon agent that has a hard checker. First, accept a task decomposition only when the checker proves the subgoals imply the parent. Second, keep an experience pool so findings survive context compaction. Third, spend semantic-review calls only on nodes the worker flags as risky. Comparing against Claude Code and Codex CLI with the model held fixed is also the right experimental frame. However, the evidence does not show that the specialized harness beats a general coding harness on proof construction: with DeepSeek-V4-Pro, Claude Code scores higher. The statement-formalization gain rests on a single LLM judge whose criterion overlaps the harness's own internal review. The paper does not compare against existing Lean-specific agents, and it gives no ablations, cost figures or code, so builders cannot tell which component matters. Outside formal verification, the main takeaways are checker-gated decomposition and memory that survives compaction.
Issues (10)
- High severityWhere in paper: Abstract vs Section 4.2 (Proof Formalization) and Figure 6
The abstract presents '83% overall pass@1' on RLMEval as a result 'against ... Claude Code'. With DeepSeek-V4-Pro, Turing Agent scores 82.66% and Claude Code 86.69%, so the specialized harness loses to the general coding harness on proof construction for one of the two models. With GLM-5.3 the margin is only 2.42 points, and the exact score is not printed.
- High severityWhere in paper: Section 4.1 (Evaluation Metrics), Section 3.1, Section 4.2
Formal TCS Bench success is scored by a single LLM judge (GPT-6.1 Sol), which replaces the benchmark's own BEq+ metric. No human agreement study, second-judge check or error analysis is reported, even though TCS collaborators are said to provide expertise for assessing faithfulness. Turing Agent internally runs LLM semantic review on the same properties (domains, assumptions, quantifiers, conclusion). The 16.78 and 32.17 point margins over Claude Code may therefore partly reflect alignment with the judge rather than faithfulness.
- High severityWhere in paper: Section 4.1 (Models and Baselines), Section 3.2.1
No compute or budget parity is reported. The paper gives no token counts, wall-clock limits, number of parallel PF agents, turn limits, or Lean tooling configuration for Codex CLI and Claude Code. Turing Agent runs several concurrent agents plus extra review calls, so higher scores may reflect more inference spent. Baseline prompts and setup are not described.
- High severityWhere in paper: Abstract; Section 5.1; Appendix A
The open-problem results are framed as completing 'previously unresolved TCS tasks'. All three build on OpenAI's existing partial Lean developments. The matrix-hitting case targets the characteristic-zero companion result, not the positive-characteristic theorem in the problem title. Its final proof reuses the existing universal_hitting theorem, and none of the agent's eight planned auxiliary goals entered the final proof. The agent's work was mainly representation translation and dependent-type plumbing.
- Medium severityWhere in paper: Section 2.2; Section 4.1 (Models and Baselines)
Only general-purpose coding harnesses and direct generation are baselines. Lean-specific agent systems that the related work names as doing similar decomposition, retrieval and formalization (Hilbert, Numina-Lean-Agent, Aristotle, Archon, MMAT) are not compared. The claimed novelty of DAG decomposition and semantic checking is therefore not tested against the closest prior work.
- Medium severityWhere in paper: Figure 3 caption; Section 5 intro; Section 5.3; Figure 7; Section 1
Selection is unclear. Of 22 TCS problems not fully verified in Lean, three are highlighted. The paper does not say how many were attempted, how many failed, or at what cost. The paper notes that verified DAG nodes can go unused in final proofs (Figure 7 shows gray unused nodes for the hypergraph case). It does not report how many of the 108 3-SAT nodes the final proof actually uses, so 'about a hundred formally verified auxiliary claims' does not measure what the proof requires.
- Medium severityWhere in paper: Sections 3 and 4
There are no ablations of DAG decomposition, parallelism, the experience pool or selective review. The gains cannot be attributed to any of the advertised components.
- Medium severityWhere in paper: Section 4.1 (Evaluation Metrics), Section 4.2
Pass@1 comes from one final submission per problem, with no repeated runs, variance or confidence intervals. Gaps of 2.42 and 4.03 points on 248 theorems are small enough that run-to-run variance could change them.
- Medium severityWhere in paper: Section 4.1 (Benchmarks)
RLMEval PFR and TLB are public Lean projects, and easy mode exposes project-specific auxiliary lemmas. Training-data contamination is not discussed for either model. The same concern applies to Formal TCS Bench, whose statements come from recent papers and have reference formalizations.
- Low severityWhere in paper: Front matter; Section 3.2.2; Section 4.2
No code or data URL is printed, only a project website. Prompts, review rubrics and the trigger policy for selective review are described only in prose. Direct-generation baseline numbers are not printed in the text.
Questions for the authors
- What were the token, wall-clock and parallel-agent budgets for Turing Agent versus Codex CLI and Claude Code on each benchmark?
- How well does GPT-6.1 Sol semantic review agree with the TCS researchers' judgments, and how do the methods rank under BEq+?
- Which components (DAG gating, experience pool, selective review, parallelism) account for the Formal TCS Bench gains?
- How does Turing Agent compare with Lean-specific agents such as Hilbert or Numina-Lean-Agent under the same model?
- Of the 22 TCS problems not fully verified, how many were attempted, how many completed, and at what cost?
- How many of the 108 verified nodes in the 3-SAT case are used in the final proof?
- What is the exact GLM-5.3 RLMEval score, and how much does it vary across repeated runs?
Related in the directory
Open-source projects
Open source · Anthropic
Open source · OpenAI
Related papers
arXiv:2611.26595 · cs.CV · Nov 19, 2026
Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.
Yiyan Ji, Qixun Wang +11
arXiv:2611.26553 · cs.CR · Nov 19, 2026
Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.
Mengxiao Wang, Mahbub Alam +2
arXiv:2611.26450 · cs.AI · Nov 19, 2026
Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).
Tanmay Sah