Harness Hub
All papers

CompCert4Lean: Porting a Verified C Compiler to Lean in Six Weeks with Parallel Coding Agents

Cong Wang, Zhiyang Chen, Fan Long, Martin Rinard

Verdict: Useful

The migration is a substantial, checkable artifact. The orchestration workflow, defect history and token accounting are informative for anyone running large parallel agent fleets. The comparative optimization claim is one uncontrolled run per environment against a shared external baseline, and it is presented in the abstract as a measured benefit. The human-effort figure rests on a narrow self-defined taxonomy. Reproducibility depends on proprietary models, and no code URL appears in the provided text.

  • Novelty

    4 out of 5
  • Rigor

    3 out of 5
  • Reproducibility

    2 out of 5
  • Relevance to harnesses

    3 out of 5

TL;DR

One expert directed parallel Codex agents (GPT-5.6 Sol, then GPT-6-astra) to port CompCert 3.17's verified core from Rocq to Lean in about six weeks, closing all 4,908 recorded proof obligations. The workflow partitions work by dependency, audits definitions before proving, and grinds proofs in parallel. The headline Lean-vs-Rocq optimization result rests on one campaign per environment. It is measured against original CompCert 3.17, not against each campaign's own starting compiler.

Adversarial review by Harness Hub · Nov 19, 2026 · read full text (truncated) · Suggest a correction

Original abstract (authors’ text, select to expand)

We present CompCert4Lean, a large-scale port of CompCert 3.17's verified compiler core from Rocq to Lean, completed in approximately six weeks by one expert directing parallel LLM agents. Our migration workflow partitions the development along its dependencies, audits translated definitions and theorem statements before large-scale proof reconstruction, and then reconstructs proofs in parallel against stable interfaces. The resulting compiler covers typed CompCert C through x86-64 assembly, discharges all 4,908 recorded proof obligations without project-specific axioms, and produces executable Linux binaries. The workflow enabled the expert to concentrate on organizing the work rather than supplying technical solutions. Of 1,071 human interactions, 88.8% concerned workflow and coordination, while only three supplied technical direction. Optimization campaigns further demonstrate the benefit of the Lean port for agent-driven verified development. Under matched budgets of 24 active hours, GPT-6-astra produced a Lean candidate with 11.7% lower generated-program runtime across 29 small-tests programs, compared with 2.3% lower for its Rocq candidate, both relative to original CompCert 3.17 and while preserving the end-to-end correctness theorem. These results show that parallel LLM agents can support large-scale migration with limited human technical direction and subsequently improve the migrated compiler while retaining its formal guarantees.

Authors’ note

39 pages, including appendices

Filed undercoding-agentsruntimeagentsllm

Summary

The paper ports CompCert 3.17's compilation pipeline into Lean. The scope runs from the typed CompCert C AST to the x86-64 assembly AST. It covers all 21 configured pass slots, the shared foundations including Flocq, and the end-to-end backward-simulation theorem. Agents ran in Codex: GPT-5.6 Sol before September 17, 2026, and GPT-6-astra afterwards. The workflow has seven stages: - Risk discovery found 30 issues, which fed 5 feasibility gates. - Definitions-first translation across 166 units, with canonical ownership and disjoint writable files. - Iterative audit and repair over 11 broad review rounds. The first round found 1,091 theorem statements reduced to True. - Bounded, batched proof grinding: up to four obligations per batch, with partial acceptance of checked progress. - Human-guided agent sessions to close the 471 obligations left when the expert stopped automatic reconstruction. - Integration with the reused OCaml frontend and printer through an unverified serialization bridge. Reported results: - No axioms beyond propext, Classical.choice and Quot.sound across 75,348 project declarations. - All 139 applicable small tests pass at O0 and O2. - RTL is identical after all nine stages for 18,807 programs at O2 and 18,808 at O0. - Results are bit-identical on 37.4 million specification-function inputs. Correspondence with the Rocq development is supported by audits and testing, not by a mechanized equivalence. A second experiment gives GPT-6-astra 24 active hours to optimize each compiler. It reports runtime reductions relative to original CompCert 3.17. The provided text is truncated before Section 5.3 and the appendices. Table 7, the campaign details and the human-message classification procedure were not available for review.

Key findings

  • All 4,908 recorded proof obligations discharged with no project-specific axioms. A transitive audit of 75,348 project declarations finds only Lean's three standard axioms (Section 1, Table 1).
  • The migrated core is 173 Lean modules and 391,963 physical lines, including comments and blanks. The corresponding 154 Rocq modules are 149,038 lines (Section 4, Table 1).
  • The first definition audit found 1,091 theorem statements reduced to True that translation-time builds had missed (Sections 1, 3.3, Figure 2).
  • Recorded usage is 83,528.3M tokens over 12,435 sessions, and cached input is 93.75% of input. 13 coordinator sessions and 2,989 external-worker sessions lack usage counters (Table 2, Section 4).
  • Review and repair (phase 4) took about 59% of recorded tokens. The last broad-impact defect was found in round 3 of 11 completed rounds (Section 4, Table 3).
  • Of 1,071 human messages across 211 sessions: 951 (88.8%) were workflow, 117 (10.9%) technical review or discussion, and 3 (0.3%) technical direction (Section 4.1, Figure 3).
  • The expert stopped automatic proof reconstruction with 471 obligations remaining. These were closed in human-guided agent sessions (Sections 3.4-3.5).
  • Two floating-point translation defects surfaced only during proof finalization: a missing normalization and a theorem missing its format assumptions (Section 3.5).
  • After 24 active hours of GPT-6-astra optimization, runtime was 11.7% lower for the Lean candidate vs 2.3% for Rocq on 29 small-tests programs. On ten SPEC inputs it was 6.9% vs 2.2%. Both are relative to original CompCert 3.17 and include pre-existing differences between the starting compilers (Section 1, Figure 1).
  • In the optimization campaigns, 106 Lean candidate variants completed full compiler and proof builds vs 40 for Rocq, under 'broadly comparable' recorded token usage (Section 1).

The paper at a glance

01Determine scope andtrust boundary02Resolve migrationgates (30 risks, 5gates)03Migrate definitionsin parallel (166units)04Audit and repairdefinitions (11rounds)05Parallel proofgrinding (batchesof up to 4)06Human-guidedsessions onremaining 471obligations07Integrate,validate, optimize
  1. Determine scope and trust boundary
  2. Resolve migration gates (30 risks, 5 gates)
  3. Migrate definitions in parallel (166 units)
  4. Audit and repair definitions (11 rounds)
  5. Parallel proof grinding (batches of up to 4)
  6. Human-guided sessions on remaining 471 obligations
  7. Integrate, validate, optimize
Fig. 1 — How the paper works, step by step (arXiv:2611.21084).
CompCert4Lean candidate (this paper)11.7%Rocq CompCert candidate2.3%
Fig. 2 — Generated-program runtime reduction vs original CompCert 3.17, 29 small-tests programs, after 24 active hours of GPT-6-astra optimization (includes pre-existing differences between starting compilers), higher is better. Source: Figure 1 / Section 1, arXiv:2611.21084.
Data table
Generated-program runtime reduction vs original CompCert 3.17, 29 small-tests programs, after 24 active hours of GPT-6-astra optimization (includes pre-existing differences between starting compilers)
Method%
CompCert4Lean candidate (this paper)11.7%
Rocq CompCert candidate2.3%

Why it matters

The compiler itself is not something most harness builders will use. The orchestration account is relevant: it is a detailed public record of about 12,435 coding-agent sessions (coordinator, built-in subagents and external workers) run against one tightly coupled codebase. Patterns that transfer to other harnesses: - Dependency-ordered leasing with disjoint writable files and canonical ownership. - Freezing definitions and statements before workers prove against them. - Audit rounds that invalidate prior evidence when upstream providers change. - Accepting checked partial progress from a worker. - Small obligation batches to manage context. - A rule that a stalled search never justifies weakening a target. The 1,091 statements silently reduced to True show that a green build and a passing checker do not confirm intent. Any harness that lets agents write both specs and proofs (or tests and code) needs an independent statement audit. The token accounting shows that review and repair, not proving, took about 59% of recorded usage. The cost figures, however, are incomplete. The Lean-vs-Rocq optimization result is a single uncontrolled comparison against a shared external baseline. It should not be read as evidence that one proof assistant makes agents several times more effective.

Issues (8)

  • High severity

    The headline 11.7% vs 2.3% gap does not isolate what the optimization campaigns achieved. - Both numbers are measured relative to original CompCert 3.17, not to each campaign's starting compiler. - The paper states they 'include differences already present in the starting compilers and do not measure gains relative to each campaign's own baseline'. - The Lean starting compiler already differs from CompCert 3.17: it uses a Lean-native register allocator. RTL identity is only shown up to the nine RTL stages, which precede allocation. The abstract, the 'pays off immediately' sentence and contribution bullet 3 ('agents optimize the Lean compiler more effectively') present the gap without this caveat.

  • High severity

    There is one optimization campaign per environment, with no repeats or variance. The paper concedes this 'cannot separate the effects of model capability, proof automation, repository structure, and search choices.' Further weaknesses: - Only active hours are matched; token usage is described as 'broadly comparable'. - Each environment reports a 'selected' candidate, and the selection rule is not given in the provided text. - SPEC uses selected test/train inputs, which the paper says are not reportable scores. - The Lean repository was itself built by the same model family (GPT-6-astra from September 17), which may favor that environment. The Lean-vs-Rocq claim is anecdotal.

  • High severity

    Reproducibility is limited. - The provided text prints no repository or data URL; only a commit hash (801facb) and a tag (rc-0.0.1) are named. - Migration and optimization depend on proprietary models (GPT-5.6 Sol, GPT-6-astra) in Codex at the 'xhigh' setting. - The model changed mid-migration on September 17. - No dollar or compute cost is given. An outside group could not rerun the workflow and expect comparable results.

  • Medium severity

    Migration fidelity is not mechanized, yet the guarantee depends on it. - The source C and target Asm semantics were translated by agents and sit in the trusted base. - The theorem matches CompCert's only 'provided that the source and target semantics match CompCert's'. That match rests on agent audits and differential testing. - RTL differential testing exercises the executable compiler, not the semantic relations the theorem quantifies over. The defect history argues for caution. The first audit found 1,091 statements erased to True. Two floating-point defects passed all 11 audit rounds and surfaced only in proof finalization.

  • Medium severity

    The 'only three technical-direction messages' figure depends on an author-defined taxonomy. - It covers all 1,071 messages across 211 sessions, but places technical judgment into other buckets. Approving an agent's proposal and asking why a proof fails count as 'review/discussion'. Naming a theorem to prove counts as workflow. - Choices such as stopping automation with 471 obligations open, managing context and allowing longer timeouts are described as expert actions, not technical direction. - The classification procedure is in Appendix G, which is not in the provided text. Inter-rater agreement is not shown. - The expert is a co-author whose domain knowledge shapes which proposals are approved.

  • Medium severity

    Cost is underreported, and the 'six weeks' framing understates it. - Recorded usage is 83,528.3M tokens, but 13 coordinator and 2,989 external-worker sessions lack usage counters. - The figures are 'locally recorded usage with partial coverage of provider billing'. - The comparison to 'several dozen person-years' of original CompCert development conflates porting an existing, proved design with creating one.

  • Low severity

    The port adds unverified trust that CompCert does not need: a serialization bridge to the reused OCaml frontend and printer, validated only by testing. It also replaces OCaml extraction with the Lean compiler, a C compiler and the Lean runtime. The Lean core is 391,963 lines against 149,038 Rocq lines. The Lean count includes comments and blanks, and the Rocq counting basis is not stated. Possible proof bloat is not discussed.

  • Low severity

    There are no ablations of the workflow components: audit-before-prove vs prove-first, number of review rounds, or batch size. The batch size of four is described as 'not an experimentally established optimum'. The paper also concedes that review could likely have stopped after round 3. The workflow's causal contribution is asserted, not measured.

Questions for the authors

  • What runtime reduction does each selected candidate achieve relative to its own campaign's starting compiler?
  • How does the unoptimized CompCert4Lean compiler's generated-code runtime compare with original CompCert 3.17, given the different register allocator?
  • How was the 'selected' candidate chosen in each campaign, and were repeated campaigns run to measure variance?
  • What were the exact token and dollar costs of each 24-hour optimization campaign?
  • Beyond agent audits and differential testing of the executable compiler, what checks establish that the Lean CompCert C and Asm semantic relations match Rocq's?
  • Who classified the 1,071 messages, and how many would count as technical under a definition that includes approvals, target selection and stopping decisions?
  • Is the repository, including the orchestration harness (leasing, ownership checks, partial acceptance), publicly released, and at what URL?

Related in the directory

Open-source projects

All migration and optimization agents ran inside Codex; the paper is a large-scale case study of a Codex-based multi-agent harness.

Code2Video uses agent-authored 3D proxies to guide video generation, enabling explicit control over structure and dynamics. However, how to construct proxies and prompts for optimal downstream quality remains underexplored, as do the capabilities and limitations of this paradigm. Through controlled experiments and blinded human evaluation, we study proxy granularity, physics simulation, and prompting. The results reveal task-dependent trade-offs: coarse proxies favor natural motion, while medium-grained proxies favor consistency. Physics simulation and proxy-aligned prompting further improve average quality. Guided by these findings, we introduce Coda-DaVinci, a task-adaptive agentic framework for Code2Video generation that selects between proxy-guided and direct generation, adapts modeling granularity to each entity, and coordinates simulation with proxy-grounded prompting. Its multi-agent harness preserves scene assets and guides repairs through visual inspection and independent review. Experiments demonstrate leading instruction following on VTR and scene consistency on R2M, improved physical plausibility over direct generation, and higher visual realism than the compared Code2Video baselines. Long-video evaluations further show gains in event structure and consistency across successive interactions, while revealing limitations in transition stability and adherence to scene settings.

Yiyan Ji, Qixun Wang +11

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

Mengxiao Wang, Mahbub Alam +2

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).

Tanmay Sah

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.