Harness Hub
All open source

SWE-bench

Can language models resolve real-world GitHub issues?

SWE-bench/SWE-bench

Maintained by SWE-bench

SWE-bench is the canonical evaluation harness for coding agents: real GitHub issues, a Docker-based execution harness, and instance-level scoring. The benchmark the whole coding-agent leaderboard conversation is anchored to.

Activity

Last push
· 2mo ago
Latest release
v5.0.1
· 3mo ago
Last 90 days
0 releases
  1. v5.0.1

    · 3mo ago
  2. v5.0.0

    · 3mo ago
  3. v4.1.0

    · 1y ago

No GitHub releases: showing the newest tags.

Checked against GitHub on Nov 21, 2026. Release notes are shortened to plain text.

In the news

Arena.ai's HarnessTax study tests 21 model–harness combinations across Claude Code, Codex CLI and Pi, finding harness choice can significantly change cost even at similar task success rates. Weeks earlier, SWE-bench Pro analysis showed swapping harnesses moved pass@1 more than model upgrades (23%→52% on GLM-5.2) — and that harness rankings barely transfer across models.

SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress. Our analysis shows flawed tests and training leakage. We recommend SWE-bench Pro.

In research (41)

Showing the 6 newest of 41 papers.

alibaba/open-code-review

Alibaba's hybrid deterministic-plus-agent code review CLI.

Maintained by Alibaba

v1.13.13released
4 pros · 4 cons

stanfordnlp/dspy

Programming, not prompting: optimize the whole pipeline.

Maintained by Stanford NLP

3.5.0released
3 pros · 4 cons
Featured

langfuse/langfuse

Open-source LLM engineering platform: traces, evals, prompts.

Maintained by Langfuse

v4.83.0released
4 pros · 5 cons

Connections

Relations from the Harness Hub knowledge graph, as charted in the State of Harness reports.

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.