Open sourcePython · ≈6.2k stars · MIT
SWE-bench
Can language models resolve real-world GitHub issues?
SWE-bench is the canonical evaluation harness for coding agents: real GitHub issues, a Docker-based execution harness, and instance-level scoring. The benchmark the whole coding-agent leaderboard conversation is anchored to.
Activity
- Last push
- · 2mo ago
- Latest release
- v5.0.1
· 3mo ago - Last 90 days
- 0 releases
No GitHub releases: showing the newest tags.
Checked against GitHub on Nov 21, 2026. Release notes are shortened to plain text.
In the news
Community · Arena.ai · Sep 16, 2026
Arena.ai's HarnessTax study tests 21 model–harness combinations across Claude Code, Codex CLI and Pi, finding harness choice can significantly change cost even at similar task success rates. Weeks earlier, SWE-bench Pro analysis showed swapping harnesses moved pass@1 more than model upgrades (23%→52% on GLM-5.2) — and that harness rankings barely transfer across models.
Funding · OpenAI Blog · Jul 8, 2026
A new analysis from OpenAI reveals issues in SWE-Bench Pro, a popular coding benchmark, raising concerns about reliability and accuracy in evaluating AI models.
Community · OpenAI Blog · Feb 23, 2026
SWE-bench Verified is increasingly contaminated and mismeasures frontier coding progress. Our analysis shows flawed tests and training leakage. We recommend SWE-bench Pro.
In research (41)
- Higher-Scoring Coding Models Did Not Get More Work DoneVerdict: Useful
Nov 17, 2026 · SWE-Bench Pro and SWE-bench Verified score gaps are the benchmark claims the paper tests against real merges.
- Mind the Gap: A Deployed Multi-Agent System for Documentation MaintenanceVerdict: Incremental
Nov 17, 2026 · Cited as the oracle-backed agent repair setting the paper contrasts with documentation, which lacks test oracles.
- Beyond Leaderboard Ranks: Selecting Software Engineering Agents Under Budget ConstraintsVerdict: Useful
Nov 16, 2026 · Primary benchmark; the analysis uses 48 shared tasks of HAL's SWE-bench Verified Mini subset.
- A Taxonomy and Taxonomy-Guided Benchmark Reduction for Spreadsheet TasksVerdict: Incremental
Nov 16, 2026 · Cited as an agent benchmark that offers no subset-selection or coverage framework, the gap the paper targets.
- Agent-Native by Design: A Programming Language for Agentic ApplicationsVerdict: Incremental
Nov 16, 2026 · The CodeAgent benchmark is evaluated on 360 pinned SWE-bench instances.
- From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution LoopVerdict: Useful
Nov 16, 2026 · SWE-bench Verified, Pro and Multilingual are three of the five evaluation benchmarks.
Showing the 6 newest of 41 papers.
Neighbours on the workbench
Go · ≈55k stars · Apache-2.0
alibaba/open-code-review
Alibaba's hybrid deterministic-plus-agent code review CLI.
Maintained by Alibaba
v1.13.13releasedPython · ≈39k stars · MIT
stanfordnlp/dspy
Programming, not prompting: optimize the whole pipeline.
Maintained by Stanford NLP
3.5.0releasedTypeScript · ≈38k stars · Other
langfuse/langfuse
Open-source LLM engineering platform: traces, evals, prompts.
Maintained by Langfuse
v4.83.0releasedConnections
Relations from the Harness Hub knowledge graph, as charted in the State of Harness reports.
Maintained by · 1
- SWE-benchOrganisation
Mentioned in · 28
- Beyond Weight Updates: Evolving Agent Harnesses for Domain SpecializationPaper · 2026-11-19
- Clover-35B-A3B-Preview Technical ReportPaper · 2026-11-18
- Reducing Inference Cost of Coding Agents with Timeout CascadesPaper · 2026-11-18
- Context Has a Price: Adding and Removing Context for LLM Coding Agents under Prefix CachingPaper · 2026-11-17
- Done, But Still Working: Anytime Evaluation of When Coding Agents Should StopPaper · 2026-11-17
- Harnesses That Improve Themselves: A Unifying Framework for Self-Improving Agent HarnessesPaper · 2026-11-17
and 22 more
