Open sourcePython · ≈3.3k stars · MIT
Inspect
The evaluation framework used by a national safety institute.
UKGovernmentBEIS/inspect_ai
Maintained by UK AI Safety Institute
Inspect is the UK AI Safety Institute's open-source framework for large-scale model evaluations — tasks, solvers, scorers and tool sandboxes. The serious-science end of agent evals.
What the community says
User reports from Hacker News, Reddit, GitHub and reviews, as of Oct 29, 2026. Not our verdict.
Pros people report
- Used for rigorous, large-scale public evals (e.g. Epoch AI) with a shareable interactive log viewerHacker News
- Composable solvers and scorers, async high concurrency, retries/resume, rich logs, VS Code extension, and an Agent Bridge to evaluate existing agentsHamel Husain
- Large community eval suite (inspect_evals, 70+ evals) and epoch resampling for variancearXiv
- Maintainers fix reported issues quicklyGitHub issue UKGovernmentBEIS/inspect_ai#1507
Cons people report
- Built for evals, not as a general agent/LLM framework (per its creator); the Agent Bridge needs a message-in/message-out APIHamel Husain
- Fast-moving internals can cause regressions (e.g. a change made the bash tool unusable for Anthropic models until patched)GitHub issue UKGovernmentBEIS/inspect_ai#1507
- Community evals often deviate from reference results; deep validation is still manualarXiv
- Some features came late (cost reporting, annotation UI were roadmap items)Hamel HusainGitHub issue UKGovernmentBEIS/inspect_ai#4739
- Best for
- Research-grade, reproducible agentic and safety evals with sandboxed tools.
- Watch out for
- Python-first and eval-centric; pin versions, since rapid releases can change tool behavior. Little community feedback beyond research users.
Adopting an open-source harness? Read the best-practices guide.
Activity
- Last push
- · yesterday
- Latest release
- 0.3.291
· 4d ago - Last 90 days
- 14 releases
0.3.291v0.3.291
· 4d ago0.3.291 (2026-11-17) Features count samples of a finished log as failures with an empty-attempt run and inspect log zero (#6579) (7e056a2) eval-set selection workers can run their tasks with empty_attempt() for zero_samples() (#6605) (2dada81) exclude samples of a finished log…
0.3.290v0.3.290
· 7d ago0.3.290 (2026-11-13) Features active_model_event_id() returns the in-flight model call attempt's event id (#6529) (c060c20) each message keeps one id through the agent bridge and the log, and tool events record the message that proposed them (#6143) (187d19f) excluded epochs…
0.3.289v0.3.289
· 11d ago0.3.289 (2026-11-10) Features anthropic: support Claude Fable 5.2 (#6443) (cd2d730) sentinel: act on a sentinel sidecar's decisions when model calls go through a proxy (#6303) (a0ce5ab) Bug Fixes ACP code no longer imports acp.helpers, which agent-client-protocol 1.0 removed…
0.3.288v0.3.288
· 2w ago0.3.288 (2026-11-06) Breaking Changes environments with google-genai 2.x fail the validate_google_client version check. Install google-genai=3.0.0. Evaluations using the automatically generated Docker sandbox must provide an explicit image if they need the nonroot account,…
0.3.287v0.3.287
· 3w ago0.3.287 (2026-10-31) Features a runtime can hold and rewrite a react() conversation at turn boundaries, and compaction handlers gain reset() (#6201) (7340817) reserve the range_goal registry type for inspect_ranges goal types (#6224) (3dcbf00) sentinel: the sentinel host gives…
Checked against GitHub on Nov 21, 2026. Release notes are shortened to plain text.
In research (2)
- Work Agents: Systems, Capabilities, and OptimizationVerdict: Incremental
Oct 26, 2026 · Standardised evaluation framework named in §5.4.
- OpenDraco: A Platform for Engineering and Experimenting with Configurable Multi-Agent Architectures for Software EngineeringVerdict: Incremental
Oct 23, 2026 · Inspect is an existing evaluation framework that does a similar job, and the paper does not compare against it.
Neighbours on the workbench
JavaScript · ≈308k stars · MIT
affaan-m/ECC
Agent harness performance optimization system: skills, instincts, memory and security for coding agents.
Maintained by Affaan Mustafa
JavaScript · ≈76k stars · MIT
cloudflare/security-audit-skill
Cloudflare's skill for multi-phase security audits with independently verified findings.
Maintained by cloudflare
Go · ≈55k stars · Apache-2.0
alibaba/open-code-review
Alibaba's hybrid deterministic-plus-agent code review CLI.
Maintained by Alibaba
v1.13.13releasedConnections
Relations from the Harness Hub knowledge graph, as charted in the State of Harness reports.
Maintained by · 1
- UK AI Safety InstituteOrganisation
Similar entries · 1
- GiskardStartup
