Open sourceTypeScript · ≈28k stars · MIT
promptfoo
Test your prompts, agents and RAG — plus red-teaming.
promptfoo is an open-source testing framework for LLM apps: eval matrices, assertions, RAG metrics and an AI red-teaming scanner for vulnerabilities. Config-in-YAML, CI-friendly.
What the community says
User reports from Hacker News, Reddit, GitHub and reviews, as of Oct 29, 2026. Not our verdict.
Pros people report
- Users find it very useful for managing large, interconnected prompt collections and workflow-specific testsHacker News 1Hacker News 2
- Long-time users praise it and call it an industry standard; open source (MIT), continuing after the OpenAI acquisitionHacker News 1Hacker News 2Hacker News 3
- Flexible: custom asserts and any OpenAI-compatible or local endpoint (e.g. llama-server)Hacker News 1Hacker News 2
- Good fit for building your own benchmarks for your workflow, as opposed to generic leaderboardsHacker News
Cons people report
- Acquired by OpenAI (Mar 2026), which raises neutrality and roadmap questions for multi-provider usersHacker News 1Hacker News 2
- Doesn't cover MCP transport, tool-schema validation or MCP-specific attacksHacker News
- Large install footprint when used as a library dependencyGitHub issue promptfoo/promptfoo#7254
- Some users run it alongside ad-hoc scripts and still aren't happy with their eval setupHacker News
- Best for
- Config-driven prompt/model regression tests and red-teaming in CI.
- Watch out for
- Long-term vendor neutrality after the OpenAI acquisition, and dependency weight.
Adopting an open-source harness? Read the best-practices guide.
Activity
- Last push
- · today
- Latest release
- code-scan-action-0.2.3
· 3d ago - Last 90 days
- 13 releases
code-scan-action-0.2.3code-scan-action: 0.2.3
· 3d ago0.2.3 (2026-11-18) Changes Full-repository scans use the patched filesystem MCP server 2.3.0 and can share eligible committed text files from the compared revision with the hosted scanner. Working-tree changes and untracked files stay outside that snapshot. Set diffsOnly: true…
- · 3d ago
0.126.0 (2026-11-18) Before upgrading Derived metrics. Check string formulas in derivedMetrics. Arithmetic, comparisons, ternaries and score names such as count still work. Function calls, property access, assignments and formulas longer than 1,024 characters now stop the…
- · 3w ago
0.125.1 (2026-10-25) Bug Fixes restore the authentic Discord invite (#11967) (98000c4)
- · 4w ago
0.125.0 (2026-10-23) ⚠ BREAKING CHANGES deps: make jks-js an optional peer dependency (#11739) providers: make Claude Agent and Codex SDKs opt-in (#11134) Features add provider-only environment overrides (#11889) (2eed6e1) assertions: add rouge-l and rouge-s assertion types…
code-scan-action-0.2.2code-scan-action: 0.2.2
· 6w ago0.2.2 (2026-10-08) Bug Fixes code-scan: mint OIDC after CLI installation (#11467) (9a404c8)
Checked against GitHub on Nov 21, 2026. Release notes are shortened to plain text.
In research (1)
Oct 22, 2026 · Red-teaming and eval tool that could host this kind of adaptive social-engineering test for email agents.
Neighbours on the workbench
JavaScript · ≈308k stars · MIT
affaan-m/ECC
Agent harness performance optimization system: skills, instincts, memory and security for coding agents.
Maintained by Affaan Mustafa
JavaScript · ≈76k stars · MIT
cloudflare/security-audit-skill
Cloudflare's skill for multi-phase security audits with independently verified findings.
Maintained by cloudflare
Go · ≈55k stars · Apache-2.0
alibaba/open-code-review
Alibaba's hybrid deterministic-plus-agent code review CLI.
Maintained by Alibaba
v1.13.13releasedConnections
Relations from the Harness Hub knowledge graph, as charted in the State of Harness reports.
Maintained by · 1
- promptfooOrganisation
Similar entries · 2
- GiskardStartup
- Patronus AIStartup
