promptfoo
promptfoo/promptfoo
Test your prompts, agents and RAG — plus red-teaming.
promptfoo is an open-source testing framework for LLM apps: eval matrices, assertions, RAG metrics and an AI red-teaming scanner for vulnerabilities. Config-in-YAML, CI-friendly.
In the news
HarnessTax: harness choice swings agent cost more than model choice
Arena.ai's HarnessTax study tests 21 model–harness combinations across Claude Code, Codex CLI and Pi, finding harness choice can significantly change cost even at similar task success rates. Weeks earlier, SWE-bench Pro analysis showed swapping harnesses moved pass@1 more than model upgrades (23%→52% on GLM-5.2) — and that harness rankings barely transfer across models.
ClickHouse acquires Langfuse
ClickHouse acquires the open-source LLM observability platform Langfuse as part of its $400M Series D at a $15B valuation — the first big consolidation in agent observability. The open-source core remains maintained; the deal signals that agent tracing is becoming a database-scale problem.
Related projects
DSPy
stanfordnlp/dspy
Programming, not prompting: optimize the whole pipeline.
DSPy introduced programming models for LLM pipelines — declarative modules whose prompts and weights are optimized against metrics. Its optimizer-first philosophy reshaped how production harnesses treat prompts.
Langfuse
langfuse/langfuse
Open-source LLM engineering platform: traces, evals, prompts.
Langfuse is the open-source observability layer for LLM apps — tracing, prompt management, datasets, evals and cost tracking — deployable self-hosted or as a cloud.
Mastra
mastra-ai/mastra
The TypeScript framework for production agents.
Mastra is a full TS-native agent framework — workflows, tool calling, RAG, evals and a dev playground — that has become the default choice for JavaScript teams building agent backends.