The index · everything in the almanac
Search
Look up a team, a repository, a gathering or a paper, by keyword or by meaning.
No matches for “”
The index · everything in the almanac
Look up a team, a repository, a gathering or a paper, by keyword or by meaning.
No matches for “”
No matches for “”
Layer
Startupkeyword match
The de-facto standard toolkit for building LLM applications and agents. · San Francisco, CA
LangChain makes the LangChain framework and LangGraph, the low-level orchestration standard for stateful, controllable agents, plus the LangSmith platform for tracing, evaluating and monitoring them in production.
Open sourcekeyword match
Mastra/mastra · TypeScript
Mastra is a full TS-native agent framework — workflows, tool calling, RAG, evals and a dev playground — that has become the default choice for JavaScript teams building agent backends.
Open sourcekeyword match
Langfuse/langfuse · TypeScript
Langfuse is the open-source observability layer for LLM apps — tracing, prompt management, datasets, evals and cost tracking — deployable self-hosted or as a cloud.
Open sourcekeyword match
promptfoo/promptfoo · TypeScript
promptfoo is an open-source testing framework for LLM apps: eval matrices, assertions, RAG metrics and an AI red-teaming scanner for vulnerabilities. Config-in-YAML, CI-friendly.
Open sourcekeyword match
SWE-bench/SWE-bench · Python
SWE-bench is the canonical evaluation harness for coding agents: real GitHub issues, a Docker-based execution harness, and instance-level scoring. The benchmark the whole coding-agent leaderboard conversation is anchored to.
Startupkeyword match
Enterprise-grade evals, data and AI gateway. · San Francisco, CA
Braintrust turns prompts and agent traces into testable datasets and CI, plus an inference gateway — helping teams measure whether harness changes actually improve agent quality.
Startupkeyword match
Observability and evaluation for AI — makers of OSS Phoenix. · Berkeley, CA
Arize traces and evaluates LLM and agent systems in production; their open-source Phoenix tracer is a common instrumentation choice for agent harnesses that need to be debuggable.
Startupkeyword match
Evaluation platform for LLMs and agentic systems. · San Francisco, CA
Galileo (makers of the Luna evaluation models) scores agent runs for hallucination, tool-use errors and task completion — automated quality gates for harness engineering. Cisco acquired Galileo in April 2026 and folded it into Splunk's agent observability; the old rungalileo.io domain has lapsed.
Startupkeyword match
Open-source LLM engineering platform: tracing, prompts, evals. · Berlin, Germany
Langfuse is an open-core platform for observing and improving LLM apps — trace-level debugging, prompt management, datasets and evals. Self-hostable, with a cloud offering. Acquired by ClickHouse in January 2026 as part of its push into AI observability; the open-source core remains actively maintained.
Startupkeyword match
Agent observability, testing and session replay. · San Francisco, CA
AgentOps instruments agents to record every step, cost and tool call — then lets teams replay sessions and build regression tests from real failures.
Startupkeyword match
Red-teaming and robustness for AI agents. · New York, NY
Haize Labs stress-tests agents with automated adversarial attacks — prompt injection, tool misuse, unsafe actions — and ships an SDK for continuous safety evaluation in CI.
Startupkeyword match
Automated evaluation and guardrails for LLM systems. · San Francisco, CA
Patronus AI scores outputs against policy and business criteria in CI and production, catching regressions before users do — 'unit tests for AI'.
Startupkeyword match
The TypeScript-first company behind the Mastra agent framework. · United States
Mastra builds the eponymous open-source TypeScript framework — workflows, tools, RAG, evals and a local dev playground — and is becoming the default 'agent backend' choice for JS/TS teams.
Startupkeyword match
Type-safe Python infrastructure — now including agents.
Pydantic builds the data-validation library underlying most of the Python AI ecosystem, plus Logfire (observability) and Pydantic AI (typed agent framework) — the type-safety layer of the harness.
Startupkeyword match
Open-source TypeScript agent engineering platform. · United States
VoltAgent ships a batteries-included open-source TS agent framework — agents, workflows, observability UI and evals — targeted at product teams that want the JS ecosystem's answer to LangGraph.
Startupkeyword match
Open-source LLM observability and AI gateway. · San Francisco, CA
Helicone is an open-source LLM observability platform — logging, caching, rate limits and evals behind a one-line proxy — with an AI gateway for routing and cost control. A YC company that became the self-hostable counterweight to LangSmith.
Startupkeyword match
Self-improving coding agents that recover and verify.
Relay is a coding-agent harness where every run teaches the next: a queue with one worktree per task, models you already pay for (Claude, ChatGPT, Copilot or API keys), tests re-run by Relay, and lessons kept for the next lap. Free, local-first, BYOM.
Startupkeyword match
An engineering 'superharness' that runs coding agents as one governed team.
Palgu describes itself as an engineering superharness that takes a task from intent to a proven, governed code change. It runs coding agents such as Claude Code, Codex, Cursor and aider as one team and wraps them in eight layers: intent, architecture, knowledge, coordination, evidence, recovery, governance and learning. Its products are the palgu CLI and the Palgu Studio desktop app for macOS and Windows, plus open-source Maggy (a single-developer harness), the Mixture of Models mod for Claude Code and other agents, and AWG, a shared work graph for coding agents. Features include a code graph with blast-radius mapping, a five-seat review council of models other than the author, an audit ledger, BYOK and a self-hosted gateway.
Startupkeyword match
Open drivers, cross-OS sandboxes and benchmarks for computer-use agents. · San Francisco, CA
Cua (YC Spring 2025) builds infrastructure for computer-use agents. Cua Spaces is a desktop app that gives agents full desktops: macOS VMs built locally, or Linux images, on the user's own machines or cloud. Cua Driver lets agents inspect and operate native apps and browsers on macOS, Windows and Linux through a CLI, MCP or typed SDKs. The stack also includes Lume (macOS VMs), the Cua SDK and CLI for local sandboxes, CUA-S1 small models for computer-use decisions, and Cua Bench for simulated tasks. The repo is MIT-licensed, while the Spaces app is source-available under FSL-1.1-MIT.
Startupkeyword match
ART: open-source reinforcement learning trainer for multi-step agents.
OpenPipe maintains ART (Agent Reinforcement Trainer), an open-source (Apache-2.0) RL framework that lets LLM agents learn from experience by integrating GRPO into any Python application. Project news covers a LangGraph integration, MCP-RL for training models to use MCP server tools, AutoRL for zero-data training and RULER for LLM-judged rewards. The README also promotes W&B Training (Serverless RL), a managed service for running ART training and inference. Supported open models include Qwen, GPT-OSS and Llama.
Startupkeyword match
Plano: AI-native proxy and data plane for agentic apps.
Katanemo develops Plano, an open-source (Apache-2.0) AI-native proxy server and data plane for agentic applications, built on Envoy and written in Rust. Plano moves delivery concerns out of application code into an out-of-process data plane: low-latency routing and orchestration between agents, LLM routing by model name, alias or preferences, zero-code capture of 'Agentic Signals' plus OpenTelemetry traces and metrics, and filter chains for jailbreak protection, moderation and memory hooks. It works with any language or agent framework.
Startupkeyword match
Open-source evals, red teaming and test generation for agentic systems.
Giskard builds an open-source (Apache-2.0) Python library for testing and evaluating agentic systems, alongside a commercial AI red-teaming and LLM security platform. Giskard v3 is a modular rewrite for dynamic, multi-turn agent testing: Giskard Checks for evals from simple assertions to LLM-as-judge, and giskard-scan for vulnerability scanning and RAG evaluation, built on giskard-core, giskard-llm and giskard-agents. It can wrap an LLM, a black-box agent or a multi-step pipeline. v2 remains available but is no longer actively maintained.
Startupkeyword match
Open-source agent observability that dispatches coding agents to fix failures.
Latitude (Latitude Data S.L.) builds an open-source, MIT-licensed platform for monitoring AI agents in production. One line of telemetry captures multi-turn sessions, tool calls and full execution paths, with OTEL compatibility. Failing traces are auto-grouped into tracked signals and issues with alerts. Latitude then dispatches a coding agent such as Claude Code or Cursor with sample traces to write a fix and open a PR, and replays fixes against the real failing traces as regression tests. Everything is also exposed through an MCP server and CLI. Hosted data stays on European servers.
Startupkeyword match
Open-source observability platform purpose-built for AI agents. · London, UK
Laminar (YC S24) is an open-source observability platform for AI agents. Its OpenTelemetry-native SDK auto-traces Vercel AI SDK, Browser Use, Stagehand, LangChain, OpenAI, Anthropic and Gemini. Signals let teams describe behaviors in plain English, such as an agent stuck in a loop, and get Slack alerts when they occur. It also offers an evals SDK and CLI for local or CI runs, SQL-queryable traces via MCP and CLI so coding agents can debug from traces, custom dashboards, and dataset annotation. The backend is written in Rust and can be self-hosted with Docker Compose.
Startupkeyword match
Open-source platform for evaluating, tracing and improving AI agents.
Future AGI builds an open-source, self-hostable platform for shipping AI agents that covers the lifecycle in one feedback loop: simulations of edge cases before launch, evaluations, tracing and observability in production, real-time guardrails, an LLM gateway, datasets and optimization. It positions itself as a replacement for stitching together separate tools for evals, observability, guardrails and simulation. The platform is Apache 2.0 licensed, with a free cloud tier alongside self-hosting.
Startupkeyword match
Multi-agent code review and quality governance for AI-written code.
Qodo builds a code review and governance layer for agent-written software. Its platform runs multiple specialized review agents on pull requests, with a coordinating layer that filters and prioritizes findings, plus cross-repo review, a software map of service dependencies, a codebase knowledge base and an 'Agentic Toolbox' CLI that gives coding agents such as Codex, Claude and Kiro a quality counterpart inside their loop. In March 2026 it raised a $70M Series B led by Qumra Capital, bringing total funding to $120M.
Startupkeyword match
AI code review and change governance for agentic development.
CodeRabbit runs automated AI review on every pull request, posting line-level findings and committable suggestions, and has expanded into what it calls agentic change management: PR triage that prioritizes the review queue against each reviewer's capacity, change explanations and continuous security monitoring of dependencies and code health. It positions itself as an independent review and control layer as AI-generated code outpaces human review capacity. In September 2025 it raised a $60M Series B led by Scale Venture Partners at a $550M valuation, with NVIDIA's NVentures participating.
Startupkeyword match
AI code review agents with full-codebase context. · San Francisco, CA
Greptile reviews and tests pull requests with context from the whole repository. It builds a graph index of files, functions and dependencies, then runs a swarm of parallel agents that assess a change's impact beyond the diff, and learns team standards from reviewers' comments over time. The product line also includes a CLI, security review and TREX runtime validation (beta), framed as an independent check on code from both humans and coding agents. Founded in 2023 and part of YC W24, the San Francisco company raised a $50M Series B led by Menlo Ventures at a $750M valuation in October 2026.
Startupkeyword match
Agent enablement platform and registry for building, evaluating and distributing agent skills. · London, UK
Tessl, founded by Guy Podjarny, describes itself as the management layer between developers and their coding agents (Claude Code, Cursor, GitHub Copilot, Gemini and others). The Tessl Platform acts as a control plane for agent context, tackling duplicate skills, tracking skill quality and usage with before-and-after evals, and scanning skills for security. The Tessl Registry indexes skills across an organisation, and Loops run skills on a schedule for tasks like code review and docs maintenance. In September 2025 it launched the Tessl Framework and Spec Registry for spec-driven development.
Startupkeyword match
A self-improving software factory with evaluator and healer meta-agents.
Autoheal runs always-on cloud agents for SDLC work such as incident response, vulnerability remediation, release readiness and AI coding cost control. Supervisory Evaluator agents score worker output using signals like review comments, CI failures and linked incidents, while Healer agents propose changes to prompts, skills, tools or model choice, test them against historical benchmarks and submit them for engineer approval in Git. Each run gets a fresh sandbox in the customer's cloud with policy checks on every tool call, per-agent budgets and audit trails. It raised a $7.9M seed led by Innovation Endeavors (Sept 2026).
Startupkeyword match
Model routing, code review and cost insights for engineering agents.
Entelligence builds an engineering 'world model' with products on top: a Model Router that sends each turn to the cheapest model meeting a set quality floor and escalates to frontier models only when needed; Code Review that indexes the codebase and past incidents and runs parallel security, performance and precedent passes on PRs; and Agent Insights that records agent sessions and reports cost per session, retry rate and token waste. A CLI sits in front of Claude Code, Codex and Cursor, with MCP plugins. Founded by Aiswarya Sankar; raised $5M co-led by Mayfield and Correlation Ventures.
Startupkeyword match
Devbox sandboxes and agent infrastructure for AI coding agents.
Runloop Devboxes are VM-isolated sandboxes where AI agents build, run and test code, starting in seconds and supporting snapshot, suspend and resume for long-running work. Blueprints define team-shared images; network policies restrict egress, while Agent Gateways and an MCP Hub connect agents to LLM APIs and MCP servers without exposing real credentials. The platform also lists measurement for evaluating agent performance, multi-agent coordination, Reflex debugging/observability and VPC deployment on AWS, GCP or Azure, with Python and TypeScript SDKs and a CLI.
Startupkeyword match
AI code review agent for pull requests · Montreal
Korbit AI is an AI code review tool for GitHub, GitLab and Bitbucket that gives real-time, actionable feedback in every pull request. It also generates PR descriptions, review summaries and team reports. Boost Security announced in 2026 that it had acquired Korbit.ai as part of its SDLC defense platform.
Startupkeyword match
AI testing agent for enterprise unit testing
Diffblue describes itself as an AI testing agent for enterprise unit testing. It automatically generates unit tests for large organizations' codebases.
Startupkeyword match
AI QA engineer that tests your app end to end
Momentic's agent Mo tests apps end to end from plain-English flows stored as YAML in the repo. It runs locally and in CI as a merge check. Coding agents can author and verify tests through MCP. It triages failures as app bugs, flakiness or environment issues, and proposes repairs to tests when the UI changes.
Startupkeyword match
Local validation for your agent's code
Chunk is a CircleCI tool that validates code written by Claude Code, Cursor or other agents before it is pushed to CI. It syncs local code to a remote Linux microVM sidecar and runs tests there as 'microbuilds' inside the agent loop, using hooks to give deterministic feedback. Sidecars can run integration suites with services like Postgres and Redis. It is free on every CircleCI plan.
Startupkeyword match
Sentry's AI debugging agent
Seer uses Sentry telemetry to automatically investigate new issues and find root causes. Autofix traces the cause through the codebase and drafts a fix. AI Code Review checks pull requests against production errors and traces. Fixes can be handed to Claude, Copilot or Cursor, and investigations can be run from Slack. It costs $40 per active contributor per month on paid plans.
Startupkeyword match
AWS platform for building, connecting and running production AI agents with any framework or model
Amazon Bedrock AgentCore is AWS's platform for building, connecting and running production AI agents with any framework or model. It includes a harness for deploying agents and a Gateway that handles authentication and access control for tool calls to MCP servers, APIs and Lambda functions. It also has observability and evaluation tools for tracing agent behavior and testing variations against real traffic.
Startupkeyword match
Observability and continuous improvement for production agents
W&B Weave is Weights & Biases' observability and evaluation product for production AI agents. Its traces are organized around sessions, turns, steps, tools and sub-agents. It adds behavior monitoring with alerts, an evaluation API, guardrail scorers and leaderboards. Coding agents such as Claude Code can connect through W&B skills and an MCP server to read production data and run evaluations.
Startupkeyword match
Detect agent issues. Prevent them from happening.
Raindrop monitors AI agents in production, finds silent failures and tests fixes before they ship. It traces every message, tool call and decision, groups recurring failures into issues, and runs a Triage Agent that investigates them in Slack, on the web or through MCP. It also runs simulations of agent changes in CI and offers Workshop, an open-source local tool for replaying and debugging agents.
Startupkeyword match
Production AI starts with observability
HoneyHive is an agent observability and evaluation platform for enterprises. It traces prompts, tool calls, outputs and handoffs, monitors live traffic, and alerts on scores. It runs offline experiments, builds regression suites from production traces and runs evals in CI. It is OpenTelemetry-native and can be deployed as SaaS, hybrid or fully self-hosted.
Startupkeyword match
Scale voice AI agents with confidence
Coval is a testing and evaluation platform for voice AI agents. It simulates thousands of realistic caller conversations before launch and evaluates live conversations for quality drift, safety issues and drops in resolution. It also runs regression tests across prompt, model and vendor changes, and has a CLI for running simulations.
Startupkeyword match
The simulation platform for agent self-improvement
Scorecard is a platform for testing, evaluating and improving AI agents. It runs agents through large sets of realistic scenarios before release, and provides a metric library, prompt versioning and a playground. It also supports deploying agents and detecting real-world usage problems.
Startupkeyword match
Find and fix what your agents get wrong
Judgment Labs makes a monitoring and improvement platform for long-horizon AI agents. It traces agent runs including subagents and tool calls, monitors custom signals with code checks or agentic judges, and surfaces recurring production failures. It also runs offline regression evals, and offers a Slack agent and an MCP server for use with coding agents. It can run in Judgment's cloud or in the customer's own AWS or GCP account.
Startupkeyword match
Test, Monitor and Self Improve voice agents
Cekura is an automated QA platform for voice and chat AI agents. It runs synthetic conversations before an agent goes live, monitors production calls for drift, and can block deploys when release test suites fail. It also does adversarial red-teaming for jailbreaks and data leaks, and suggests prompt fixes for issues it reproduces in simulation.
Startupkeyword match
Simulate real users. Ship reliable voice and text agents.
Okareo tests and monitors voice, text and headless AI agents using synthetic users called Drivers. Drivers have a personality, context and goal, and hold long multi-turn conversations to surface edge cases. Simulations and evaluations run on each commit or pull request, and production failures can be turned into new test scenarios. It also lists agent monitoring and an MCP integration.
Startupkeyword match
LLM reliability platform that turns evals and monitors into a continuous feedback loop
Traceloop is an LLM reliability platform for monitoring, evaluating and improving LLM applications. It is built on OpenTelemetry and ships with its open-source OpenLLMetry SDK. It also offers Hub, an OpenTelemetry-based gateway. The site announces that Traceloop is joining ServiceNow.
Startupkeyword match
AI quality platform to standardize evals and observability across the organization
Confident AI is an AI quality platform for enterprise teams to standardize evals and observability. It builds and maintains the open-source DeepEval evaluation framework and DeepTeam red-teaming framework. For agents it tests tool selection, arguments and full trajectories, and simulates multi-turn conversations. It blocks releases that fail evals or red-team checks and offers an MCP server and agent skills.
Startupkeyword match
AI governance for everyone
Arthur offers one control plane that brings together security, finance and the teams shipping AI. It measures, monitors and governs enterprise AI at scale.
Startupkeyword match
The AI Reliability Platform
Guardrails AI provides a guardrails framework for building, governing and scaling production GenAI with any LLM and deployment environment. It offers runtime guardrails that block bad outputs, a Guardrails Hub of validators, and tooling to generate eval datasets that surface agent failures. The open-source guardrails library is Apache-2.0 licensed Python.
Startupkeyword match
Production infrastructure for AI agents
Bento is a production infrastructure platform for AI agents that monitors what runs, flags regressions and turns failures into fixes. It provides OpenTelemetry-native traces, natural-language alerts, drift detection, evaluations and versioning. The site says it is backed by Y Combinator.
Startupkeyword match
Sandboxes with stateful twins of the services agents act on
Arga Labs provides sandboxes for testing and training AI agents, with stateful twins of third-party APIs, CLIs and MCPs such as Slack, GitHub and Gmail. Teams point their agents at the twins, seed them with data and review full traces of every call and side effect.
Startupkeyword match
MCP integration platform and live environments for AI agents
Klavis AI's open-source platform provides MCP integrations so agents can use tools reliably at scale. The company also builds live environments and training data for agents, including Dockerized long-horizon coding tasks and tool-use data across 600+ tools and SaaS apps. It is backed by Y Combinator.
Startupkeyword match
An AI on-call agent that drops read-only probes into production to debug live systems.
HyperProbe launched on Hacker News in August 2026 with a way for Cursor and Claude to place read-only probes in production and debug live systems. Its site now leads with an "AI on-call agent" built on the same idea. Y Combinator S26.
Startupkeyword match
Observability for AI agents: catch drift and silent regressions across sessions and tool calls.
Sentrial monitors AI agents in production for loops, hallucinations, tool misuse and silent regressions across every session, tool call and model interaction ("Catch agent drift before your users do"). It launched on Hacker News in March 2026. Y Combinator W26.
Startupkeyword match
Monitoring and continuous improvement for AI agents: detects failures and opens fix PRs.
Agnost AI launched on Hacker News in July 2026 with product analytics drawn from agent conversations, such as rage-prompting and rephrasing. Its site now reads "Monitoring and Continuous Improvement for AI Agents": it detects failures and opens pull requests that fix them. Y Combinator S26.
Startupkeyword match
Analytics for AI agents: an SDK that shows product teams how their agents are used.
Voker makes an analytics SDK for teams that ship AI agents, so they can see how agents are used and where conversations fail. It launched on Hacker News in May 2026. Y Combinator S24.
Startupkeyword match
Custom models and harnesses trained on your data; launched to debug and evaluate agents in production.
Lucidic launched on Hacker News in July 2025 to debug, test and evaluate AI agents in production (116 points). Its site now reads "Custom Models & Harnesses Trained on Your Data", a move from observing agents to training the models and harnesses behind them. Y Combinator W25.
Open sourcekeyword match
Stanford NLP/dspy · Python
DSPy introduced programming models for LLM pipelines — declarative modules whose prompts and weights are optimized against metrics. Its optimizer-first philosophy reshaped how production harnesses treat prompts.
Open sourcekeyword match
VoltAgent/voltagent · TypeScript
VoltAgent ships a TypeScript agent framework with observability, evals and a UI out of the box — aiming to be the batteries-included option for JS/TS teams.
Open sourcekeyword match
Princeton NLP / SWE-agent/SWE-agent · Python
SWE-agent turns an LLM into a software engineer that navigates repos, edits files and runs tests inside a minimal 'agent-computer interface'. The research harness that defined agentic SWE benchmarks.
Open sourcekeyword match
Arize AI/phoenix · Python
Phoenix traces LLM/agent runs, evaluates them (LLM-as-judge, retrieval metrics) and surfaces regressions — Arize's open-source core for debugging agents in production.
Open sourcekeyword match
AgentOps/agentops · Python
AgentOps records every agent session — prompts, tool calls, costs, errors — with replay and evaluation tooling for debugging agent behavior in production.
Open sourcekeyword match
Confident AI/deepeval · Python
DeepEval grades LLM outputs with 20+ metrics (hallucination, faithfulness, task completion) inside pytest — unit-testing for prompts, RAG and agents, backed by Confident AI.
Open sourcekeyword match
UK AI Safety Institute/inspect_ai · Python
Inspect is the UK AI Safety Institute's open-source framework for large-scale model evaluations — tasks, solvers, scorers and tool sandboxes. The serious-science end of agent evals.
Open sourcekeyword match
Comet/opik · Python
Opik is Comet's open-source LLM evaluation platform — tracing, prompt playgrounds, dataset-driven evals and online monitoring for RAG and agent systems. The MLflow lineage applied to the agent loop.
Open sourcekeyword match
PrimeIntellect/prime-agent · Python
Prime Agent is PrimeIntellect's open-source recursive-language-model agent: context as a variable, sub-agents as function calls inside a persistent IPython REPL, plus a continual-harness loop for self-improvement. Its harness famously took Opus 5 from ~30% to 95.5% on ARC-AGI-3.
Open sourcekeyword match
Google/agents-cli · TypeScript
Google's Agents CLI turns any coding assistant into an expert at creating, evaluating and deploying agents — layering official skills on top of the harness you already run instead of shipping another agent loop. The harness as config, from Google.
Open sourcekeyword match
Harbor/terminal-bench · Python
Terminal-Bench is the terminal-task benchmark coding agents now cite next to SWE-bench: hard, containerized tasks scored end to end. Version 2.0 runs on the Harbor evaluation framework; the 1.0 task set lives on in the org's repos.
Open sourcekeyword match
Microsoft/agent-lightning · Python
Agent Lightning is Microsoft's framework for training AI agents with reinforcement learning. Version 1.0 (open-sourced August 2026) uses three components: a Trainer running verl and vLLM, an API Gateway that proxies model requests and captures training data, and a Rollout Controller that runs agents locally or as Kubernetes Jobs. Agents keep their real harnesses, tools and control flow with no code changes. The project reports a Qwen3.5-9B coding-agent example improving SWE-bench Verified from 41.8% to 56.4% with 6K training samples.
Open sourcekeyword match
Nasiko Labs/nasiko · Rust
Nasiko calls itself the 'OpenRuntime' for agents, coding harnesses, frameworks and tools. It discovers coding agents already running (Claude Code, Codex, Cursor, OpenCode) without changing their configuration, puts their spend into one schema broken down by harness and model, and can route their model calls to models the operator chooses while developers keep their existing harness workflows. Its site says it helps enterprises account for what their agents cost, can access and do across providers. Written in Rust, with a Docker-only quick start.
Open sourcekeyword match
SWE-agent (Princeton & Stanford)/mini-swe-agent · Python
mini-swe-agent is a deliberately minimal software-engineering agent from the Princeton and Stanford team behind SWE-bench and SWE-agent. The agent class is about 100 lines of Python, with small environment, model and run modules, and the project reports scores above 74% on SWE-bench Verified. It runs in local, Docker/Podman, Singularity/Apptainer and bubblewrap environments and supports any model via litellm, OpenRouter or Portkey. The README lists users including Meta, NVIDIA, IBM and Anyscale and says it powers Ramp's SWE-Bench. Now at v2.
Open sourcekeyword match
FailproofAI (ExosphereHost Inc)/failproofai · TypeScript
Failproof AI provides observability and enforcement for the harnesses agents run in. It hooks 12 harnesses, ten coding CLIs such as Claude Code and Codex plus the Hermes and OpenClaw chat/assistant gateways, capturing every run into one session history and blocking dangerous tool calls before they execute using 40 built-in policies. Pre-execution blocking is verified on all twelve and turn-end gates on eight. It runs locally (npm i -g failproofai); custom agents can report via a Python SDK. MIT-licensed, copyright ExosphereHost Inc.
Open sourcekeyword match
Builderz Labs/mission-control · TypeScript
Mission Control, from Builderz Labs, is a self-hosted control plane for operating AI agents: dispatch tasks, inspect runs, review failures, track spend and coordinate agent runtimes from one local dashboard backed by SQLite. It sits above runtimes such as OpenClaw, Claude Code and Codex without replacing their reasoning or tool loops, and supports workspaces, schedulers, webhooks and alerts; live session messaging needs a connected runtime gateway. It is alpha software, runs from source (Node.js 22+, pnpm) or Docker, and includes a hardened Compose overlay.
Open sourcekeyword match
evalstate/fast-agent · Python
fast-agent is a Python toolkit for interacting with LLMs that can be used as a coding agent, development toolkit, evaluation harness or workflow platform. An interactive session with shell support starts via uvx. It manages skills from a registry (LSP, hooks, compaction strategies, automation), connects to stdio or streamable-HTTP MCP servers with OAuth, and can run inside any ACP client through fast-agent-acp. It supports many providers, including Hugging Face inference providers, OpenAI Codex plans and local llama.cpp models.
Open sourcekeyword match
OpenLIT/openlit · TypeScript
OpenLIT is an Apache-2.0 platform that describes itself as built for agent harness engineering. It provides OpenTelemetry-native tracing of LLM calls, tool calls, MCP requests, retrieval and agent steps; evaluations (LLM-as-judge, programmatic and human feedback) with CI gates; runtime guardrails against prompt injection and restricted topics; a prompt hub, rule engine and secrets vault; model comparison; and cost and GPU monitoring. The self-hosted edition is free; an Enterprise Edition exists, and OpenLIT Cloud was not generally available as of this check.
Open sourcekeyword match
Kocoro Lab/Shannon · Go
Shannon is a multi-agent orchestration framework aimed at production use. It runs agents on Temporal workflows with time-travel replay debugging, enforces hard per-task and per-agent token budgets with model fallback, and supports human approval workflows and swarm collaboration. Code execution is sandboxed with WASI, policies use OPA, and observability comes via event streaming, Prometheus metrics and OpenTelemetry. It works with OpenAI, Anthropic, Google, DeepSeek, xAI and local models via Ollama, and exposes a REST API, Python SDK and desktop app.
Open sourcekeyword match
Mozilla.ai/any-agent · Python
any-agent, from Mozilla.ai, provides a single Python interface for defining, running, tracing, serving and evaluating agents across frameworks such as Agno, Google ADK, LangChain, LlamaIndex, OpenAI Agents SDK and smolagents. It began as a research project comparing agent frameworks. The README states it is now in soft deprecation: its minimal common surface has graduated to the mozilla-ai-tinyagent package, and any-agent still accepts security and bug-fix PRs but has no new features planned.
Open sourcekeyword match
TruEra/trulens · Python
TruLens instruments LLM apps and agents with a decorator, records latency, inputs, outputs, tokens and cost per step, and scores each step with LLM judges that explain their ratings. Tracing is OpenTelemetry-native, so traces are portable to any OTLP backend, and evaluations run either as traces arrive or over a dataset afterwards. It includes the RAG Triad metrics and an Agent GPA judge, and supports comparing app versions on quality, latency and cost. The README lists adopters including Walmart Global Tech, Cisco and J.P. Morgan Chase.
Open sourcekeyword match
Alibaba/open-code-review · Go
Open Code Review is an AI code review CLI open-sourced by Alibaba, where it served as the internal official review assistant for two years. It reads Git diffs and passes changed files to a configurable OpenAI- or Anthropic-compatible LLM via a tool-using agent that can read files and search the codebase, producing line-level comments. Its design pairs deterministic pipelines (hard constraints on coverage and positions) with the agent, plus a built-in multi-language ruleset. Its README reports higher precision and F1 than Claude Code on a 200-PR benchmark at about 1/9 the tokens, with lower recall.
Open sourcekeyword match
Traceloop/openllmetry · Python
OpenLLMetry is a set of extensions on top of OpenTelemetry that instrument LLM providers and vector databases, plus a Traceloop SDK that starts tracing with one init call. Because it emits standard OpenTelemetry data it connects to existing backends — the README lists Traceloop, Axiom, Azure Application Insights, Braintrust, Dash0, Datadog, Dynatrace, Google Cloud and more. Its semantic conventions have been contributed to OpenTelemetry. Built by Traceloop; a JS/TS version lives in openllmetry-js.
Open sourcekeyword match
The PR-Agent community/pr-agent · Python
PR-Agent is an open-source AI code-review agent that runs on pull requests via slash commands such as /describe, /review, /improve and /ask, each using a single LLM call. It works with GitHub, GitLab, Bitbucket, Azure DevOps and Gitea, deploys as a GitHub Action, webhook or Docker image, and reaches OpenAI, Anthropic, Gemini, DeepSeek and other models through LiteLLM. It is a community-maintained legacy project of Qodo (earlier images were published under the codiumai namespace), distinct from Qodo's commercial review product, with Qodo as a sponsor.
Open sourcekeyword match
ccusage/ccusage · Rust
ccusage is a CLI that analyzes token usage and cost for Claude Code and Codex sessions from local logs, with a website at ccusage.com. The package is MIT licensed.
Open sourcekeyword match
AutoCodeRoverSG/auto-code-rover · Python
AutoCodeRover is a project-structure-aware autonomous software engineer aimed at autonomous program improvement. Its repo reports resolving 37.3% of SWE-bench Lite and 46.2% of SWE-bench Verified tasks (pass@1), at under $0.7 per task. It is released under the SONAR Source-Available License v1.0.
Open sourcekeyword match
Maciek-roboblog/Claude-Code-Usage-Monitor · Python
A terminal tool that tracks Claude Code token usage in real time. It shows predictions and warns you about upcoming usage limits.
Open sourcekeyword match
jarrodwatts/claude-hud · JavaScript
Claude HUD is a Claude Code plugin that shows context usage, active tools, running agents and todo progress as a session runs.
Open sourcekeyword match
apache/burr · Python
Apache Burr is a framework for building applications that make decisions, such as chatbots, agents and simulations. It lets you monitor, trace, persist and execute them on your own infrastructure.
Open sourcekeyword match
vibrantlabsai/ragas · Python
Open-source toolkit for evaluating LLM applications, including RAG pipelines and agents.
Open sourcekeyword match
mlflow/mlflow · Python
Open-source AI engineering platform for agents, LLMs and ML models. It lets teams debug, evaluate, monitor and optimize production AI applications while controlling costs and managing access to models and data.
Open sourcekeyword match
wandb/weave · Python
Weave is a toolkit from Weights & Biases for developing AI-powered applications, used for tracing and evaluating LLM apps and agents.
Open sourcekeyword match
openai/evals · Python
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. Maintained by OpenAI.
Open sourcekeyword match
harbor-framework/harbor · Python
Harbor is a framework for evaluating and improving agents, from the harbor-framework organization that also maintains Terminal-Bench.
Open sourcekeyword match
THUDM/AgentBench · Python
AgentBench is a comprehensive benchmark to evaluate LLMs as agents, published at ICLR 2024 by THUDM.
Open sourcekeyword match
sierra-research/tau-bench · Python
Tau-bench is Sierra Research's benchmark code and data for evaluating tool-using agents in conversations with simulated users.
Open sourcekeyword match
meta-llama/PurpleLlama · Python
PurpleLlama is Meta's set of tools to assess and improve LLM security. The repo contains LlamaFirewall, the Llama Guard safeguards, Prompt Guard (protection against malicious prompts), Code Shield and the CybersecurityBenchmarks (CyberSecEval).
Open sourcekeyword match
NVIDIA/garak · Python
garak is NVIDIA's open-source LLM vulnerability scanner. It probes LLM systems for weaknesses such as jailbreaks, prompt injection and data leakage.
Open sourcekeyword match
web-infra-dev/midscene · TypeScript
Midscene.js is vision-driven UI automation driven by natural language. The repo describes it as E2E testing in natural language.
Open sourcekeyword match
hex/claude-council · Shell
Claude Code plugin that asks several AI coding agents the same question and shows their answers side by side. It supports Gemini, OpenAI, Grok, Perplexity, Kimi and OpenRouter models over API; codex, antigravity, grok, kimi and cursor CLIs on subscription auth; and ollama locally. Codex specialists can take coding tasks on their own branch (experimental).
Open sourcekeyword match
yeameen/claude-code-review-council · See repository
A Claude Code plugin that runs seven reviewers in parallel on the same diff: Codex CLI, Gemini via the Antigravity CLI, and five Claude specialist subagents (security, performance, logic, regression, robustness). A synthesis step de-duplicates findings, verifies each against the actual code, tags severity P0-P3 and surfaces disagreements before any fixes are applied.
Open sourcekeyword match
AltimateAI/claude-consensus · See repository
Multiple AI models independently review code, implementation plans, documents or technical proposals, then converge through structured synthesis and approval rounds. The lead host takes part directly: Claude leads in Claude Code, and Codex leads in Codex. It installs as a plugin and provides commands such as /code-review and /plan-review.
Open sourcekeyword match
nyldn/claude-octopus · Shell
Claude Code plugin that runs multiple models (Codex, Gemini, Copilot and others) against the same research, design or coding task and surfaces disagreements.
Open sourcekeyword match
Q00/ouroboros · Python
Interview-gated, staged-evaluation, budgeted evolution loop delivered as an MCP server across 14 runtimes including Claude Code, Codex CLI, Gemini CLI and OpenCode.
Open sourcekeyword match
disler/claude-code-hooks-multi-agent-observability · Python
Real-time monitoring for Claude Code agents through simple hook event tracking.
Open sourcekeyword match
millionco/react-doctor · TypeScript
Diagnostic tool and agent skill that finds React anti-patterns in code written by coding agents.
Open sourcekeyword match
NousResearch/hermes-agent-self-evolution · Python
Nous Research layer that optimizes Hermes Agent skills, prompts and code using DSPy and GEPA evolutionary search.
Open sourcekeyword match
hoangsonww/Claude-Code-Agent-Monitor · TypeScript
Monitoring dashboard (SQLite, Express, React) tracking sessions, subagent orchestration and tool usage for Claude Code and Codex.
Open sourcekeyword match
ZHangZHengEric/Sage · Python
Sage is an MIT-licensed agent harness for developers using Desktop, Server/Studio or the SAgents Python runtime. It supports versioned agent packages, persistent sessions, streamed output, tools, Skills, MCP and multi-agent workflows. Its RSI workflow uses goals or curated history to produce candidate versions with evaluation evidence; users choose whether to adopt them. Desktop RSI currently supports simple types and native tools, not every general MCP integration. Built-in configurations target one process or host; Server supports one worker. Local execution is not container isolation, and persistent state does not provide distributed job takeover or automatic checkpoint resume. AI-assisted submission on behalf of the maintainer, based on the official README; no independent testing is claimed.
Open sourcekeyword match
raga-ai-hub/RagaAI-Catalyst · Python
Python SDK for agent observability, monitoring, and evaluation. It includes agent, LLM, and tool tracing, multi-agent debugging, a self-hosted dashboard, and analytics with timeline and execution-graph views. Apache-2.0 licensed.
Open sourcekeyword match
algorithmicsuperintelligence/openevolve · Python
OpenEvolve is an Apache-2.0 Python implementation of AlphaEvolve. Repository topics include evolutionary algorithms, iterative refinement, LLM ensembles, and coding agents.
Open sourcekeyword match
gepa-ai/gepa · Jupyter Notebook
GEPA applies reflective optimization to prompts, code, and other artifacts. Topics include harness engineering, agent skills, coding agents, and LLM evaluation.
Open sourcekeyword match
SolaceProducts/solace-agent-mesh · Shell
Platform for building and running AI agents across build, test, deploy, monitor, and improve. It hosts agents, runs their tools, and coordinates them over an event mesh, with sessions, artifacts, auth, evaluators, and observability built in.
Open sourcekeyword match
NVlabs/SoL-Pi · TypeScript
NVlabs project on scaling auto-research loops for efficient agent harnesses. TypeScript codebase under the MIT license, with a project page.
Open sourcekeyword match
Sahir619/fable-method · Python
fable-method distills how Claude Fable 5 worked into skills any model can run, plus an eval. It targets coding-agent workflows with a think, act, and prove loop and is MIT-licensed Python.
Open sourcekeyword match
ShenSeanChen/waku-agent · Python
Waku Agent is a local-first AI agent harness that includes the loop, memory, and eval in code intended to stay legible as it grows. It is written in Python and released under the MIT license.
Open sourcekeyword match
peteromallet/dataclaw · Python
Dataclaw is an agent harness for publishing agent chat history as Hugging Face datasets. It is written in Python and released under the MIT license.
Open sourcekeyword match
msoedov/agentic_security · Python
Agentic Security is a Python vulnerability scanner and red-teaming kit for agentic LLMs. It is released under the Apache-2.0 license.
Open sourcekeyword match
trpc-group/trpc-agent-go · Go
A Go framework for building production agent systems. It includes graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation, and observability.
Open sourcekeyword match
langwatch/better-agents · TypeScript
TypeScript project from LangWatch described as standards for building agents. Topics include agent testing, a CLI, MCP, and LLMOps. MIT-licensed.
Open sourcekeyword match
microsoft/SkillOpt · Python
Text-space optimizer that trains reusable natural-language skills for frozen LLM agents. It applies trajectory-driven edits and validation-gated updates, and emits a deployable best_skill.md artifact.
Open sourcekeyword match
alchaincyf/darwin-skill · HTML
Autoresearch-inspired skill optimization for Claude Code, Codex, OpenClaw, and other harnesses. It evaluates a skill, improves it, tests the change, and keeps or reverts the result.
Open sourcekeyword match
Gen-Verse/OpenClaw-RL · Python
Project described as training any agent by talking. Repository topics include GRPO, RLHF, on-policy distillation, skill learning, memory systems, and SGLang, and the homepage links an arXiv paper. It targets OpenClaw and other harnesses.
Open sourcekeyword match
NVIDIA/SkillEvaluator · Python
Multi-tier framework for evaluating AI agent skills and plugins. It includes quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
Open sourcekeyword match
rpamis/comet · JavaScript
Comet is an agent skill harness for turning ideas into evaluated workflows. It focuses on skill creation, specs, phase-guarded loops, and evals.
Open sourcekeyword match
yaojingang/yao-meta-skill · Python
YAO (Yielding AI Outcomes) is an MIT-licensed system for engineering, evaluation, governance, and portability of reusable agent skills.
Open sourcekeyword match
AMAP-ML/LongHorizon-Harness · Python
A computer-use harness for running AI agents across desktop apps and the CLI for extended periods while preserving task state. It provides fresh-context execution, durable verified state, independent auditing, and recoverable progress, with native Claude Code, Codex, and OpenClaw integration.
Open sourcekeyword match
reticlehq/reticle · TypeScript
Reticle is an agent skill and MCP server for testing and verifying what agents build. It adds machine-native runtime perception for web and desktop applications and is listed for Claude Code.
Open sourcekeyword match
alexeyw/lot-map · TypeScript
lot-map shows a live map beside the chat of what Claude Code is doing. Tool calls, model steps, hooks, skills, and subagents light up as they happen. It is local only and available in 7 languages.
Open sourcekeyword match
sno-ai/sno-station-skills · Shell
Open-source skills for Claude Code, Codex, Cursor, and Hermes Agent. They cover review by a different agent, a handover before a usage limit stops the work, treating done as proven by a real run, and a nightly loop that learns from your own sessions.
Open sourcekeyword match
hamelsmu/claude-review-loop · Shell
Claude Code plugin that runs an automated code review loop with Codex. It is listed for both the Claude Code and Codex harnesses.
Open sourcekeyword match
gmickel/flow-next · Python
Workflow plugin that takes a bug, idea, or ticket to a verified pull request, with specs, cross-model review by risk, live QA, and receipts in the repo. It lists Claude Code, Codex, Cursor, Droid, Grok Build, OpenCode, and Muse Code.
Open sourcekeyword match
karanb192/claude-code-hooks · JavaScript
Claude Code hooks plus an installable plugin marketplace covering safety, cost, observability, and productivity.
Open sourcekeyword match
stormzhang/token-tracker · Python
Local token tracker for Claude Code and Codex. It adds a status line, a GitHub-style usage heatmap, and multi-model cost analysis.
Open sourcekeyword match
boshu2/agentops · Go
DevOps discipline for AI coding agents that shapes the work, tracks it as a graph, and has each change judged by a context that did not write it. Plugins target Claude Code, Codex, OpenCode, and Cursor.
Open sourcekeyword match
seyedehsanhadi/sloptrim · Python
sloptrim is a local detector for AI-writing patterns that scores every prose file an agent saves. It is a Claude Code plugin written with the Python standard library only, with no network access and no model.
Open sourcekeyword match
leeguooooo/claude-code-usage-bar · Python
claude-code-usage-bar is a lightweight Claude Code statusLine for 5-hour and 7-day rate-limit usage, reset countdowns, model and context window, and prompt-cache age. It includes 3 styles, 9 themes, and a daemon fast-mode, and is published on PyPI.
Open sourcekeyword match
Ibrahim-3d/orchestrator-supaconductor · Python
Multi-agent orchestration system for Claude Code with parallel execution, automated quality gates, a Board of Directors, and bundled Superpowers skills.
Open sourcekeyword match
hi0001234d/nexpath · TypeScript
A local-first AI coding workflow that checks for missing tests and safety checks across Claude Code, Cursor, Windsurf, Replit, Lovable, and Bolt.new. It is written in TypeScript and licensed Apache-2.0.
Open sourcekeyword match
Hotakus/opencode-visual-cache · TypeScript
Sidebar plugin for the OpenCode TUI that displays real-time token cache hit rate. Written in TypeScript, MIT-licensed, and published as the npm package opencode-visual-cache.
Gatheringkeyword match
Recurring — next: Oct 27, 2026 · San Francisco, CA
A highly technical San Francisco meetup series covering 'all sorts of agent engineering: prompt, context, eval, harness, memory, guardrails, inference, loop'. Next up: 'Code Engineering — From Coding Agents to Software Factories' (Oct 27, AWS Builder Loft).
Gatheringkeyword match
Nov 4, 2026 · London, UK
The Friendly Data Meetup runs an evening hackathon (17:30–21:00) at 230 Blackfriars Rd on building evaluation harnesses for AI agents. Registration via Luma.
Gatheringkeyword match
Nov 2, 2026 · San Francisco, CA
A one-day buildathon: take open models and make your own frontier on a task you define, by training a specialised model, building a smarter harness or routing across models, then evaluating it. $10k prize.
Gatheringkeyword match
Nov 21, 2026 · Online
An online session (09:00–11:00 ET) on instrumenting AI agents' LLM calls, tool calls and retrieval with OpenTelemetry, then running that telemetry in production: redaction, cost and latency metrics, tail sampling and diagnosing agent failures.
Gatheringkeyword match
Dec 4, 2026 · Online
A three-hour online workshop (09:00–12:00 ET) that takes a refund agent from prototype to production: it reproduces prompt injection, duplicate payments and runaway retries, then applies eval gates, guardrails, tracing and rollback.
Gatheringkeyword match
Oct 29, 2026 · San Francisco, CA
Live demos and deep dives at the AWS Builder Loft, 525 Market St, on building production agentic AI systems with AWS, Temporal (durable execution), Arize (observability and evals) and Databricks. Register on aicamp.ai.
Gatheringkeyword match
Dec 15–16, 2026 · Jersey City, NJ
A practitioner AI engineering conference at The Westin Jersey City Newport focused on production agents: agent identity and authorisation (1Password keynote), guardrails on ops agents and eval loops for agentic systems.