Braintrust

The positioning is deliberate. Observability is the entry point but evaluation is the product, and everything is organised around one loop: capture production traces, promote the interesting ones into a dataset, write scorers, run experiments, compare. Scorers can be plain code, an LLM judge or a human in a review queue. Runs get diffed side by side, so a prompt change that improves factuality while quietly wrecking p95 latency shows up as two numbers moving in opposite directions instead of a general feeling that things got better.

Two pieces are worth knowing about independently. Autoevals is the MIT-licensed scorer library, usable standalone if you want the graders without the platform behind them. Brainstore is a custom datastore built because Postgres and generic OLAP engines both struggle with full-text search across very large, deeply nested agent traces. Loop, added during 2026, is an in-product agent that reads your traces and proposes candidate prompts, scorers and dataset rows. Useful for getting a first eval suite off the ground. Much less useful as a substitute for knowing what quality actually means in your domain.

Commercially this is SaaS. Starter is free with 1 GB and 10k scores a month at 14-day retention, Pro is a flat $249/month with 5 GB and 50k scores, and only Enterprise offers on-prem or hybrid deployment. The genuinely unusual choice is seats: unlimited users on every plan, free included, which sidesteps the per-seat friction that keeps logins rationed inside large organisations using per-seat competitors. Billing tracks processed data and score count instead, and that is the number to model carefully before committing, because agent traces carrying tool arguments and retrieved documents are large.