Promptfoo
Open source CLI for LLM evaluation and red teaming. Generates adversarial cases against your prompt templates and agents, runs them in CI, reports vulnerabilities. MIT licensed.
Overview
Promptfoo started as a prompt evaluation harness and grew a security half. Both run from one CLI against a YAML config, which is the design decision that made it stick. There is no server to stand up and no SDK to adopt. You describe the thing under test, whether that is a raw model, a prompt template with variables, a HTTP endpoint or an agent with tools, then describe assertions or ask for a red team run, and it executes locally.
The red team module generates adversarial inputs across more than fifty plugin categories. Direct and indirect prompt injection, jailbreaks, PII disclosure, SSRF, SQL injection through a tool, excessive agency and hallucination all have generators. Because it targets the assembled application, it can find the class of bug that matters most in practice: a document in your vector store carrying instructions that the model obeys. A model-level scanner will never see that. The Community tier caps cloud probes at 10,000 a month, and local runs are unmetered.
Two honest caveats. First, depth. Against a hardened frontier model, garak's probe library will surface things promptfoo's generators do not, so they complement rather than replace each other. Second, ownership. OpenAI announced the acquisition on 9 March 2026, said the project stays open source under MIT and said existing customers keep support. The repo banner confirms the licence today. Whether an OpenAI-owned tool keeps investing in adversarial coverage of competing models is a reasonable thing to keep an eye on.
Key Features
- ✓ YAML config plus a CLI that drops into an existing CI job without a separate service to run
- ✓ Red team mode auto-generates adversarial cases across 50 plus attack plugins including direct and indirect prompt injection, PII leakage, SSRF, SQL injection and excessive agency
- ✓ Tests the assembled application, so prompt template, retrieval context and tool definitions are all in scope
- ✓ Side-by-side eval matrix for comparing prompts, models and parameters on the same test set
- ✓ Local-first execution, with test data staying on your machine unless you opt into the cloud dashboard
- ✓ Machine readable results suitable for a release gate, plus a shareable web view
Where it holds
- • The lowest friction path from zero to a working red team run of anything in this category
- • MIT licence on the core, with the OpenAI deal explicitly preserving it
- • Application-level scope catches indirect injection through retrieved documents, which model-level scanners miss
- • Very large user base, so failure modes and workarounds are already documented by someone else
Where it breaks
- • Jailbreak coverage is broad but shallower than a dedicated probe library like garak
- • Attack generation itself calls an LLM, so runs cost tokens and are not fully deterministic
- • Enterprise pricing is opaque, and the useful team features sit behind it
- • New owner risk is real. OpenAI keeping the licence open is a promise, not a guarantee about roadmap priorities
My Take
Config file, YAML, run it in CI. That is the whole pitch, and it explains the spread: 24.6k stars, version 0.122.1 published this week, and by the company's own account more than a quarter of the Fortune 500 using it somewhere. The red team side generates adversarial cases against your prompt template and tool definitions rather than the bare model, which is exactly the gap garak leaves open. OpenAI announced it was buying the eleven-person team on 9 March 2026 and committed to keeping the licence MIT. Worth watching. The eval half is still the stronger half, and if you want deep jailbreak taxonomy you will end up running both tools.
Quick Info
- Pricing:
- freemium
- Openness:
- Open source
- Licence:
- MIT
- Starting at:
- Core CLI is MIT and free forever, including local red team runs and all evaluation features. The self-serve Community tier caps cloud red teaming at 10,000 probes a month. Enterprise and on-premise are quote-only with no published list price. OpenAI announced the acquisition on 9 March 2026 and the repo still ships under MIT.
- Added:
- Aug 2026
- Updated:
- Aug 2026
Use Cases
Judge it on your own work
The notes above say where Promptfoo holds and where it breaks. The fastest check is your own workload.
Visit website ↗Alternatives to Promptfoo
garak
open-sourceCommand line vulnerability scanner for LLMs, maintained by NVIDIA. Fires adversarial probe suites at a model endpoint, scores replies with detectors, writes JSONL plus an HTML digest.
Braintrust
freemiumEval-first observability for agents: datasets, scorers and experiments in one loop. Unlimited seats on every tier, Pro at $249 a month flat. SaaS only unless you buy Enterprise.
Langfuse
open-sourceOpen-source LLM tracing, prompt management and evals. MIT core with no seat or usage caps when self-hosted, cloud from $29 a month. Acquired by ClickHouse in January 2026.
Arize Phoenix
open-sourceSelf-hosted LLM tracing and evals built on OpenTelemetry. Free and uncapped under Elastic License 2.0, source-available rather than OSI open source. Runs in a notebook or on Kubernetes.