Promptfoo

What it is for

Promptfoo is a local LLM eval-and-attack harness: write promptfooconfig.yaml, compare prompts × providers × tests, then red-team the same app. Use it when the job is fail the build before a prompt ships, not when you need a hosted trace desk.

What makes it unique: the cover is the official promptfoo/promptfoo GitHub file list — examples/, src/, MIT in the About pane — not a marketing orb.

Key features

• Eval matrix — one YAML: prompts, 60+ providers, and asserts (contains, llm-rubric, cost/latency) then promptfoo eval + view (Getting started). Prompt / RAG / agent fixtures are this matrix, not three products.
• Red team / vuln scan — adversarial probes (Community 10k / month) that produce a vulnerability report (Red team; pricing). One attack stream; the scan plugins are this, not a second product.
• CI/CD gate — run the same config in a pipeline and fail the job (Getting started CI). A gate, not a local view.
• Live-app target — point the harness at your own app, not only a prompt string (pricing “Custom integration with your own app”). A target, not a YAML prompt.
• Enterprise dashboard — centralized monitoring, team sharing, and remediation on the paid cards (pricing Enterprise). A hosted SOC view, not promptfoo view.

The CLI / web setup (promptfoo eval setup) are the same harness two ways. Vulnerability scanning sits on the red-team stream. Promptfoo API access is install / SDK (Enterprise). Probes and eval runs are the meter. Model names are SKU.

Who it's for

Best fit: app teams who will run Community for $0 (MIT, self-host) and only buy Enterprise when they need SSO, a shared dashboard, and more than 10k probes.

Pricing

Verified 2026-10-22 (Asia/Shanghai) from promptfoo.dev/pricing and the MIT LICENSE.

• Community — Free Forever. All LLM evaluation features; all providers; red team 10k probes/month; vulnerability scanning; custom app integration; run locally or self-host; community support.
• Enterprise — custom. Team sharing; continuous monitoring; centralized dashboard; custom attack profiles; SSO / granular roles; Promptfoo API; managed cloud; priority SLA.
• On-Premise — custom. Enterprise on your infrastructure; data isolation; dedicated runner; assigned deployment engineer.

Do not invent a $ / seat. Community is $0; paid dollars are a quote.

Limitations / costs

• Community red team is 10k probes/month. Extra probes are an Enterprise purchase.
• No public Enterprise dollar. Confirm the quote; do not copy a blog “from $X”.
• Eval/attack CLI, not a trace warehouse. For production spans + experiments, see Phoenix.

Alternatives

• Phoenix — OTel traces + experiments when you need a local observability UI.
• Confident AI — hosted DeepEval desk when you do not want to self-host.
• Langfuse — open-source traces when Promptfoo’s YAML matrix is not the job.

Official tutorial

Official promptfoo channel exists. We do not embed a video that is not a first-hour getting-started lesson. Use written Getting started plus Pricing.

Website

https://www.promptfoo.dev

Repository

https://github.com/promptfoo/promptfoo

Source

Catalog row curated for AI Tools Nav. Cover is the official promptfoo/promptfoo GitHub file list (MIT + examples/). Dollars checked on promptfoo.dev/pricing (Community Free Forever; Enterprise / On-Prem custom). Channel: @promptfoo. No getting-started embed.

Last verified

2026-10-22 (Asia/Shanghai) editorial check.