Inspect AI
Inspect AI is a local LLM eval log viewer: you run a task (dataset × model × scorer), then open Samples with input / target / answer / score.
OUR TAKE
Pick Inspect AI for eval / safety teams who will run Inspect locally at $0 (you bring model API keys) — not Promptfoo first unless they want a YAML CI red-team harness. $0 is the toolkit, not the model bill.
Good fit
Eval / safety teams who will run Inspect locally at $0 (you bring model API keys) — not Promptfoo first unless they want a YAML CI red-team harness.
Poor fit
Inspect AI
USD, before tax · Checked 2026-10-27 · inspect.aisi.org.uk
OUR TAKE
Pick Inspect AI for eval / safety teams who will run Inspect locally at $0 (you bring model API keys) — not Promptfoo first unless they want a YAML CI red-team harness. $0 is the toolkit, not the model bill. simpleqa × 10k samples still burns your API.
Good fit
Eval / safety teams who will run Inspect locally at $0 (you bring model API keys) — not Promptfoo first unless they want a YAML CI red-team harness.
Poor fit
Inspect AI
USD, before tax · Checked 2026-10-27 · inspect.aisi.org.uk
What it costs
USD, before tax · Checked 2026-10-27 · inspect.aisi.org.uk
Inspect AI
MIT. License fee . Local CLI + log viewer. You pay model inference and machines
$0
No public hosted $.
Not published by the vendor
Not published by the vendor
Checked 2026-10-27·inspect.aisi.org.uk
Before you pay
Good fit
Eval / safety teams who will run Inspect locally at $0 (you bring model API keys) — not Promptfoo first unless they want a YAML CI red-team harness.
Watch out
$0 is the toolkit, not the model bill. simpleqa × 10k samples still burns your API.
Instead, try
Promptfoo if YAML eval + red team when you want Community Free Forever; Phoenix if OTel traces when you want Phoenix $0 self-host
Compare Inspect AI with
Limits, in full
- 01
$0 is the toolkit, not the model bill. simpleqa × 10k samples still burns your API.
- 02
A research log viewer, not a production trace warehouse. For OTel spans see Phoenix.
- 03
No official paid tile. Quote only MIT $0.
If Inspect AI isn't it
Why: YAML eval + red team when you want Community Free Forever.
Free· open source,MIT
Community red team is 10k probes/monthChecked 2026-10-27·promptfoo.dev/pricing
vs Inspect AI
Why: OTel traces when you want Phoenix $0 self-host.
Free· paid from$50/mo
ELv2, not ApacheChecked 2026-10-27·arize.com/pricing
vs Inspect AI
Why: a Python eval harness when you want DeepEval + Confident AI $0 / $200.
Free· paid from$2/mo
$0 is the library, not judge tokensChecked 2026-10-27·confident-ai.com/pricing
vs Inspect AILimits & alternatives3 limits · Promptfoo, Phoenix, DeepEval instead
Limits, in full
- 01
$0 is the toolkit, not the model bill. simpleqa × 10k samples still burns your API.
- 02
A research log viewer, not a production trace warehouse. For OTel spans see Phoenix.
- 03
No official paid tile. Quote only MIT $0.
If Inspect AI isn't it
Why: YAML eval + red team when you want Community Free Forever.
Free· open source,MIT
Community red team is 10k probes/monthChecked 2026-10-27·promptfoo.dev/pricing
vs Inspect AIWhy: OTel traces when you want Phoenix $0 self-host.
Free· paid from$50/mo
ELv2, not ApacheChecked 2026-10-27·arize.com/pricing
vs Inspect AIWhy: a Python eval harness when you want DeepEval + Confident AI $0 / $200.
Free· paid from$2/mo
$0 is the library, not judge tokensChecked 2026-10-27·confident-ai.com/pricing
vs Inspect AI