Promptfoo CLI
A CLI for testing and red teaming LLM applications: compares models, runs automated CI checks and finds vulnerabilities
- 27,080 ★ on the repository
- Last commit
- MIT
Extended access
Why this level
- Makes network calls to model APIs and can consume paid tokens
- Red team mode sends adversarial test attacks against your LLM application
Install
- Works in
In your terminal, with SkillFoxx CLI
npx skillfoxx add cli/promptfooDetects the agents on your machine, shows the access level and pins the version.
Other ways to install
Access and checks
How we reviewExtended accessThe tool runs code or makes network calls and only reads external data.
- Makes network calls to model APIs and can consume paid tokens
- Red team mode sends adversarial test attacks against your LLM application
This is third-party code. Review the repository files before installing.
What it does
Promptfoo runs test suites against prompts, agents and RAG pipelines and compares outputs from different models against defined criteria. A separate red team mode simulates attacks on an LLM application and looks for issues like prompt leakage or guardrail bypass. Test suites are described in a declarative YAML config, and results open in a local web viewer or plug into CI. A Claude Code plugin ships a skill that helps write and maintain eval suites directly in conversation with the agent.
Who it is for. For developers and engineers who verify prompt quality and LLM application security before release.
Good fit when
- You need to compare several models' answers on the same prompt set
- You need regression checks for answer quality in CI
- You need to test an application for prompt injection and other vulnerabilities
Not a fit when
- You just need a one-off check of a single prompt without keeping history
- You don't have API access to the models you want to compare
Example request
Set up a promptfoo config comparing GPT and Claude on my system prompts and run the evalLimitations
Comparing models needs separate provider API keys (OpenAI, Anthropic and others), some of which are unavailable from Russia without a VPN. Red teaming and some checks require target configuration and can consume a noticeable amount of tokens on large test suites.
How to disable. Remove the promptfoo package (npm uninstall -g promptfoo or the brew/pip equivalent) and delete promptfooconfig.yaml files from the project. For the Claude Code plugin, remove it via /plugin.
FAQ
How is promptfoo different from manually testing prompts?
It runs the same test suite across multiple models and providers automatically and keeps a comparison history, instead of one-off checks in a chat window.
Do I need a promptfoo account to use it?
No, evaluations run locally with your own model API keys, no registration required.
Discussion
No questions or reviews yet.
Related
CLI toolsA CLI that tests AI agents end to end: derives scenarios from code, runs them live and separates unverified from failed.
TesterArmy CLI and MCP run end-to-end browser and mobile tests in the cloud from plain-language steps.
CLI checks a web page in Chromium: finds every control, uses it and reports what broke. No tests to write and no LLM.
QA agents, skills and an MCP server: test generation, coverage gaps, flaky tests, security scans and chaos testing
Behavioral eval lab Quorum for the superpowers project: drives real agent CLIs and grades them against private criteria
Qlty CLI reports duplicate and complex code before refactoring; automatic formatting runs as a separate action.
Comparison with similar tools
Cards that share the main search query, and how they differ.
| Tool | Official | Repository stars | Main difference |
|---|---|---|---|
| Promptfoo (this card) | no | 27,080 | |
| OpenART | no | 275 | A guide skill for OpenART, an executable benchmark that tests the safety of coding agents in long-running, changing environments |
| Overmind | yes | 757 | Overmind CLI and MCP server |
| iFixAi | no | 45,036 | iFixAi probes an AI agent in simulated business situations, grades its replies and saves evidence of failures in reports. |
README in short
The README describes promptfoo as a CLI and library for evaluating and red teaming LLM applications. Main scenarios: comparing models and providers, regression checks in CI, vulnerability scanning and security reports. It installs via npm, brew or pip, and evaluations run locally without sending prompts to promptfoo's servers. The project recently joined OpenAI but stays open source under the MIT license.