promptfoo

(β˜… 24,435)

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

File Explorer

Download Latest Version (.zip)

Showing a partial file list β€” download the zip above to see everything.

  • .biomeignore
  • .dockerignore
  • .git-blame-ignore-revs
  • .gitignore
  • .mailmap
  • .npmignore
  • .npmrc
  • .nvmrc
  • .prettierignore
  • .prettierrc.yaml
  • .release-please-manifest.json
  • .rubocop.yml
  • .ruff.toml
  • AGENTS.md
  • biome.jsonc
  • CHANGELOG.md
  • CITATION.cff
  • CLAUDE.md
  • CODE_OF_CONDUCT.md
  • codecov.yml
  • CONTRIBUTING.md
  • Dockerfile
  • drizzle.config.ts
  • knip.jsonc
  • LICENSE
  • package-lock.json
  • package.json
  • pnpm-workspace.yaml
  • README.md
  • release-please-config.json
  • renovate.json
  • SECURITY.md

πŸš€ Installation Guide

1. Get the code
git clone https://github.com/promptfoo/promptfoo

Downloads the entire project code from GitHub to your computer.

cd promptfoo

Moves into the project folder you just downloaded.

2. Official Install Script

Easy Recommended
Prerequisites
  • Node.js Node.js must be installed to use npm.
npm install -g promptfoo

Installs the package published on the npm registry globally β€” no need to clone the source.

βœ… After installing, open a new terminal and run the program's version command (e.g. --version) to confirm it worked.

Pulled directly from this repo's README.

3. Docker

Easy
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Docker Desktop Needed to build and run containers. Install it and keep it running in the background.
docker build -t promptfoo .

Builds a runnable image based on the Dockerfile.

docker run -p 8080:80 promptfoo

Runs the built image as an actual container.

βœ… Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

4. Node.js

Easy
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Node.js Node.js must be installed to use npm. The LTS version is recommended.
npm install -g promptfoo

Installs the package published on the npm registry globally β€” no need to clone the source.

βœ… After running the command, open the address shown in the terminal (usually something like http://localhost:3000) in your browser.

Pulled directly from this repo's README.

5. Python

Easy
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.
pip install -r examples/eval-bert-score/requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

python <μ‹€ν–‰ν•  파일λͺ…>.py # READMEμ—μ„œ μ •ν™•ν•œ μ‹€ν–‰ 파일λͺ…을 ν™•μΈν•˜μ„Έμš”

Runs the Python script (or module).

βœ… If it runs without errors and prints output in the terminal, it worked.

6. Go

Medium
Prerequisites
  • Git Needed to download the project code from GitHub.
  • Go The Go compiler and toolchain.
cd examples/provider-golang

This project's files live in a subfolder, so move into it first.

go build ./...

Compiles the Go program into an executable.

go run .

Runs the Go program directly without a separate build step.

βœ… If the build finishes without errors, it worked. If you used go run ., check the terminal output.

Today's picks

superset β˜… 74,282

Apache Superset is a Data Visualization and Data Exploration Platform

Python
rtk β˜… 76,287

CLI proxy that reduces LLM token consumption by 60-90% on common dev commands. Single Rust

Rust
Deep-Live-Cam β˜… 95,973

real time face swap and one-click video deepfake with only a single image

Python
vscode β˜… 188,801

Visual Studio Code

TypeScript
langchain β˜… 144,349

The agent engineering platform.

Python
mermaid β˜… 89,792

Generation of diagrams like flowcharts or sequence diagrams from text in a similar manner

TypeScript
// repository documentation