prompt-siren

KO | EN

gitlite — search

#php #javascript #deprecated #linux #java #library #ruby #python #objective-c #ios #dotfiles #nodejs

★ 59 Open GitHub ↗

A research workbench for developing and testing attacks against large language models, with a focus on prompt injection vulnerabilities and defenses.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Prompt Injection Workbench

[![CI](https://github.com/facebookresearch/prompt-siren/actions/workflows/ci.yml/badge.svg)](https://github.com/facebookresearch/prompt-siren/actions/workflows/ci.yml)

A research workbench for developing and testing attacks against large language models, with a focus on prompt injection vulnerabilities and defenses.

## Key Features

- **State Machine Design**: Fine-grained control over agent execution for advanced attack scenarios
- **SWE-bench Support**: Benchmark agents on real-world code editing tasks from SWE-bench
- **Hydra Configuration**: Powerful experiment orchestration with parameter sweeps
- **Extensible Architecture**: Plugin system for custom agents, attacks, and environments
- **Usage Limits**: Built-in cost and resource controls
- **Experiment Tracking**: Automatic caching and result organization

## Quick Start

### Installation

Install the core package with desired optional features:

```sh
# Full installation (all features)
uv sync --all-extras

# Or install only what you need:
uv sync --extra agentdojo      # AgentDojo benchmark support
uv sync --extra swebench       # SWE-bench support
uv sync --extra docker         # Docker sandbox manager
uv sync --extra playwright     # Web automation environment

# Combine multiple extras
uv sync --extra agentdojo --extra docker
```

**Available optional dependencies:**

| Extra | Description |
|-------|-------------|
| `agentdojo` | AgentDojo dataset, environment, and attacks |
| `swebench` | SWE-bench dataset for code editing benchmarks |
| `docker` | Docker sandbox manager |
| `playwright` | Web automation environment |

Set up environment variables:
```sh
cp .env.example .env  # Fill in API keys
```

Export default configuration:
```sh
# Export to ./config (default)
uv run prompt-siren config export

# Export to custom directory
uv run prompt-siren config export ./my_config
```

Run experiments:
```sh
# Run benign-only evaluation
uv run prompt-siren run benign +dataset=agentdojo-workspace

# Run with attack
uv run prompt-siren run attack +dataset=agentdojo-workspace +attack=template_string

# Run SWE-bench evaluation (requires Docker)
uv run prompt-siren run benign +dataset=swebench

# Run SWE-bench Lite (smaller benchmark)
uv run prompt-siren run benign +dataset=swebench dataset.config.dataset_name="SWE-bench/SWE-bench_Lite"

# Override parameters
uv run prompt-siren run benign +dataset=agentdojo-workspace agent.config.model=azure:gpt-5

# Parameter sweep (multirun)
uv run prompt-siren run benign --multirun +dataset=agentdojo-workspace agent.config.model=azure:gpt-5,azure:gpt-5-nano

# Validate configuration without running
uv run prompt-siren config validate +dataset=agentdojo-workspace

# Use config file with environment/attack included (no overrides needed)
uv run prompt-siren run attack --config-dir=./my_config
```

**Tip**: Environment and attack can be specified via CLI overrides or included directly in config files. See the [Configuration Guide](docs/configuration.md) for details.

## Analyzing Results

After running experiments, use the `results` command to aggregate and analyze results:

```sh
# View results with default settings (pass@1, grouped by all configs)
uv run prompt-siren results

# Specify custom jobs directory
uv run prompt-siren results --jobs-dir=./jobs

# Group results by different dimensions
uv run prompt-siren results --group-by=model
uv run prompt-siren results --group-by=env
uv run prompt-siren results --group-by=agent
uv run prompt-siren results --group-by=attack

# Compute pass@k metrics (k>1)
uv run prompt-siren results --k=5
uv run prompt-siren results --k=10

# Compute multiple pass@k metrics simultaneously
uv run prompt-siren results --k=1 --k=5 --k=10

# Different output formats
uv run prompt-siren results --format=json
uv run prompt-siren results --format=csv
```

### Understanding pass@k Metrics

- **pass@1** (default): Averages scores across all runs for each task. A continuous metric showing average performance.
- **pass@k** (k>1): Binary success metric. A task "passes" if at least one of k runs achieves a perfect score (1.0). Uses statistical estimation when more than k samples are available.

### Results Columns

The results table includes:
- **Configuration columns**: `dataset`, `agent_type`, `agent_name`, `attack_type`
- **Metric columns**: `benign_pass@k`, `attack_pass@k` - The pass@k scores
- **Metadata columns**:
  - `n_tasks` - Total number of tasks aggregated
  - `avg_n_samples` - Average number of runs per task
  - `k` - The k value (when computing multiple pass@k metrics)
## Platform Requirements

- **Python**: 3.10+
- **Package Manager for development**: `uv` (for dependency management)
- **Operating System**: Linux or macOS (Windows not supported)
- **Docker**: Required for SWE-bench integration and some environments
  - Must be running and accessible
  - Base images should have `/bin/bash` available (Alpine images need `bash` package)

## Documentation

- **[Configuration Guide](docs/configuration.md)** - Hydra configuration and parameter overrides
- **[Usage Limits](docs/usage_limits.md)** - Resource limits and cost controls
- **[Plugins](docs/plugins/README.md)** - Adding custom agents, attacks, and environments

## Development

### Linting, Formatting and Typechecking

```sh
# Lint and format
uv run ruff check --fix
uv run ruff format
uv run ty check

# Test
uv run pytest -v
```

### Pre-building Docker Images for SWE-bench

The main `prompt-siren` CLI only works with pre-built Docker images. To build all required images for SWE-bench evaluations, use the `prompt-siren-build-images` command:

Default registry for pushes: `ghcr.io/ethz-spylab/prompt-siren-images`. Use `--registry ""`
to disable pushes or override with your own registry.

```sh
# Build all images (benign, malicious service containers, and pairs)
uv run prompt-siren-build-images

# Build images for specific instances only
uv run prompt-siren-build-images --instance-ids django__django-11179 --instance-ids astropy__astropy-12907

# Limit the number of instances to build (useful for testing)
uv run prompt-siren-build-images --max-instances 5

# Use SWE-bench Lite dataset
uv run prompt-siren-build-images --dataset "SWE-bench/SWE-bench_Lite"

# Skip building certain image types
uv run prompt-siren-build-images --skip-benign      # Skip benign task images
uv run prompt-siren-build-images --skip-malicious   # Skip malicious task service images
uv run prompt-siren-build-images --skip-pairs       # Skip combined pair images

# Rebuild existing images instead of skipping them
uv run prompt-siren-build-images --rebuild-existing

# Tag and push images to a registry
uv run prompt-siren-build-images --registry my-registry.com/myrepo

# Enable verbose logging for debugging
uv run prompt-siren-build-images --verbose

# Specify a custom cache directory
uv run prompt-siren-build-images --cache-dir /path/to/cache
```

**Common workflows:**

```sh
# Quick test: Build images for a single instance
uv run prompt-siren-build-images --instance-ids django__django-11179

# Full build for production: Build all images and push to registry
uv run prompt-siren-build-images --registry my-registry.com/swebench

# Rebuild after updates: Force rebuild of existing images
uv run prompt-siren-build-images --rebuild-existing

# Build only benign images (no attack scenarios)
uv run prompt-siren-build-images --skip-malicious --skip-pairs
```

> **Note:** Once images are built, they are cached locally and will be reused by the main `prompt-siren run` command. The main CLI does not build images on-the-fly—it expects pre-built images to be available.

# License
Prompt Siren is licensed under an [MIT License](https://github.com/facebookresearch/prompt-siren/blob/main/LICENSE.md).

prompt-siren

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README