augustus

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 279 Open GitHub ↗

LLM security testing framework for detecting prompt injection, jailbreaks, and adversarial attacks — 190+ probes, 28 providers, single Go binary

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Augustus - LLM Vulnerability Scanner

> Test large language models against 210+ adversarial attacks covering prompt injection, jailbreaks, encoding exploits, and data extraction.

[![CI](https://github.com/praetorian-inc/augustus/actions/workflows/ci.yaml/badge.svg)](https://github.com/praetorian-inc/augustus/actions/workflows/ci.yaml)
[![Go Version](https://img.shields.io/github/go-mod/go-version/praetorian-inc/augustus)](https://raw.githubusercontent.com/praetorian-inc/augustus/main/go.mod)
[![License](https://img.shields.io/github/license/praetorian-inc/augustus)](https://raw.githubusercontent.com/praetorian-inc/augustus/main/LICENSE)
[![Go Report Card](https://goreportcard.com/badge/github.com/praetorian-inc/augustus)](https://goreportcard.com/report/github.com/praetorian-inc/augustus)
[![GitHub Release](https://img.shields.io/github/v/release/praetorian-inc/augustus?include_prereleases&sort=semver)](https://github.com/praetorian-inc/augustus/releases)

**Augustus** is a Go-based LLM vulnerability scanner for security professionals. It tests large language models against a wide range of adversarial attacks, integrates with 28 LLM providers, and produces actionable vulnerability reports.

Unlike research-oriented tools, Augustus is built for production security testing — concurrent scanning, rate limiting, retry logic, and timeout handling come out of the box.

## Table of Contents

- [Why Augustus](#why-augustus)
- [Features](#features)
- [Quick Start](#quick-start)
- [Supported Providers](#supported-providers)
- [Usage](#usage)
  - [Single Probe](#single-probe)
  - [Multiple Probes](#multiple-probes)
  - [Buff Transformations](#buff-transformations)
  - [Output Formats](#output-formats)
  - [Custom REST Endpoints](#custom-rest-endpoints)
- [How It Works](#how-it-works)
- [Architecture](#architecture)
- [Configuration](#configuration)
- [FAQ](#faq)
- [Troubleshooting](#troubleshooting)
- [Contributing](#contributing)
- [Security](#security)
- [Support](#support)
- [License](#license)

## Why Augustus

| Feature | Augustus | garak | promptfoo |
|---------|----------|-------|-----------|
| Language | Go | Python | TypeScript |
| Single binary | Yes | No | No |
| Concurrent scanning | Goroutine pools | Multiprocessing pools | Yes |
| LLM providers | 28 | 35+ | 80+ |
| Probe types | 210+ | 160+ | 119 plugins + 36 strategies |
| Enterprise focus | Yes | Research | Yes |

## Features

| Feature | Description |
|---------|-------------|
| **210+ Vulnerability Probes** | 47 attack categories: jailbreaks, prompt injection, adversarial examples, data extraction, safety benchmarks, agent attacks, and more |
| **28 LLM Providers** | OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Ollama, and 22 more with 43 generator variants |
| **90+ Detectors** | Pattern matching, LLM-as-a-judge, HarmJudge (arXiv:2511.15304), Perspective API, unsafe content detection |
| **7 Buff Transformations** | Encoding, paraphrase, poetry (5 formats, 3 strategies), low-resource language translation, case transforms |
| **Flexible Output** | Table, JSON, JSONL, and HTML report formats |
| **Production Ready** | Concurrent scanning, rate limiting, retry logic, timeout handling |
| **Single Binary** | Go-based tool compiles to one portable executable |
| **Extensible** | Plugin-style registration via Go `init()` functions |

### Attack Categories

- **Jailbreak attacks**: DAN, DAN 11.0, AIM, AntiGPT, Grandma, ArtPrompts
- **Prompt injection**: Encoding (Base64, ROT13, Morse), Tag smuggling, FlipAttack, Prefix/Suffix injection
- **Adversarial examples**: GCG, PAIR, AutoDAN, TAP (Tree of Attack Prompts), TreeSearch, DRA
- **Multi-turn attacks**: Crescendo (gradual escalation), GOAT (adaptive technique switching)
- **Data extraction**: API key leakage, Package hallucination, PII extraction, LeakReplay
- **Context manipulation**: RAG poisoning, Context overflow, Multimodal attacks, Continuation, Divergence
- **Format exploits**: Markdown injection, YAML/JSON parsing attacks, ANSI escape, Web injection (XSS)
- **Evasion techniques**: Obfuscation, Character substitution, Translation-based attacks, Phrasing, ObscurePrompt
- **Safety benchmarks**: DoNotAnswer, RealToxicityPrompts, Snowball, LMRC
- **Agent attacks**: Multi-agent manipulation, Browsing exploits
- **Security testing**: Guardrail bypass, AV/spam scanning, Exploitation (SQLi, code exec), BadChars

> **Warning**: The `lmrc` probe uses profane and offensive language as part of its jailbreak testing. Use only in authorized testing environments.

## Quick Start

### Installation

Requires Go 1.25.3 or later.

```bash
go install github.com/praetorian-inc/augustus/cmd/augustus@latest
```

Or build from source:

```bash
git clone https://github.com/praetorian-inc/augustus.git
cd augustus
make build
```

### Basic Usage

```bash
export OPENAI_API_KEY="your-api-key"
augustus scan openai.OpenAI \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --verbose
```

### Example Output

### List Available Capabilities

```bash
# List all registered probes, detectors, generators, harnesses, and buffs
augustus list
```

## Supported Providers

Augustus includes 28 LLM provider categories with 43 generator variants:

| Provider           | Generator Name(s)         | Notes                          |
|--------------------|---------------------------|--------------------------------|
| OpenAI             | `openai.OpenAI`, `openai.OpenAIReasoning` | GPT-3.5, GPT-4, GPT-4 Turbo, o1/o3 reasoning models |
| Anthropic          | `anthropic.Anthropic`     | Claude 3/3.5/4 (Opus, Sonnet, Haiku) |
| Azure OpenAI       | `azure.AzureOpenAI`       | Azure-hosted OpenAI models     |
| AWS Bedrock        | `bedrock.Bedrock`         | Claude, Llama, Titan models    |
| Google Vertex AI   | `vertex.Vertex`           | PaLM, Gemini models            |
| Cohere             | `cohere.Cohere`           | Command, Command R models      |
| Replicate          | `replicate.Replicate`     | Cloud-hosted open models       |
| HuggingFace        | `huggingface.InferenceAPI`, `huggingface.InferenceEndpoint`, `huggingface.Pipeline`, `huggingface.LLaVA` | HF Inference API, endpoints, pipelines, multimodal |
| Together AI        | `together.Together`       | Fast inference for OSS models  |
| Anyscale           | `anyscale.Anyscale`       | Llama and Mistral hosting      |
| Groq               | `groq.Groq`               | Ultra-fast LPU inference       |
| Mistral            | `mistral.Mistral`         | Mistral API models             |
| Fireworks          | `fireworks.Fireworks`     | Production inference platform  |
| DeepInfra          | `deepinfra.DeepInfra`     | Serverless GPU inference       |
| NVIDIA NIM         | `nim.NIM`, `nim.NVOpenAICompletion`, `nim.NVMultimodal`, `nim.Vision` | NVIDIA AI endpoints, multimodal |
| NVIDIA NeMo        | `nemo.NeMo`               | NVIDIA NeMo framework          |
| NVIDIA NVCF        | `nvcf.NvcfChat`, `nvcf.NvcfCompletion` | NVIDIA Cloud Functions   |
| NeMo Guardrails    | `guardrails.NeMoGuardrails` | NVIDIA NeMo Guardrails       |
| IBM watsonx        | `watsonx.WatsonX`         | IBM watsonx.ai platform        |
| LangChain          | `langchain.LangChain`     | LangChain LLM wrapper          |
| LangChain Serve    | `langchain_serve.LangChainServe` | LangChain Serve endpoints |
| Rasa               | `rasa.RasaRest`           | Rasa conversational AI         |
| GGML               | `ggml.Ggml`               | GGML local model inference     |
| Function           | `function.Single`, `function.Multiple` | Custom function generators |
| Ollama             | `ollama.Ollama`, `ollama.OllamaChat` | Local model hosting    |
| LiteLLM            | `litellm.LiteLLM`         | Unified API proxy              |
| REST API           | `rest.Rest`               | Custom REST endpoints (SSE support) |
| Test               | `test.Blank`, `test.Repeat`, `test.Lipsum`, `test.Nones`, `test.Single`, `test.BlankVision` | Testing and development |

All providers are available in the compiled binary. Configure via environment variables or YAML configuration files. See [Configuration](#configuration) for setup details.

## Usage

### Single Probe

```bash
# Test for DAN jailbreak
augustus scan openai.OpenAI \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --config-file config.yaml \
  --verbose
```

### Multiple Probes

```bash
# Use glob patterns to run related probes
augustus scan openai.OpenAI \
  --probes-glob "dan.*,goodside.*,grandma.*" \
  --detectors-glob "*" \
  --config-file config.yaml \
  --output batch-results.jsonl

# Run all probes against Claude
augustus scan anthropic.Anthropic \
  --all \
  --config '{"model":"claude-3-opus-20240229"}' \
  --timeout 60m \
  --output comprehensive-scan.jsonl \
  --html comprehensive-report.html
```

### Buff Transformations

Apply prompt transformations to test evasion techniques:

```bash
# Apply base64 encoding buff to all probes
augustus scan openai.OpenAI \
  --all \
  --buff encoding.Base64 \
  --config '{"model":"gpt-4"}'

# Apply poetry transformation
augustus scan anthropic.Anthropic \
  --probes-glob "dan.*" \
  --buff poetry.MetaPrompt \
  --config '{"model":"claude-3-opus-20240229"}'

# Chain multiple buffs
augustus scan openai.OpenAI \
  --all \
  --buffs-glob "encoding.*,paraphrase.*" \
  --output buffed-results.jsonl
```

### Output Formats

```bash
# Table format (default) - human-readable
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format table

# JSON format - structured output
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format json

# JSONL format - one JSON object per line, ideal for piping
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format jsonl

# HTML report - visual reports for stakeholders
augustus scan openai.OpenAI --all --html report.html
```

### Custom REST Endpoints

```bash
# Test proprietary LLM endpoint (OpenAI-compatible API)
augustus scan rest.Rest \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --config '{
    "uri": "https://api.example.com/v1/chat/completions",
    "method": "POST",
    "headers": {"Authorization": "Bearer YOUR_API_KEY"},
    "req_template_json_object": {
      "model": "custom-model",
      "messages": [{"role": "user", "content": "$INPUT"}]
    },
    "response_json": true,
    "response_json_field": "$.choices[0].message.content"
  }'

# Test with proxy interception (Burp Suite, mitmproxy)
augustus scan rest.Rest \
  --probes-glob "goodside.*" \
  --config '{
    "uri": "https://internal-llm.corp/generate",
    "proxy": "http://127.0.0.1:8080",
    "headers": {"X-API-Key": "$KEY"},
    "api_key": "your-key-here",
    "req_template": "{\"prompt\":\"$INPUT\",\"max_tokens\":500}",
    "response_json": true,
    "response_json_field": "output"
  }'
```

**REST Configuration Keys:**
- `uri`: Target API endpoint (required)
- `method`: HTTP method (default: POST)
- `headers`: HTTP headers as key-value pairs
- `req_template`: Raw request body with `$INPUT` placeholder
- `req_template_json_object`: JSON request body (auto-marshaled, use `$INPUT` in strings)
- `response_json`: Parse response as JSON (default: false)
- `response_json_field`: JSONPath to extract (e.g., `$.data.text` or simple field name)
- `api_key`: API key for `$KEY` placeholder substitution
- `proxy`: HTTP proxy URL for traffic inspection

### Advanced Options

```bash
# Adjust concurrency (default: 10)
augustus scan openai.OpenAI --all --concurrency 20

# Increase timeout for complex probes like TAP or PAIR
augustus scan openai.OpenAI --probe tap.TAPv1 --timeout 60m

# Use a specific harness strategy
augustus scan openai.OpenAI --all --harness batch.Batch

# Test local model with Ollama (no API key needed)
augustus scan ollama.OllamaChat \
  --probe dan.Dan_11_0 \
  --config '{"model":"llama3.2:3b"}'
```

## How It Works

Augustus uses a pipeline architecture to test LLMs against adversarial attacks:

```mermaid
flowchart LR
    A[Probe Selection] --> B[Buff Transform]
    B --> C[Generator / LLM Call]
    C --> D[Detector Analysis]
    D --> E{Vulnerable?}
    E -->|Yes| F[Record Finding]
    E -->|No| G[Record Pass]

subgraph Scanner
        B
        C
        D
        E
    end
```

### Scan Pipeline

1. **Probe Selection**: Choose probes by name, glob pattern, or `--all`
2. **Buff Transformation**: Optionally transform prompts (encode, paraphrase, translate, poeticize)
3. **Generator Call**: Send adversarial prompts to the target LLM via its provider integration
4. **Detector Analysis**: Analyze responses using pattern matching, LLM-as-a-judge, or specialized detectors
5. **Result Recording**: Score each attempt and produce output in the requested format
6. **Attack Engine**: For iterative probes (PAIR, TAP), the single-turn attack engine refines prompts across iterations with candidate pruning and judge-based scoring
7. **Multi-Turn Engine**: For conversational probes (Crescendo, GOAT), the multi-turn engine maintains full conversation history with the target across turns, with refusal detection and dynamic adaptation

### Multi-Turn Attack Strategies

Multi-turn attacks maintain a persistent conversation with the target LLM, exploiting the fact that models may disclose information incrementally across turns that they would refuse in a single prompt. The multi-turn engine uses three LLMs: an **attacker** (generates questions), a **target** (the system under test), and a **judge** (scores progress and detects refusals).

#### Crescendo

Crescendo uses gradual escalation (foot-in-the-door technique) to slowly shift a conversation from benign topics toward a prohibited objective.

- **Paper**: [Russinovich et al., 2024](https://arxiv.org/abs/2404.01833)
- **Approach**: Starts with genuinely benign, educational questions and incrementally increases specificity over many turns
- **Escalation pattern**: Historical context → technical mechanisms → specific details → direct requests framed as natural follow-ups
- **Strength**: Effective against models that track conversation tone — the gradual shift avoids triggering safety filters

```bash
augustus scan rest.Rest \
  --probe crescendo.Crescendo \
  --config-file crescendo.yaml \
  --html report.html -v
```

#### GOAT (Generative Offensive Agent Tester)

GOAT uses an aggressive, adaptive approach with 7 adversarial techniques and Chain-of-Attack-Thought reasoning to dynamically switch strategies based on what works or fails.

- **Paper**: [Pavlova et al., 2024](https://arxiv.org/abs/2410.01606)
- **Approach**: Targets the objective from turn 1 using indirect framing, never mentions the goal directly to the target
- **7 techniques** across 3 categories:
  - *Output Manipulation*: Refusal Suppression, Response Priming
  - *Safe Response Distractors*: Dual Response, Topic Splitting, Opposite Intent
  - *Fictional Scenarios*: Persona Modification, Hypothetical
- **Chain-of-Attack-Thought**: Each turn the attacker reasons through Observation → Thought → Strategy → Response before crafting its message
- **Technique stacking**: Multiple techniques can be combined in a single turn for stronger effect
- **Strength**: Achieves high success rates in fewer turns (typically 3-5) by aggressively switching between fundamentally different approaches

```bash
augustus scan rest.Rest \
  --probe goat.Goat \
  --config-file goat.yaml \
  --html report.html -v
```

#### Hydra

Hydra maintains a single conversation path and rolls back entire turns when the target refuses, asking the attacker for a completely different approach. Unlike Crescendo/GOAT (which rephrase on refusal), Hydra's backtracking completely removes refused turns from the target's view.

- **Approach**: Single-path with turn-level backtracking — refused turns are erased and replaced
- **Techniques**: Decomposition (break goal into innocent sub-questions), context leveraging (build on target's own words), authority simulation, emotional framing, progressive normalization
- **Key feature**: `max_backtracks` controls how many times Hydra can erase and retry a turn
- **Stateful mode**: Set `stateful: true` for targets where messages cannot be unsent (disables backtracking)
- **Strength**: Keeps the target's conversation history clean — the target never sees failed approaches, preventing defensive escalation

```bash
augustus scan rest.Rest \
  --probe hydra.Hydra \
  --config-file hydra.yaml \
  --html report.html -v
```

#### Mischievous User

The Mischievous User simulates an innocent, curious person who subtly probes AI boundaries through natural conversation. Unlike adversarial strategies, the attacker persona is a casual user who drifts toward prohibited topics through seemingly innocent engagement.

- **Inspired by**: [Tau-bench](https://github.com/sierra-research/tau-bench) and promptfoo's mischievous-user strategy
- **Approach**: Rapport-building phase → boundary probing → assertive pushing
- **7 techniques**: Naive curiosity, deliberate misunderstanding, social proof, gradual drift, emotional appeal, assumed permission, recontextualization
- **Goal-specific playbooks**: Built-in scripts for extracting system instructions, finding secrets/flags, and bypassing content policy
- **Default turns**: 5 (fewer than other strategies — the subtle approach either works quickly or not at all)
- **Strength**: Effective against models trained to resist obvious adversarial patterns — the casual persona bypasses "attack detection" heuristics

```bash
augustus scan rest.Rest \
  --probe mischievous.MischievousUser \
  --config-file mischievous.yaml \
  --html report.html -v
```

#### Strategy Selection Guide

| Strategy | Turns | Style | Best For | Cost |
|----------|-------|-------|----------|------|
| `crescendo.Crescendo` | 10 | Gradual escalation | Models with tone-tracking safety filters | Medium (10 turns × 3 LLM calls) |
| `goat.Goat` | 10 | Aggressive technique switching | Quick results, models weak to framing tricks | Medium-High (technique reasoning overhead) |
| `hydra.Hydra` | 10 | Backtracking on refusal | Stubborn models that refuse frequently | High (backtracking adds retries) |
| `mischievous.MischievousUser` | 5 | Casual persona probing | Models trained against adversarial patterns | Low (5 turns, subtle approach) |

**Quick decision**:
- Start with **Crescendo** — it's the most general-purpose strategy
- Try **GOAT** if Crescendo is too slow (GOAT typically succeeds in 3-5 turns)
- Use **Hydra** if the target refuses frequently (backtracking keeps conversation clean)
- Use **Mischievous** for targets with strong adversarial-pattern detection (the persona bypasses heuristics)

#### Multi-Turn Configuration

All four strategies share the same configuration structure:

```yaml
generators:
  rest.Rest:
    uri: "https://your-llm-endpoint/chat"
    method: POST
    req_template: '{"input": "$INPUT"}'
    response_json: true
    response_json_field: "response"

judge:
  generator_type: openai.OpenAI
  config:
    model: gpt-4
    api_key: "${OPENAI_API_KEY}"

probes:
  attacker_generator_type: openai.OpenAI
  attacker_config:
    model: gpt-4
    api_key: "${OPENAI_API_KEY}"
  settings:
    crescendo.Crescendo:  # or goat.Goat, hydra.Hydra, mischievous.MischievousUser
      goal: "your test objective"
      max_turns: 10
      success_threshold: 0.8
      use_secondary_judge: true
```

| Setting | Default | Description |
|---------|---------|-------------|
| `goal` | (required) | The objective the attacker tries to achieve |
| `max_turns` | 10 | Maximum conversation turns before stopping |
| `success_threshold` | 0.8 | Judge score (0.0-1.0) that triggers early exit |
| `max_refusal_retries` | 10 | Rephrase attempts per turn when target refuses |
| `attack_max_attempts` | 5 | Retries for attacker LLM JSON parsing failures |
| `use_secondary_judge` | true | Enable secondary judge to catch false negatives |
| `max_backtracks` | 10 | Turn-level rollbacks on refusal (Hydra only) |
| `enable_fast_refusal` | true | Pattern-based refusal detection before LLM judge call |
| `enable_scan_memory` | false | Cross-test-case learning (shares tactics across probes) |
| `stateful` | false | Disable backtracking for stateful targets |
| `exclude_target_output` | false | Hide target responses from attacker feedback (privacy mode) |
| `attacker_model` | (auto) | Override attacker model name for context window sizing |

#### Multi-Turn Troubleshooting

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| `no turns completed (attacker_parse_failures=N)` | Attacker LLM returning invalid JSON | Use a stronger attacker model (GPT-4, Claude Opus). Increase `attack_max_attempts`. |
| `no turns completed (target_empty=N)` | Target returning empty/null responses | Check target endpoint is responding. Verify REST config template. |
| All turns score 0.0 | Goal too vague or attacker not engaging | Make `goal` more specific. Try different strategy. |
| High scores but no success | `success_threshold` too high | Lower `success_threshold` from 0.8 to 0.6-0.7 |
| Runs too long / expensive | Too many turns and retries | Reduce `max_turns` (try 5). Set `enable_fast_refusal: true`. |
| Hydra keeps backtracking | Target refuses everything | Try `stateful: true` or switch to Mischievous strategy |

## Architecture

```
cmd/augustus/          CLI entrypoint (Kong-based)
pkg/
  attempt/            Probe execution lifecycle and result tracking
  buffs/              Buff interface for prompt transformations
  config/             Configuration loading (YAML/JSON) with profiles
  detectors/          Public detector interfaces and registry
  generators/         Public generator interfaces and registry
  harnesses/          Harness interface for execution strategies
  lib/http/           Shared HTTP client with proxy support
  lib/stego/          LSB steganography for multimodal attacks
  logging/            Structured slog-based logging
  metrics/            Prometheus metrics collection
  prefilter/          Aho-Corasick keyword pre-filtering
  probes/             Public probe interfaces and registry
  ratelimit/          Token bucket rate limiting
  registry/           Generic capability registration system
  results/            Result types and multi-format output
  retry/              Exponential backoff with jitter
  scanner/            Scanner orchestration with concurrency
  templates/          YAML probe template loader (Nuclei-style)
  types/              Canonical shared interfaces (Prober, Generator, Detector)
internal/
  probes/             210+ probe implementations (47 categories)
  generators/         28 LLM provider integrations (43 variants)
  detectors/          90+ detector implementations (35 categories)
  harnesses/          3 harness strategies (probewise, batch, agentwise)
  buffs/              Buff interface for prompt transformations
  attackengine/       Iterative adversarial attack engine (PAIR/TAP backend)
  multiturn/          Multi-turn conversational attack engine (Crescendo/GOAT/Hydra/Mischievous)
  ahocorasick/        Internal Aho-Corasick keyword matching
benchmarks/           Performance benchmarks
tests/                Integration and equivalence tests
research/             Research documentation and analysis
examples/             Example configurations
docs/                 Documentation
```

### Key Design Decisions

- **Concurrent scanning** with bounded goroutine pools via `errgroup`
- **Plugin-style registration** using Go `init()` functions for probes, generators, detectors, buffs, and harnesses
- **Iterative attack engine** with multi-stream conversation management, candidate pruning, and judge-based scoring for PAIR/TAP
- **Multi-turn attack engine** with persistent conversation history, refusal detection, strategy-agnostic design for Crescendo/GOAT
- **YAML probe templates** (Nuclei-style) for declarative probe definitions alongside Go-based probes
- **Aho-Corasick pre-filtering** for fast keyword matching in detectors

## Configuration

### YAML Configuration File

Create a `config.yaml` file:

```yaml
# Runtime configuration
run:
  max_attempts: 3
  timeout: "30s"

# Generator configurations
generators:
  openai.OpenAI:
    model: "gpt-4"
    temperature: 0.7
    api_key: "${OPENAI_API_KEY}"  # Environment variable interpolation

anthropic.Anthropic:
    model: "claude-3-opus-20240229"
    temperature: 0.5
    api_key: "${ANTHROPIC_API_KEY}"

ollama.OllamaChat:
    model: "llama3.2:3b"
    temperature: 0.8

# Judge configuration (required for judge.Judge, judge.Refusal, and multi-turn probes)
judge:
  generator_type: openai.OpenAI
  model: gpt-4o-mini
  config:
    api_key: "${OPENAI_API_KEY}"

# Output configuration
output:
  format: "jsonl"
  path: "./results.jsonl"

# Named profiles for different scenarios
profiles:
  quick:
    run:
      max_attempts: 1
      timeout: "10s"
    generators:
      openai.OpenAI:
        model: "gpt-3.5-turbo"
        temperature: 0.5
    output:
      format: "table"

thorough:
    run:
      max_attempts: 5
      timeout: "60s"
    generators:
      openai.OpenAI:
        model: "gpt-4"
        temperature: 0.3
    output:
      format: "jsonl"
      path: "./thorough_results.jsonl"
```

### Environment Variables

```bash
# API Keys
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export COHERE_API_KEY="..."

# Debug mode
export AUGUSTUS_DEBUG=true
```

### Proxy Configuration

Route HTTP traffic through a proxy (e.g., Burp Suite) for inspection:

```bash
# Method 1: Via config parameter
augustus scan rest.Rest \
  --probe dan.Dan_11_0 \
  --detector dan.DAN \
  --config '{"uri":"https://api.example.com","proxy":"http://127.0.0.1:8080"}' \
  --output results.jsonl

# Method 2: Via environment variables
export HTTP_PROXY=http://127.0.0.1:8080
export HTTPS_PROXY=http://127.0.0.1:8080
augustus scan rest.Rest --probe dan.Dan_11_0 --config '{"uri":"https://api.example.com"}'
```

- TLS verification automatically disabled for proxy inspection
- HTTP/2 support enabled for modern APIs
- Server-Sent Events (SSE) responses automatically detected and parsed

### CLI Reference

```
Usage: augustus scan <generator> [flags]

Arguments:
  <generator>                 Generator name (e.g., openai.OpenAI, anthropic.Anthropic)

Probe Selection (choose one):
  --probe, -p                 Probe name (repeatable)
  --probes-glob               Comma-separated glob patterns (e.g., "dan.*,goodside.*")
  --all                       Run all registered probes

Detector Selection:
  --detector                  Detector name (repeatable)
  --detectors-glob            Comma-separated glob patterns

Buff Selection:
  --buff, -b                  Buff names to apply (repeatable)
  --buffs-glob                Comma-separated buff glob patterns (e.g., "encoding.*")

Configuration:
  --config-file               Path to YAML config file
  --config, -c                JSON config for generator

Execution:
  --harness                   Harness name (default: probewise.Probewise)
  --timeout                   Overall scan timeout (default: 30m)
  --probe-timeout             Per-probe timeout (default: 5m)
  --concurrency               Max concurrent probes (default: 10, env: AUGUSTUS_CONCURRENCY)

Output:
  --format, -f                Output format: table, json, jsonl (default: table)
  --output, -o                JSONL output file path
  --html                      HTML report file path
  --verbose, -v               Verbose output

Global:
  --debug, -d                 Enable debug mode
```

**Commands:**

```bash
augustus version              # Print version information
augustus list                 # List available probes, detectors, generators, harnesses, buffs
augustus scan <generator>     # Run vulnerability scan
augustus completion <shell>   # Generate shell completion (bash, zsh, fish)
```

**Exit Codes:**

| Code | Meaning |
|------|---------|
| 0 | Success - scan completed |
| 1 | Scan/runtime error |
| 2 | Validation/usage error |

## FAQ

### How does Augustus compare to garak?

Augustus is a Go-native reimplementation inspired by [garak](https://github.com/NVIDIA/garak) (NVIDIA's Python-based LLM vulnerability scanner). Key differences:
- **Performance**: Go binary vs Python interpreter — faster execution and lower memory usage
- **Distribution**: Single binary with no runtime dependencies vs Python package with pip install
- **Concurrency**: Go goroutine pools (cross-probe parallelism) vs Python multiprocessing pools (within-probe parallelism)
- **Probe coverage**: Augustus has 210+ probes; garak has 160+ probes with a longer research pedigree and published paper (arXiv:2406.11036)
- **Provider coverage**: Augustus has 28 providers; garak has 35+ generator variants across 22 provider modules

### Can I test local models without API keys?

Yes! Use the Ollama integration for local model testing:

```bash
# No API key needed
augustus scan ollama.OllamaChat \
  --probe dan.Dan_11_0 \
  --config '{"model":"llama3.2:3b"}'
```

### How do I add custom probes?

1. Create a new Go file in `internal/probes/`
2. Implement the `probes.Probe` interface
3. Register using `registry.RegisterProbe()` in an `init()` function
4. Rebuild: `make build`

See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed instructions.

### What output formats are supported?

Augustus supports four output formats:

| Format | Flag | Use Case |
|--------|------|----------|
| Table | `--format table` | Human-readable terminal output |
| JSON | `--format json` | Single JSON object for parsing |
| JSONL | `--format jsonl` | Line-delimited JSON for streaming |
| HTML | `--html report.html` | Visual reports for stakeholders |

### How do I test multiple models at once?

```bash
# Test multiple models sequentially
for model in "gpt-4" "gpt-3.5-turbo"; do
  augustus scan openai.OpenAI \
    --all \
    --config "{\"model\":\"$model\"}" \
    --output "results-$model.jsonl"
done
```

### Is Augustus suitable for production environments?

Yes, Augustus is designed for production use with:
- Concurrent scanning with configurable limits
- Rate limiting to respect API quotas
- Timeout handling for long-running probes
- Retry logic for transient failures
- Structured logging for observability

## Troubleshooting

### Error: "API rate limit exceeded"

**Cause**: Too many concurrent requests or requests per minute.

**Solutions**:
1. Reduce concurrency: `--concurrency 5`
2. Use provider-specific rate limit settings in YAML config:
   ```yaml
   generators:
     openai.OpenAI:
       rate_limit: 10  # requests per minute
   ```

### Error: "context deadline exceeded" or "timeout"

**Cause**: Complex probes (like TAP or PAIR) exceed default timeout.

**Solution**:
```bash
augustus scan openai.OpenAI \
  --probe tap.TAPv1 \
  --timeout 60m \
  --config-file config.yaml
```

### Error: "invalid API key" or "authentication failed"

**Cause**: Missing or invalid API credentials.

**Solutions**:
1. Verify environment variable is set: `echo $OPENAI_API_KEY`
2. Check for typos in config file
3. Ensure API key has required permissions
4. For Ollama, ensure the service is running: `ollama serve`

### Error: "probe not found" or "detector not found"

**Cause**: Typo in name or probe not registered.

**Solution**:
```bash
# List all available probes and detectors
augustus list

# Use exact names from the list
augustus scan openai.OpenAI --probe dan.Dan_11_0  # Correct
```

### Scan produces no results

**Cause**: Detector didn't match any responses, or output not written.

**Solutions**:
1. Run with `--verbose` to see detailed output
2. Check that detector matches probe type
3. Verify output file path is writable

## Contributing

We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for:

- Adding new vulnerability probes
- Creating new detector implementations
- Adding LLM provider integrations
- Testing guidelines
- Code style requirements

### Development

```bash
# Run all tests
make test

# Run specific package tests
go test ./pkg/scanner -v

# Run equivalence tests (compare Go vs Python implementations)
go test ./tests/equivalence -v

# Build binary
make build

# Install to $GOPATH/bin
make install
```

### Benchmark Environment (DevPod)

A ready-to-go cloud development environment for benchmarking LLMs is available via [DevPod](https://devpod.sh/). It provisions a remote container with Augustus, Ollama, Go, and all dependencies pre-installed.

```bash
cd devpod

# CPU-only instance (~$0.08/hr) - cloud APIs only
make devpod-up-cpu

# GPU instance with NVIDIA T4 (~$0.53/hr) - local models up to 14B
make devpod-up-gpu

# GPU Pro instance with NVIDIA L4 (~$0.80/hr) - local models up to 32B
make devpod-up-gpu-pro
```

Inside the devpod:

```bash
devpod/scripts/setup.sh        # Configure LLM provider API keys
devpod/scripts/pull-models.sh   # Pull local Ollama models (GPU only)
devpod/scripts/benchmark.sh     # Run benchmarks with comparison reports
```

The environment also works as a standard [dev container](https://containers.dev/) — open the repo in VS Code or Cursor and select the CPU or GPU configuration from `.devcontainer/`.

## Security

Augustus is designed for **authorized security testing only**.

- Augustus sends adversarial prompts to LLMs you specify - always ensure you have authorization
- Never test systems you don't own or have explicit permission to test
- Some probes generate offensive content by design (for testing safety filters)
- Results may contain harmful content produced by target LLMs

Report security issues via [GitHub Issues](https://github.com/praetorian-inc/augustus/issues).

## Support

If you find Augustus useful, please consider:

- Giving it a **star** on GitHub
- [Opening an issue](https://github.com/praetorian-inc/augustus/issues) for bugs or feature requests
- [Contributing](CONTRIBUTING.md) new probes, detectors, or provider integrations

[![Star History Chart](https://api.star-history.com/svg?repos=praetorian-inc/augustus&type=Date)](https://star-history.com/#praetorian-inc/augustus&Date)

## License

[Apache 2.0](LICENSE) - Praetorian Security, Inc.

---

**Built by [Praetorian](https://www.praetorian.com/)** - Offensive Security Solutions

augustus

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README