KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #typescript #dsh #dsh-plugin #ai-agent #codex #ai-agents #deepseek-harness #claude-code #cli #react #mcp

vespasian

★ 123 Open GitHub ↗

API discovery tool that maps attack surfaces from captured traffic and generates specs for REST, GraphQL, SOAP, and WebSocket APIs

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Discover API endpoints from real HTTP traffic. Generate OpenAPI, GraphQL SDL, and WSDL specs automatically.

---

# Vespasian: API Discovery and Specification Generation Tool

**Vespasian discovers API endpoints by observing real HTTP traffic and generates API specification files from those observations.** It captures traffic through headless browser crawling or imports it from existing sources (Burp Suite XML exports, HAR files, and mitmproxy dumps), then classifies requests, probes discovered endpoints, and outputs specifications in the native format for each API type: OpenAPI 3.0 for REST, GraphQL SDL for GraphQL, and WSDL for SOAP services.

Built for penetration testers and security engineers who need to map the API attack surface of web applications, single-page apps, and microservices when the API documentation is not available.

## Why Vespasian?

Modern applications make API calls dynamically. Single-page applications construct requests at runtime via JavaScript. Mobile apps call APIs through native HTTP clients. Real-time features communicate over WebSocket connections. Static analysis and source code review miss these runtime behaviors entirely.

Existing approaches to API discovery have limitations:

- **Checking known paths** (`/swagger.json`, `/openapi.yaml`) only finds APIs that are explicitly documented
- **Static analysis** cannot observe requests that are constructed dynamically at runtime
- **Manual proxy capture** is time-consuming and produces raw traffic without structured specifications

Vespasian takes a different approach: it observes actual network traffic at the wire level, then uses classification heuristics and active probing to produce structured API specifications automatically. Because this is inherently probabilistic, Vespasian discovers only the endpoints present in the captured traffic, but it reliably maps the API surface that an application actually exposes during use.

## Key Features

| Feature | Description |
|---------|-------------|
| **REST API Discovery** | Classifies REST endpoints via content-type, path patterns, and response structure; outputs OpenAPI 3.0 |
| **GraphQL API Discovery** | Detects GraphQL endpoints, runs tiered introspection queries, and generates GraphQL SDL schemas |
| **WSDL/SOAP Discovery** | Identifies SOAP services via SOAPAction headers and envelope detection; fetches and parses WSDL documents |
| **gRPC API Discovery** | Classifies gRPC and gRPC-Web traffic via content-type, trailer headers, and path shape; enumerates services and methods through the Server Reflection Protocol — with reflection-off fallbacks (grpc-gateway/Envoy OpenAPI scrape and gRPC-Web JS-binding recovery) — and generates `.proto` schemas |
| **API Type Auto-Detection** | Automatically determines API type (REST, GraphQL, WSDL) from captured traffic without manual selection. gRPC is opt-in via `--api-type grpc` — its binary HTTP/2 framing is not auto-detected |
| **Browser Crawling** | Two backends: headless mode drives Chrome via [go-rod](https://github.com/go-rod/rod) for full JavaScript/SPA support; non-headless mode uses a stdlib net/http engine (DFS, 150 rps, scope+SSRF redirect guard) for lightweight crawls. The headless backend uses a configured or **system Chrome** by default and does not download a browser from third-party mirrors unless explicitly opted in (supply-chain hardening); it also disables Chrome telemetry so crawl egress stays minimal |
| **SPA Bundle Extraction** | Post-crawl pass that scans JavaScript bundles for API path strings and probes them with raw HTTP, recovering endpoints the headless browser could not exercise |
| **Static Form Extraction** | Statically parses `<form>` elements in captured HTML responses — including login, search, and admin forms — to surface submission endpoints and parameters that dynamic crawling may never trigger |
| **Traffic Import** | Import existing captures from Burp Suite XML, HAR 1.2 files, and mitmproxy dumps |
| **Active Probing** | OPTIONS discovery, JSON schema inference, WSDL document fetching, GraphQL introspection, and gRPC server reflection. Every probe target is gated to the scan's own origin by default — a classified candidate whose URL doesn't share the scan target's scheme/host/port (e.g. an attacker-controlled absolute URL recovered from a JS bundle) is skipped rather than probed; internal-only, no CLI flag, same policy as SPA Bundle Extraction's cross-origin gate below |
| **Path Normalization** | `/users/42` and `/users/87` become `/users/{id}` with known literal preservation (`/me`, `/self`) |
| **SSRF Protection** | Blocks crawling and probing of private and loopback addresses by default. Pass `--dangerous-allow-private` to test internal targets (localhost, 127.0.0.1, RFC1918, link-local); the flag is required when the seed URL is itself a private host. |
| **JS Bundle Static Analysis** | Statically analyses captured JavaScript bundles to recover API endpoints, path parameters, and request-body fields missed by dynamic crawling. Also reconstructs concat / `+`-chain / literal service-prefix forms (e.g. `"/api/posts/".concat(id, "/comment")`, `"identity/" + "api/auth/login"`) fully offline as unprobed candidates, with no network access required — non-literal operands become a numeric sentinel, parameterized downstream. Enabled by default via `--analyze-js`; sourcemap recovery is controlled by `--fetch-sourcemaps` (default: `true` for `scan`/`crawl`, `false` for `generate`). |
| **Proxy Support** | Route traffic through Burp Suite or other intercepting proxies; http/https/socks5. Covers the crawl stage on both backends (headless Chrome and `--headless=false` net/http) plus the post-crawl active stages — probe (OPTIONS/schema/WSDL-fetch/GraphQL introspection/gRPC reflection), JS-replay, and jsstatic sourcemap fetches. socks5 always verifies TLS; `--proxy-insecure` (http/https only) accepts a MITM proxy's certificate. |
| **Two-Stage Pipeline** | Capture once, generate many: separate capture and generation steps for maximum flexibility |

## How It Works

Vespasian uses a two-stage pipeline that separates traffic capture from specification generation:

```mermaid
flowchart LR
 subgraph Capture
 A["Crawler headless go-rod or net/http"] --> C["capture.json ObservedRequest array"]
 B["Traffic Importers Burp Suite XML, HAR, mitmproxy"] --> C
 end
 subgraph Generate
 C --> S["Static Analyzer HTML form extraction"]
 S --> D["Classifier REST, GraphQL, WSDL, gRPC"]
 D --> E["Prober OPTIONS, schema, WSDL, introspection, gRPC reflection"]
 E --> F["Spec Generator OpenAPI 3.0, GraphQL SDL, WSDL, .proto"]
 end
```

**Why two stages:**

- **Capture once, generate many.** Run different generators against the same capture without re-scanning.
- **Debuggable.** The capture file is inspectable JSON, isolating capture bugs from generation bugs.
- **Composable.** Import traffic from any source (browser crawls, proxy captures, mobile testing).
- **Offline analysis.** Generate specifications without network access, useful during limited engagement windows.

> **Breaking change:** The `query_params` field of `capture.json` is now `map[string][]string` (multi-value). Capture files generated by previous versions must be regenerated.

### SPA Bundle Extraction

Many single-page applications bundle their API paths as string literals inside
JavaScript files (`/api/v2/users`, `` `/api/items/${id}` ``,
`"identity/" + "api/auth/login"`). The headless browser only exercises the
paths the user actually clicks on; the rest stay hidden. After the crawl,
Vespasian re-scans every captured JavaScript bundle for API-path patterns
and probes the discovered URLs with raw HTTP requests. Wrong combinations
(typically caused by the regex matching unrelated string literals) come back
404 and are dropped.

This step runs in both the single-stage `scan` command and the two-stage
`generate` command (gated on `--probe` and `--analyze-js`, both on by
default), so the `crawl` → `generate` workflow recovers these JS-bundle-only
endpoints too. Because replay re-fetches the bundles over HTTP, the target
must still be reachable when `generate` runs; the origin is taken from the
capture. The `crawl` command alone stays passive — it records a browser
capture without running JS-replay.

`generate`'s replay accepts the same `--header`/`-H` flag as `scan`, forwarding
those headers (e.g. `Authorization`) to same-origin bundle fetches and probes so
authenticated SPAs can be recovered in the two-stage flow. Since captures don't
preserve the crawl origin for imported/mixed-origin sources, `--target-url`
overrides the capture-derived origin (which otherwise defaults to the capture's
first HTML page).

The extractor also reconstructs paths built by runtime string concatenation —
both `String.prototype.concat` (`"/api/posts/".concat(id, "/comment")`) and the
`+` operator (`"/api/users/" + uid + "/profile"`). Operands that are not string
literals (identifiers, function calls, expressions) are replaced with a numeric
placeholder, so `"/api/posts/".concat(id, "/comment")` becomes the probeable
path `/api/posts/0/comment`, which the OpenAPI generator then parameterizes to
`/api/posts/{postId}/comment`.

These concat / `+`-chain / service-prefix paths are also recovered by a
second, fully-offline static analysis pass (see "JS Bundle Static Analysis"
above) that needs no reachable target at all — it emits them as unprobed
candidates directly from the captured bundle bytes. Live JS-replay,
described in this section, additionally re-fetches and probes those same
reconstructions over HTTP, drops 404 decoys, and performs a speculative
service-prefix fan-out that the offline pass deliberately omits (fan-out
combinations are only safe once probed and 404-filtered). Live replay is
additive, with one exception: a path the probe answers with a 404 is dropped
as a decoy — other statuses (200, 204, 401/403, 302, ...) never refute an
offline candidate. So pointing `generate` at a reachable target *can* yield
slightly fewer endpoints than running fully offline.

A 404 is the best signal available here, but it is not proof of absence.
Returning 404 rather than 401/403 for a real-but-unauthorized resource is a
widespread anti-enumeration convention, and a hostile target can fingerprint
the probe's `User-Agent` and 404 everything to hide its API surface. Because
an honest server also 404s a path that is genuinely absent, no additional
probe can separate the two cases — telling them apart needs credentials, not
another request. Every dropped path is therefore **named on stderr** along
with both remedies: re-run with `--header` if the endpoint is auth-gated, or
`--probe=false` to keep every offline candidate.

> **Note:** a `capture.json` produced by a build predating this feature already
> carries `static:js` entries, which makes `generate` skip the static pass
> (the idempotency guard that keeps `crawl` → `generate` identical to `scan`).
> Re-run the capture to recover concat/service-prefix endpoints from it.
> Captures from `import` (Burp/HAR/mitmproxy) are unaffected.

By default this step:

- only probes URLs whose origin matches the scan target — cross-origin URLs
  embedded in the bundle are skipped to avoid using Vespasian as a request
  reflector;
- never forwards `--header` values (Authorization, Cookie, ...) to
  cross-origin destinations;
- enforces SSRF protection on every URL unless `--dangerous-allow-private`
  is set;
- caps probe attempts at 500 per scan and total wall-clock time at 10
  minutes for the JS-replay step.

If a discovered SPA bundle is over 1 MB (the default crawl truncation
limit), Vespasian re-fetches it with a 10 MB cap before scanning so paths
embedded after the truncation point are still recovered.

## How to Install Vespasian

### Install from Source (Go)

```bash
go install github.com/praetorian-inc/vespasian/cmd/vespasian@latest
```

### Download Pre-Built Binary

Download the latest binary for your platform from the [Releases](https://github.com/praetorian-inc/vespasian/releases) page.

### Build from Source

```bash
git clone https://github.com/praetorian-inc/vespasian.git
cd vespasian
make build
```

## How to Discover APIs with Vespasian

### Quick Start: Scan a Web Application

```bash
# Crawl and generate an API spec in one step (auto-detects API type)
vespasian scan https://app.example.com -o api.yaml

# With authentication
vespasian scan https://app.example.com -H "Authorization: Bearer <token>" -o api.yaml

# Specify the API type explicitly
vespasian scan https://app.example.com --api-type graphql -o schema.graphql
```

### Two-Stage Workflow

```bash
# Stage 1: Capture traffic via headless browser
vespasian crawl https://app.example.com -o capture.json

# Stage 1 (alternative): Import traffic from Burp Suite
vespasian import burp traffic.xml -o capture.json

# Stage 1 (alternative): Import traffic from HAR archive
vespasian import har recording.har -o capture.json

# Stage 1 (alternative): Import traffic from mitmproxy
vespasian import mitmproxy flows -o capture.json

# Stage 2: Generate OpenAPI spec for REST
vespasian generate rest capture.json -o api.yaml

# Stage 2: Generate GraphQL SDL schema
vespasian generate graphql capture.json -o schema.graphql

# Stage 2: Generate WSDL from SOAP traffic
vespasian generate wsdl capture.json -o service.wsdl
```

### Common Options

```bash
# Route crawl traffic through Burp Suite (headless backend)
vespasian scan https://app.example.com --proxy http://127.0.0.1:8080 -o api.yaml

# Route the dependency-free net/http backend through a proxy (no Chrome needed)
vespasian crawl https://app.example.com --headless=false --proxy http://127.0.0.1:8080 -o capture.json

# Scan a local/private target (bypasses SSRF protection)
vespasian scan http://localhost:3000 --dangerous-allow-private -o api.yaml

# Verbose output to see discovered requests in real-time
vespasian scan https://app.example.com -v -o api.yaml

# Suppress the startup banner
vespasian --no-banner scan https://app.example.com -o api.yaml
```

> **Headless crawling requires a system Chrome/Chromium by default.** The headless
> backend resolves a locally installed browser and does not download one over the
> network unless explicitly opted in. If no system browser is found the crawl
> errors rather than fetching a Chromium from third-party mirrors. On platforms
> without a system Chrome (e.g. arm64
> Linux, where Google Chrome ships no build) either install Chromium, use the
> `--headless=false` net/http backend, or set `VESPASIAN_ALLOW_BROWSER_DOWNLOAD=true`
> to explicitly allow go-rod's managed download.

#### Path normalization & `--merge-slugs`

IDs (numeric, UUID, hashes like `/users/42`) are always normalized to parameters regardless of flags. Observation-based "slug" merging is opt-in via `--merge-slugs` (off by default):

- Feature routes (default): `/feature/login`, `/feature/export` stay distinct.
- Blog/CMS content (use `--merge-slugs`): `/posts/hello-world`, `/posts/my-trip` → `/posts/{postSlug}`

Optionally, when using `--merge-slugs`, you can also configure `--slug-threshold` (default 2) to set how many distinct values must appear at a position before merging paths into a slug; higher is more conservative.

## Use Cases

### Penetration Testing without API Documentation

During authorized security assessments, clients often cannot provide API documentation. Vespasian crawls the target application with a headless browser, captures every API call the frontend makes, and produces specifications that describe the discovered endpoints, parameters, and response schemas.

### Generating API Specs from Existing Proxy Captures

Pentesters already capture traffic in Burp Suite and mitmproxy during manual testing. Rather than re-crawling, Vespasian can import that traffic and generate specifications from work already done. This is especially useful for mobile application testing, where no browser crawl can observe the API calls.

### Mapping API Attack Surface for Web Applications

For attack surface management, Vespasian identifies which API endpoints a web application exposes by executing its JavaScript and intercepting all outbound requests. The resulting specification can feed into further security testing tools that accept OpenAPI, GraphQL SDL, or WSDL input.

### Feeding into Hadrian for Authorization Testing

Generate an API specification with Vespasian, then pass it directly to [Hadrian](https://github.com/praetorian-inc/hadrian) for automated OWASP API Top 10 authorization testing. This creates a complete discover-then-test workflow.

## API Type Support

Vespasian classifies and generates specifications for four API types:

| API Type | Classification Signals | Output Format | Probing |
|----------|----------------------|---------------|---------|
| **REST** | JSON/XML content-type, `/api/` `/v1/` path patterns, HTTP methods | OpenAPI 3.0 (YAML/JSON) | OPTIONS discovery, JSON, urlencoded, and multipart request-body inference |
| **GraphQL** | `/graphql` path, query structure in POST body, `data`/`errors` response keys | GraphQL SDL | Tiered introspection queries (3 tiers for WAF bypass) |
| **WSDL/SOAP** | SOAPAction header, SOAP envelope in body, `?wsdl` URL parameter | WSDL XML | Active `?wsdl` document fetching |
| **gRPC** | `application/grpc`/`application/grpc-web*` content-type, `grpc-status`/`grpc-message` trailers, `/<pkg.Service>/<Method>` POST path | `.proto` (proto3) | Server Reflection Protocol enumeration, with grpc-gateway OpenAPI + gRPC-Web JS-binding fallbacks when reflection is off (`--api-type grpc` only) |

### REST Classification Heuristics

1. **Content-type**: responses with `application/json` or `application/xml`
2. **Static asset exclusion**: drops `.js`, `.css`, `.png`, `.woff`, `/static/`, `/assets/`
3. **Path heuristics**: `/api/`, `/v1/`, `/v2/`, `/v3/`, `/rest/`, `/rpc/` paths boost confidence
4. **HTTP method**: POST/PUT/PATCH/DELETE to non-page URLs
5. **Response structure**: JSON object or array bodies (not HTML)
6. **Request-side signal**: an API path plus a JSON/XML `Accept` (or request content-type) classifies the endpoint even when its response was not captured, so the REST-vs-not verdict does not depend on response timing

The classification signals above are content-based and deterministic: identical input traffic yields the same endpoint set, the same classification, and a byte-identical spec every run. Run with `-v` to see the per-endpoint classification reason.

### GraphQL Classification Heuristics

1. **Path matching**: `/graphql` path (0.70 confidence)
2. **Query structure**: GraphQL query syntax in POST body (0.85 confidence)
3. **Response structure**: `data`/`errors` keys in response (0.80 confidence)
4. **Combined signals**: path + body together (0.95 confidence)

### GraphQL Introspection

Vespasian uses a tiered introspection strategy to handle WAF-protected GraphQL servers:

- **Tier 1**: Full introspection with descriptions, deprecation, and directives
- **Tier 2**: Minimal-complete query without descriptions, deprecation info, or directives
- **Tier 3**: Minimal last-resort query with the smallest payload
- **Fallback**: Traffic-based inference from observed queries and mutations when introspection is disabled

### gRPC Classification Heuristics

gRPC runs over HTTP/2 with binary framing, so it is **opt-in** rather than auto-detected — select it with `--api-type grpc`. The classifier scores observed traffic (e.g. proxy-flattened captures) on:

1. **Content-type**: `application/grpc` or `application/grpc-web*` on the request or response (0.95 confidence)
2. **Trailer headers**: `grpc-status` or `grpc-message` in the response (0.80 confidence)
3. **Path shape**: a `POST` to a `/<pkg.qualified.Service>/<Method>` path (0.60 confidence)
4. **Combined signals**: gRPC content-type **and** trailer together — the HTTP/2-plus-trailers fingerprint — boost confidence to 0.99

### gRPC Server Reflection

The probe enumerates a target's services, methods, and message types via the [gRPC Server Reflection Protocol](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md), then renders the descriptor graph to proto3 source:

- **Version negotiation**: tries reflection `v1`, falling back to `v1alpha` on `Unimplemented`.
- **Schema closure**: walks the transitive `FileDescriptorProto` import graph (the well-known `google/protobuf/*` files are omitted from output since any consumer already has them).
- **Reflection disabled**: when a server is reachable but reflection is off or gated, Vespasian reports a structured reason — `Unimplemented` (not registered), or `Unauthenticated`/`PermissionDenied` (auth-gated) — instead of failing silently.
- **Auth-gated reflection**: the probe does not yet send call credentials/metadata, so servers requiring auth for reflection are *detected and reported* as `Unauthenticated`/`PermissionDenied`, not bypassed.
- **Generator requirement**: `.proto` generation requires reflection descriptors; traffic-only inference is not yet supported, so a reflection-disabled target yields no spec.
- **Partial descriptors**: if the reflection result is missing a transitive import (e.g. a very large import graph truncated at the fetch cap), the generator emits every `.proto` it can still link and lists the omitted files in a `// WARNING:` header, rather than failing the whole generation.
- **SSRF protection**: the dial target is validated before connecting and re-checked at connect time (closing the DNS-rebinding TOCTOU window), the same as the HTTP probe path.
- **TLS targets**: certificates are verified by default. Internal gRPC services often present self-signed or internal-CA certs; to enumerate those, pass `--grpc-insecure-skip-verify` to skip verification (SSRF is still enforced by the dialer regardless). Without the flag, a target whose cert fails verification is not enumerated.

For a deeper reference on gRPC support — classification signals, reflection probe behavior, `.proto` generation, TLS, and end-to-end examples — see [docs/grpc.md](docs/grpc.md). A step-by-step tutorial lives on the [gRPC API Discovery](https://github.com/praetorian-inc/vespasian/wiki/gRPC-API-Discovery) wiki page.

### Reflection-off gRPC discovery

When reflection is disabled, two name-only techniques recover service/method names and streaming flags (but not message fields, which the wire format strips):

- **grpc-gateway OpenAPI**: fetches a bounded set of well-known swagger/OpenAPI paths over HTTP, recognizes grpc-gateway/protoc-gen-openapiv2 documents by their `operationId`/`tags` shape, and maps each operation back to `<Service>/<Method>`. SSRF-gated like the other HTTP probes; use `--dangerous-allow-private` for localhost targets.
- **gRPC-Web JS bindings**: runs static analysis over the JavaScript bundles in the capture to recover generated gRPC-Web/Connect-ES client artifacts (connect-es service objects, `*_pb_service.js` stubs, `*_grpc_web_pb.js` MethodDescriptors). No network access — it reads the captured JS.

Both feed the same generator path as reflection via synthesized `FileDescriptorProto`s (empty message stubs), so they emit byte-identical `.proto` formatting. Under `scan --api-type grpc` all three techniques are chained in precedence order — reflection > gateway > bindings — and the name-only techniques never overwrite richer reflection descriptors.

## CLI Reference

### `vespasian scan`

Convenience command that crawls a target and generates a specification in one step.

```
vespasian scan <url> [flags]
 --api-type API type: auto, rest, graphql, wsdl, grpc (default: auto;
 grpc must be selected explicitly — it is not auto-detected)
 -H, --header Auth headers to inject (repeatable)
 -o, --output Output spec file (default: stdout)
 --depth Max crawl depth (default: 3)
 --max-pages Max pages to visit — counts pages visited, not captured requests (default: 100)
 --timeout Maximum duration for the entire scan (default: 10m)
 --scope same-origin or same-domain (default: same-origin)
 --headless Headless Chrome mode (default: true); --headless=false uses the stdlib net/http engine
 --proxy Proxy URL (e.g., http://127.0.0.1:8080); http/https/socks5.
 Routes crawl, probe (OPTIONS/schema/WSDL-fetch/GraphQL introspection/gRPC
 reflection), JS-replay, and jsstatic sourcemap-fetch traffic through the proxy.
 TLS verification stays on by default (socks5 always verifies). Private targets still
 require --dangerous-allow-private (proxy relaxes only the dial-time SSRF pin, not URL scope).
 --proxy-insecure Disable TLS verification for the net/http stages routed through an http/https
 intercepting proxy (Burp/mitmproxy MITM): the crawl stage (only with
 --headless=false) plus — regardless of --headless — jsstatic sourcemap fetch,
 probe, WSDL discovery, and JS-replay. Off by default. No effect on socks5, on the
 gRPC target's certificate verification (governed solely by
 --grpc-insecure-skip-verify) — though for an https-scheme proxy --proxy-insecure
 DOES disable verification of the proxy's own certificate on that dial's CONNECT
 leg — or on the headless backend (trust the proxy CA via the OS store).
 --confidence Min classification confidence (default: 0.5)
 --probe Enable active probing (default: true)
 --deduplicate Deduplicate endpoints before probing (default: true)
 --dangerous-allow-private Disable SSRF protection for crawling and probes,
 allowing private/localhost targets (localhost, 127.0.0.1,
 RFC1918, link-local). Required when the seed URL is a
 private host, otherwise the crawl exits with an error and
 captures nothing. WARNING: Do not use on production
 systems.
 --no-request-id Disable auto X-Vespasian-Request-Id header
 -v, --verbose Show requests in real-time
```

### `vespasian crawl`

Captures HTTP traffic from the target application. By default it drives a headless Chrome browser (go-rod) for full JavaScript/SPA support; with `--headless=false` it uses a dependency-free stdlib net/http engine (no Chrome required).

```
vespasian crawl <url> [flags]
 -H, --header Auth headers to inject (repeatable)
 -o, --output Capture output file (default: stdout)
 --depth Max crawl depth (default: 3)
 --max-pages Max pages to visit — counts pages visited, not captured requests (default: 100)
 --timeout Maximum duration for the entire crawl (default: 10m)
 --scope same-origin or same-domain (default: same-origin)
 --headless Headless Chrome mode (default: true); --headless=false uses the stdlib net/http engine
 --proxy Proxy URL (e.g., http://127.0.0.1:8080); http/https/socks5.
 Routes crawl traffic on both backends plus jsstatic sourcemap fetches through the
 proxy (socks5 always verifies TLS). TLS verification stays on by default. Private
 targets still require --dangerous-allow-private (proxy relaxes only the dial-time
 SSRF pin, not URL scope).
 --proxy-insecure Disable TLS verification for the net/http stages routed through an http/https
 intercepting proxy (Burp/mitmproxy MITM): the crawl stage (only with
 --headless=false) plus jsstatic sourcemap fetch (regardless of --headless). Off
 by default. No effect on socks5 or the headless backend (trust the proxy CA via
 the OS store).
 --dangerous-allow-private Disable SSRF protection for crawling, allowing
 private/localhost targets (localhost, 127.0.0.1, RFC1918,
 link-local). Required when the seed URL is a private
 host, otherwise the crawl exits with an error and
 captures nothing. WARNING: Do not use on production
 systems.
 --no-request-id Disable auto X-Vespasian-Request-Id header
 -v, --verbose Show requests in real-time
```

### `vespasian import`

Converts traffic captures from external tools and formats into the Vespasian capture format.

```
vespasian import <format> <file> [flags]
 Formats: burp, har, mitmproxy
 -o, --output Capture output file (default: stdout)
 -v, --verbose Show imported requests
```

### `vespasian generate`

Produces an API specification from a capture file.

```
vespasian generate <api-type> <capture-file> [flags]
 API types: rest, graphql, wsdl, grpc
 (grpc emits .proto and requires reflection descriptors in the
 capture; traffic-only inference is unsupported)
 -o, --output Output file (default: stdout)
 --confidence Min classification confidence (default: 0.5)
 --probe Enable active probing (default: true)
 --deduplicate Deduplicate endpoints before probing (default: true)
 --proxy Proxy URL (e.g., http://127.0.0.1:8080); http/https/socks5.
 Routes the probe (OPTIONS/schema/WSDL-fetch/GraphQL introspection/gRPC
 reflection), JS-replay, and jsstatic sourcemap-fetch traffic through the proxy.
 TLS verification stays on by default (socks5 always verifies). Private targets still
 require --dangerous-allow-private (proxy relaxes only the dial-time SSRF pin, not URL scope).
 --proxy-insecure Disable TLS verification for the net/http stages routed through an http/https
 intercepting proxy (Burp/mitmproxy MITM): probe, JS-replay, and jsstatic sourcemap
 fetch. Off by default. No effect on socks5 or on the gRPC target's certificate
 verification (governed solely by --grpc-insecure-skip-verify) — though for an
 https-scheme proxy --proxy-insecure DOES disable verification of the proxy's own
 certificate on that dial's CONNECT leg.
 --dangerous-allow-private Disable SSRF protection on the probe path
 (OPTIONS/schema/WSDL-fetch/GraphQL introspection) for
 private/localhost targets. WARNING: Do not use on
 production systems.
 -v, --verbose Show discovered endpoints
```

## Architecture

### Pipeline Components

| Component | Purpose | Supported Types |
|-----------|---------|-----------------|
| **Crawler** | Two backends: go-rod headless Chrome (JavaScript/SPA support) and stdlib net/http (lightweight, DFS, 150 rps, SSRF guard) | Protocol-agnostic |
| **Importers** | Convert Burp Suite XML, HAR, and mitmproxy traffic to capture format | All three formats |
| **Classifier** | Separates API calls from static assets using heuristics | REST, GraphQL, WSDL, gRPC |
| **Prober** | Enriches endpoints via active requests | OPTIONS, JSON schema, WSDL fetch, GraphQL introspection, gRPC reflection |
| **Generator** | Produces specification files from classified and probed traffic | OpenAPI 3.0, GraphQL SDL, WSDL, `.proto` |

### Package Layout

```
cmd/vespasian/          CLI entry point
internal/pipeline/      Shared crawl/classify/probe/generate orchestration (CLI + SDK)
pkg/sdk/                capability-sdk Capability adapter (used by Guard hosts)
pkg/crawl/              Crawler (headless go-rod + net/http backends) + capture format
pkg/importer/           Traffic importers (Burp, HAR, mitmproxy)
pkg/analyze/            Static HTML form extraction from captured response bodies
pkg/classify/           API classification (REST, GraphQL, WSDL, gRPC)
pkg/probe/              Endpoint probing (OPTIONS, schema, WSDL, GraphQL introspection, gRPC reflection)
pkg/generate/
  ├── rest/             OpenAPI 3.0 generation, path normalization, schema inference
  ├── graphql/          GraphQL SDL generation, introspection, traffic inference
  ├── wsdl/             WSDL generation, SOAP operation extraction
  └── grpc/             .proto generation from gRPC reflection descriptors
internal/grpcwire/      gRPC length-prefix framing + protobuf wire-format parser
```

For a deeper reference on the crawler — interface, backends, options, SSRF model, and how to add a new backend — see [docs/crawler.md](docs/crawler.md).

## Frequently Asked Questions

### What types of APIs can Vespasian discover?

Vespasian discovers **REST APIs** (generating OpenAPI 3.0 specs), **GraphQL APIs** (generating SDL schemas via introspection or traffic inference), **SOAP/WSDL services** (generating WSDL documents), and **gRPC services** (generating `.proto` schemas via server reflection). It automatically detects REST, GraphQL, and WSDL from captured traffic; gRPC must be selected explicitly with `--api-type grpc` because its binary HTTP/2 framing is not auto-detected.

### How does Vespasian discover gRPC services?

gRPC services are enumerated through the [Server Reflection Protocol](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md): Vespasian asks the server to describe its own services, methods, and message types, then reconstructs a proto3 `.proto` from the returned descriptors. Run it via `--api-type grpc` in the `scan`/`generate` pipeline. If a server has reflection **disabled or auth-gated**, Vespasian reports the gRPC status reason (`Unimplemented`, `Unauthenticated`, `PermissionDenied`) rather than failing silently — but cannot reconstruct the schema, since `.proto` generation currently requires reflection descriptors.

### How is Vespasian different from running a web crawler?

Standard web crawlers follow HTML links and index pages. Vespasian intercepts **all HTTP traffic** from a headless browser, including XHR/fetch API calls, WebSocket upgrades, and dynamically constructed requests that don't appear in HTML. It then classifies those requests by API type and generates structured specifications, not just URL lists.

### Does Vespasian find undocumented APIs?

Vespasian discovers any API endpoint that the application calls during the crawl. If the frontend calls `/api/internal/debug` at runtime, Vespasian will capture and document it, even if it doesn't appear in any published API documentation.

### Can I use Vespasian with traffic I've already captured?

Yes. If you've already captured traffic using Burp Suite, browser dev tools (HAR), or mitmproxy, use `vespasian import` to convert it to the capture format, then `vespasian generate` to produce specifications. No re-crawling needed.

### Does Vespasian handle GraphQL servers that disable introspection?

Yes. Vespasian uses a tiered introspection strategy. If the full introspection query is blocked, it tries progressively simpler queries. If all introspection is disabled, it falls back to inferring the schema from observed queries and mutations in the captured traffic.

### Is it safe to run against production?

Vespasian's crawl stage drives a browser and follows links, which is read-only. The probing stage sends OPTIONS requests, fetches `?wsdl` documents, and runs GraphQL introspection queries, all of which are read-only operations. However, always coordinate with the target owner and prefer staging environments during security assessments.

### Security limitations of --analyze-js on untrusted bundles

When analyzing JavaScript bundles served by an attacker-controlled application, `--analyze-js` carries a bounded resource-exhaustion risk. The underlying tree-sitter/jsluice parser is not context-cancellable: a bundle crafted to hang the parser will keep the per-bundle goroutine alive until the parser returns or the process exits. Per-bundle timeouts (default 5 s, configurable) bound the wait per bundle, but a genuine parser deadlock leaks that goroutine for the lifetime of the process. In the worst case the number of leaked goroutines is the worker concurrency × (1 + N) where N is the number of sourcesContent entries in any recovered sourcemap. For long-running processes or automated pipelines that analyze untrusted targets, the recommended mitigation is process isolation: run vespasian with a wall-clock timeout (`--timeout`) per target so leaked goroutines are bounded by the process lifetime.

## Development

### Prerequisites

- [Go 1.24+](https://go.dev/dl/)
- [golangci-lint](https://golangci-lint.run/welcome/install/)

### Build and Test

```bash
git clone https://github.com/praetorian-inc/vespasian.git
cd vespasian
make build       # Build the binary to bin/vespasian
make test        # Run tests with race detection
make lint        # Run golangci-lint (gocritic, misspell, revive)
make check       # Run all checks (fmt, vet, lint, test)
```

```bash
make coverage    # Generate coverage report
make deps        # Download and tidy modules
make clean       # Remove build artifacts
```

## Contributing

Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, project layout, how to add a classifier, generator, importer, or probe strategy, and the commit and pull request conventions.

In short: fork the repository, create a feature branch, make sure `make check` passes, and open a Pull Request. Note that `make check` reformats as well as validates — it runs `gofmt -s -w .` first, so it may modify your working tree.

This project is governed by our [Code of Conduct](CODE_OF_CONDUCT.md), and [GOVERNANCE.md](GOVERNANCE.md) lists the maintainers and how decisions are made. For questions and support, see [SUPPORT.md](SUPPORT.md). Security vulnerabilities should be reported per [SECURITY.md](SECURITY.md) rather than as public issues.

## License

This project is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details.

## About Praetorian

[Praetorian](https://www.praetorian.com/) is a cybersecurity company that helps organizations secure their most critical assets through offensive security services and the [Praetorian Guard](https://www.praetorian.com/guard) attack surface management platform.

vespasian

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README