gorobots

KO | EN

gitlite — search

#javascript #ai #llm #machine-learning #rust #deep-learning #linux #docker #macos #java #windows #awesome #kubernetes #pytorch

(★ 19)

gorobots - robots.txt recon & path discovery in Go. Structured parsing, 29-category sensitivity classification, concurrent path probing, bulk domain scanning.

gorobots Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# gorobots

robots.txt recon and path discovery tool written in Go. Fetches, parses, classifies sensitive paths, probes discovered endpoints, and supports bulk domain scanning.

Single static binary. No runtime dependencies.

## Install

```bash
go install github.com/momenbasel/pyrobots/cmd/gorobots@latest
```

Or build from source:

```bash
git clone https://github.com/momenbasel/pyrobots.git
cd pyrobots
go build -o gorobots ./cmd/gorobots/
```

## Usage

### Fetch and parse robots.txt

```bash
# Pretty-printed analysis with sensitivity tags
gorobots fetch https://example.com

# JSON output
gorobots fetch https://example.com --json

# Only show sensitive/interesting paths
gorobots fetch https://example.com --sensitive-only

# Through proxy
gorobots fetch https://example.com -x http://127.0.0.1:8080
```

### Probe discovered paths

Fetches robots.txt, extracts all paths, then sends HTTP requests to each one and reports status codes, content length, titles, and redirects.

```bash
# Default: 10 concurrent workers
gorobots probe https://example.com

# 50 workers, through Caido/Burp
gorobots probe https://example.com -w 50 -x http://127.0.0.1:8080

# Only show interesting responses (200, 301, 302, 401, 403)
gorobots probe https://example.com --interesting-only

# Authenticated
gorobots probe https://example.com -c "session=abc123" -H "Authorization: Bearer token"

# Export results
gorobots probe https://example.com -o results.json
gorobots probe https://example.com -o results.csv --format csv
```

### Dump paths (pipe-friendly)

```bash
# All paths, one per line
gorobots paths https://example.com

# Only disallowed paths
gorobots paths https://example.com --disallow-only

# With sensitivity tags
gorobots paths https://example.com --with-tags

# Pipe to other tools
gorobots paths https://example.com --disallow-only | httpx -silent
gorobots paths https://example.com | nuclei -t exposures/
gorobots paths https://example.com | ffuf -u "https://example.com/FUZZ" -w -
```

### Bulk domain scanning

```bash
# Fetch robots.txt for all domains
gorobots bulk -l domains.txt -o results/

# Fetch + probe all paths
gorobots bulk -l domains.txt --probe -o results/

# With proxy and custom workers
gorobots bulk -l domains.txt --probe -w 30 -x http://127.0.0.1:8080 -o results/
```

## What it parses

- **User-Agent** blocks
- **Allow** / **Disallow** directives
- **Sitemap** URLs
- **Crawl-delay** values
- **Comments** (often leak developer notes, internal URLs, tech stack clues)

## Sensitivity classification

Every path is checked against 29 pattern categories:

| Category | Examples |
|---|---|
| Admin panel | `/admin`, `/wp-admin` |
| API endpoints | `/api`, `/graphql`, `/swagger` |
| Source control | `/.git`, `/.svn` |
| Configuration | `/.env`, `/config`, `/web.xml` |
| Authentication | `/login`, `/oauth`, `/sso` |
| Development | `/debug`, `/test`, `/staging` |
| Backups | `.bak`, `.old`, `.swp` |
| DevOps tools | `/jenkins`, `/gitlab`, `/jira` |
| Monitoring | `/actuator`, `/health`, `/metrics` |
| User data | `/account`, `/profile`, `/password` |
| Payment | `/payment`, `/billing`, `/checkout` |
| And more... | uploads, logs, cron, exports, install |

## Output formats

- **Terminal** - colored table with status codes, content length, titles, redirects, sensitivity tags
- **JSON** - structured output
- **JSONL** - one JSON object per line (streaming)
- **CSV** - spreadsheet-compatible

## License

MIT

## Disclaimer

For authorized security testing and research only.

gorobots

gorobots Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

gorobots

gorobots Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README