llm-gateway

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 69)

Squirrel LLM Gateway is a high-performance, enterprise-grade proxy service designed to unify and manage access to Large Language Model (LLM) providers like OpenAI and Anthropic. It provides intelligent routing, robust failover strategies, comprehensive logging, and a modern management dashboard.

llm-gateway Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

![cover](https://raw.githubusercontent.com/mylxsw/llm-gateway/main/docs/assets/cover.jpg)

<h1 align="center">Squirrel</h1>
 
 Enterprise-Grade LLM Gateway
 
 
 Unified API Proxy for OpenAI, Anthropic, and Compatible LLM Providers

<a href="README.md">English</a> ·
 <a href="README_zh-CN.md">中文</a>

## Overview

**Squirrel** is a high-performance, production-ready proxy service that unifies access to multiple Large Language Model (LLM) providers. It acts as an intelligent gateway between your applications and LLM services, providing seamless failover, load balancing, comprehensive observability, and a modern management dashboard — now with first-class OpenAI Responses support and smooth protocol conversion across OpenAI Chat, OpenAI Responses, and Anthropic Messages.

### Why Squirrel?

- **Single Integration Point**: Connect once, access multiple LLM providers through a unified API
- **Zero Code Changes**: Drop-in replacement compatible with OpenAI and Anthropic SDKs
- **Cost Optimization**: Route requests intelligently across providers based on rules, priority, or cost
- **Production Ready**: Built-in retry logic, failover mechanisms, and detailed request logging
- **Full Visibility**: Track every request with token usage, latency metrics, and cost analytics

---

## Key Features

### Unified API Interface

- **OpenAI Compatible**: Full support for `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/audio/*`, `/v1/images/*`
- **OpenAI Responses Compatible**: `/v1/responses` with streaming and tool-calls
- **Anthropic Compatible**: Native support for `/v1/messages` endpoint
- **Protocol Conversion**: Smoothly convert between OpenAI Chat ↔ OpenAI Responses ↔ Anthropic Messages (requests, responses, and streaming), powered by the built-in `llm_api_converter`
- **Streaming Support**: Full Server-Sent Events (SSE) support for real-time responses

### Intelligent Routing

- **Rule-Based Routing**: Route requests based on model name, headers, message content, or token count
- **Load Balancing Strategies**:
  - **Round-Robin**: Distribute requests evenly across providers
  - **Priority-Based**: Use preferred providers first, fallback to others
  - **Weight-Based**: Distribute by custom weight ratios
  - **Cost-Based**: Automatically select the lowest-priced model based on API pricing
- **Model Mapping**: Map virtual model names to multiple backend providers

### High Availability

- **Automatic Retries**: Configurable retry attempts for server errors (HTTP 500+)
- **Provider Failover**: Seamlessly switch to backup providers on failure
- **Timeout Management**: Configurable request timeouts with long streaming support (default: 30 minutes)

### Comprehensive Observability

- **Split Log Storage**: Summary fields stay in `request_logs`, while large request/response payloads are stored separately in `request_log_details` to reduce storage pressure and improve list-query performance
- **Independent Detail Retention**: Request/response detail payloads can expire earlier than summary logs, so you can keep analytics longer without retaining heavy bodies forever
- **Full Request/Response Capture**: Complete logging of request and response bodies (including streaming) to help debug issues and optimize AI system performance
- **Token Tracking**: Automatic token counting using [tiktoken](https://github.com/openai/tiktoken)
- **Latency Metrics**: First-byte delay and total response time
- **Cost Analytics**: Aggregated statistics by time, model, provider, and API key
- **Data Sanitization**: Automatic redaction of sensitive information in logs

### Modern Dashboard

Built with **Next.js 16** + **TypeScript** + **shadcn/ui**:

- Provider management with connection testing
- Model mapping configuration with rule editor
- API key generation and lifecycle management
- Advanced log viewer with multi-dimensional filtering
- Cost statistics and usage analytics

### MCP Management Interface

- Manage the gateway from AI agents over the **Model Context Protocol** (`/mcp`, Streamable HTTP)
- Inspect providers, model configs, request logs, and cost stats; optionally perform management writes
- Authorized with an existing API key granted **MCP admin** capability (secrets always redacted)
- See [docs/mcp.md](docs/mcp.md)

---

## Quick Start

### Docker Compose (Recommended)

The fastest way to get started with PostgreSQL:

```bash
# Clone the repository
git clone https://github.com/mylxsw/llm-gateway.git
cd llm-gateway
# Start services
docker compose -f docker-compose.prod.yml up -d
```

Access the dashboard at **http://localhost:8000** (or the port you set in `LLM_GATEWAY_PORT`)

### Docker (Single Container)

Run with SQLite for simple deployments:

```bash
docker run -d \
  -p 8000:8000 \
  -v $(pwd)/data:/data \
  --name llm-gateway \
  ghcr.io/mylxsw/llm-gateway:latest
```

### Manual Installation

#### Prerequisites

- Python 3.12+
- Node.js 18+
- npm (for frontend)

#### Backend Setup

```bash
cd backend

# Install dependencies (choose one)
uv sync          # Recommended: using uv
pip install -r requirements.txt  # Or using pip

# Initialize database
alembic upgrade head

# Start server
uvicorn app.main:app --host 0.0.0.0 --port 8000
```

#### Frontend Setup

```bash
cd frontend

# Install dependencies
npm install

# Development
npm run dev

# Production build
npm run build && npm run start
```

---

## Usage

### Basic Configuration

1. **Add a Provider**: Navigate to Providers page and add your LLM provider (e.g., OpenAI)
   - Set the base URL (e.g., `https://api.openai.com/v1`)
   - Add your API key
   - Select the protocol (OpenAI, OpenAI Responses, or Anthropic)

2. **Create Model Mapping**: Go to Models page and create a mapping
   - Define a model name (e.g., `gpt-4`)
   - Associate it with one or more providers
   - Set routing priority/weight

3. **Generate API Key**: Create a gateway API key in API Keys page

4. **Connect Your Application**:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="lgw-your-gateway-api-key"
)

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

#### OpenAI Responses example

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="lgw-your-gateway-api-key"
)

response = client.responses.create(
    model="gpt-4.1-mini",
    input="Summarize this in one sentence."
)
```

### API Endpoints

#### Proxy Endpoints (OpenAI Compatible)

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/v1/models` | List available models |
| POST | `/v1/chat/completions` | Chat completions |
| POST | `/v1/completions` | Text completions |
| POST | `/v1/embeddings` | Generate embeddings |
| POST | `/v1/audio/speech` | Text-to-speech |
| POST | `/v1/audio/transcriptions` | Speech-to-text |
| POST | `/v1/audio/translations` | Speech-to-text (translation) |
| POST | `/v1/images/generations` | Image generation |
| POST | `/v1/responses` | Responses API |

#### Proxy Endpoints (Anthropic Compatible)

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/v1/messages` | Messages API |

#### Admin Endpoints

| Resource | Endpoints |
|----------|-----------|
| Providers | `GET/POST /api/admin/providers`, `GET/PUT/DELETE /api/admin/providers/{id}` |
| Models | `GET/POST /api/admin/models`, `GET/PUT/DELETE /api/admin/models/{model}` |
| API Keys | `GET/POST /api/admin/api-keys`, `GET/PUT/DELETE /api/admin/api-keys/{id}` |
| Logs | `GET /api/admin/logs`, `GET /api/admin/logs/stats` |

See [docs/api.md](docs/api.md) for complete API documentation.

---

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `APP_NAME` | LLM Gateway | Application name |
| `DEBUG` | false | Enable debug mode |
| `DATABASE_TYPE` | sqlite | Database type: `sqlite` or `postgresql` |
| `DATABASE_URL` | sqlite+aiosqlite:///./llm_gateway.db | Database connection string |
| `RETRY_MAX_ATTEMPTS` | 3 | Max retry attempts for 500+ errors |
| `RETRY_DELAY_MS` | 1000 | Delay between retries (milliseconds) |
| `PROVIDER_HEALTH_ENABLED` | true | Enable runtime provider health degradation |
| `PROVIDER_HEALTH_WINDOW_SECONDS` | 600 | Provider health sliding-window duration |
| `PROVIDER_HEALTH_MIN_SAMPLES` | 6 | Minimum logical provider calls before degradation |
| `PROVIDER_HEALTH_FAILURE_RATE_THRESHOLD` | 0.5 | Failure rate that moves a provider behind healthy candidates |
| `HTTP_TIMEOUT` | 1800 | Upstream request timeout (seconds) |
| `API_KEY_PREFIX` | lgw- | Prefix for generated API keys |
| `API_KEY_LENGTH` | 32 | Length of generated API keys |
| `ENCRYPTION_KEY` | - | Base64-encoded 32-byte key used to encrypt stored sensitive fields (must stay stable across restarts) |
| `ENABLE_VIEW_API_KEYS` | false | Whether full API keys can be viewed/copied again on the API Keys page |
| `RATE_LIMIT_ENABLED` | false | Enable/disable built-in rate limiting middleware |
| `ADMIN_USERNAME` | - | Admin login username (optional) |
| `ADMIN_PASSWORD` | - | Admin login password (optional) |
| `ADMIN_TOKEN_TTL_SECONDS` | 86400 | Admin session TTL (24 hours) |
| `LOG_RETENTION_DAYS` | 7 | Log retention period |
| `LOG_DETAIL_RETENTION_DAYS` | 7 | Retention period for heavy request/response detail payloads; must be less than or equal to `LOG_RETENTION_DAYS` |
| `LOG_CLEANUP_INTERVAL_HOURS` | 24 | How often scheduled log cleanup runs |
| `LLM_GATEWAY_PORT` | 8000 | Host port for Docker Compose |
| `KV_STORE_TYPE` | database | KV store backend: `database` or `redis` |
| `REDIS_URL` | - | Redis connection URL (when using Redis KV store) |

### Log Retention Behavior

- `LOG_RETENTION_DAYS` controls how long summary log rows are kept.
- `LOG_DETAIL_RETENTION_DAYS` controls how long large request/response detail rows are kept.
- Once detail rows expire, the log entry still appears in the admin log list and stats, but request bodies, headers, upstream payloads, and retry/playground debug data are no longer available for that log.
- Scheduled cleanup runs every `LOG_CLEANUP_INTERVAL_HOURS`.

Generate an encryption key:
```bash
python -c "import secrets, base64; print(base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())"
```

Set it in your `.env` (required for production):
```env
ENCRYPTION_KEY=your-generated-key
```

### Database Configuration

**SQLite** (default, simple deployments):
```env
DATABASE_TYPE=sqlite
DATABASE_URL=sqlite+aiosqlite:///./llm_gateway.db
```

**PostgreSQL** (recommended for production):
```env
DATABASE_TYPE=postgresql
DATABASE_URL=postgresql+asyncpg://user:password@localhost:5432/llm_gateway
```

---

## Supported Providers

Squirrel can proxy requests to any OpenAI or Anthropic compatible API:

| Provider | Protocol | Notes |
|----------|----------|-------|
| OpenAI | OpenAI | Full support including GPT-4, GPT-3.5, embeddings, audio, images |
| OpenAI | OpenAI Responses | Responses API via `/v1/responses` |
| Anthropic | Anthropic | Claude models via Messages API |
| Azure OpenAI | OpenAI | Use Azure endpoint URL |
| Local Models | OpenAI | Ollama, vLLM, LocalAI, etc. |
| Other Providers | OpenAI/Anthropic | Any compatible API endpoint |

---

## Development

### Project Structure

```
llm-gateway/
├── backend/
│   ├── app/
│   │   ├── api/           # API routes (proxy, admin)
│   │   ├── services/      # Business logic
│   │   ├── providers/     # Protocol adapters
│   │   ├── repositories/  # Data access layer
│   │   ├── db/            # Database models
│   │   ├── domain/        # DTOs and domain models
│   │   ├── rules/         # Rule evaluation engine
│   │   └── common/        # Utilities
│   ├── migrations/        # Alembic migrations
│   └── tests/             # Test suite
├── llm_api_converter/     # Protocol conversion SDK (OpenAI/Responses/Anthropic)
├── frontend/
│   └── src/
│       ├── app/           # Next.js App Router pages
│       ├── components/    # React components
│       └── lib/           # Utilities and API client
├── docker-compose.yml
└── Dockerfile
```

### Running Tests

```bash
cd backend
pytest
```

### Database Migrations

```bash
cd backend

# Create new migration
alembic revision --autogenerate -m "description"

# Apply migrations
alembic upgrade head
```

---

## Documentation

- [Architecture Design](docs/architecture.md)
- [API Reference](docs/api.md)
- [Module Details](docs/modules.md)
- [Protocol Conversion](docs/protocol_conversion.md)
- [Requirements](docs/req.md)

---

## License

[MIT](LICENSE)

---

Made with care for the LLM community

llm-gateway

llm-gateway Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

llm-gateway

llm-gateway Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README