KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #deepseek-harness #typescript #ai-agent #codex #developer-tools #claude-code #windows #mcp #python #desktop-app

LLM_GATEWAY

★ 1 Open GitHub ↗

An LLM Gateway is a middleware layer that sits between your applications and one or more LLM providers (OpenAI, Anthropic, Groq, Azure OpenAI, etc.), centralizing and standardizing how requests flow to and from models.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LLM Gateway

A small, hands-on tour of [LiteLLM](https://docs.litellm.ai/) as a multi-provider
LLM gateway: one `completion()` call that works across OpenAI, Anthropic, Groq,
Gemini, and Hugging Face; provider fallback chains; response caching; load
balancing; cost tracking; audit logging; a LangChain adapter; a task-aware
smart router; and three input guardrails (PII redaction, prompt-injection
detection, forbidden-topic blocking).

## Project structure

```
LLM_GATEWAY/
├── main.py                     # CLI entrypoint — run any demo by name
├── requirements.txt
├── .env.example                # copy to .env and fill in your keys
├── llm_gateway/
│   ├── config.py                # loads .env, reports which provider keys are set
│   ├── providers.py              # same completion() call across 5 providers
│   ├── fallbacks.py              # fallbacks=[...] chain demos
│   ├── caching.py                # in-memory response caching
│   ├── cost_tracking.py          # per-call USD cost via completion_cost()
│   ├── routing.py                # litellm.Router: aliasing + load-balancing strategies
│   ├── logging_utils.py          # success/failure callback audit logging
│   ├── langchain_demo.py         # ChatLiteLLM inside a LangChain chain
│   ├── smart_router.py           # classify → route → fallback, with cost/latency
│   └── guardrails/
│       ├── exceptions.py         # shared GuardrailViolation exception
│       ├── pii.py                # regex PII detection + redaction
│       ├── injection.py          # prompt-injection / jailbreak pattern matching
│       └── topics.py             # forbidden-topic keyword blocking
```

Each module is self-contained and runnable on its own:

```bash
python -m llm_gateway.providers
python -m llm_gateway.guardrails.pii
```

`main.py` is a thin CLI over the same functions, useful for running one demo
(or all of them) without remembering module paths.

## Setup

```bash
python -m venv .venv
.venv\Scripts\activate        # Windows
pip install -r requirements.txt
copy .env.example .env        # then fill in the keys you have
```

You don't need every key — see [Demo → required keys](#demo--required-keys) below.

## Usage

```bash
python main.py --list              # see all available demos
python main.py keys                # check which provider keys are loaded
python main.py providers           # run one demo
python main.py all                 # run every demo, in order
```

## Demo → required keys

| Demo                  | Needs                              |
| ---------------------- | ----------------------------------- |
| `keys`                 | none (just reports what's set)      |
| `providers`             | all five keys (each call is independently try/excepted) |
| `fallbacks`             | `GOOGLE_API_KEY`, `OPENAI_API_KEY`, `GROQ_API_KEY` |
| `caching`, `cost`, `logging` | `OPENAI_API_KEY` |
| `routing-*`             | `OPENAI_API_KEY`, `GROQ_API_KEY`    |
| `smart-router`          | `OPENAI_API_KEY`, `GROQ_API_KEY`    |
| `guardrail-*`           | `OPENAI_API_KEY`                    |
| `langchain`             | `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GROQ_API_KEY` |

## What changed from the original single-file script

The original `llm_gateway.py` was a ~660-line dump of what looks like a
Jupyter notebook flattened into one `.py` file — every cell's output back to
back, with no functions or `__main__` guards. That caused a few real bugs on
top of the organizational mess, all fixed in this restructure:

- **Would not run top-to-bottom.** Two spots had leftover 4-space-indented
  `import` lines (`import os`, `import re`) sitting right after a `for` loop
  ended, which is an `IndentationError` in a plain script — only worked as
  separate notebook cells.
- **`GuardrailViolation` was defined twice.** Harmless as isolated notebook
  cells, but once everything runs in the same process the second class
  definition shadows the first — an `except GuardrailViolation` in one
  guardrail module could fail to catch an exception raised by another. Now
  there's one shared class in `guardrails/exceptions.py`.
- **PII redaction had a pattern-ordering bug.** A 16-digit credit card number
  also matches the 12-digit Aadhaar regex as a leading substring. The
  original dict checked `AADHAAR` before `CREDIT_CARD`, so a credit card
  number would get partially redacted as `<AADHAAR_REDACTED>` and the
  `CREDIT_CARD` pattern would then never match the (already-substituted)
  remainder. `pii.py` now checks `CREDIT_CARD` first.
- **Retired Anthropic model IDs.** `claude-3-5-haiku-20241022` and
  `anthropic/claude-3-5-sonnet` are retired. Swapped for the current aliases
  `claude-haiku-4-5` and `claude-sonnet-5`.
- **No `.env` loading.** Everything read `os.getenv(...)` directly with no
  `load_dotenv()` anywhere, so a `.env` file would have been silently
  ignored. `config.py` now calls `load_dotenv()` on import.
- **Duplicate/scattered imports.** `litellm`, `completion`, `os`, `re` were
  each imported three-plus times across the file. Each module now imports
  exactly what it uses, once, at the top.
- **Global side effects at import time.** The original ran every API call
  as soon as the file was imported (impossible to import one demo without
  triggering all the others). Every demo is now a plain function, so
  `import llm_gateway.caching` does nothing until you call
  `caching.run_caching_demo()`.

## Notes

- Model IDs for Gemini (`gemini/gemini-2.0-flash`) and Hugging Face
  (`huggingface/google/gemma-3-12b-it`) are best-effort — verify they're
  still current against each provider's own model catalog before relying on
  them in production; these evolve faster than this README will be updated.
- `litellm`'s in-memory `Cache(type="local")` and the various `Router`
  load-balancing strategies (`simple-shuffle`, `least-busy`,
  `latency-based-routing`) are demo-oriented; for production use, see
  LiteLLM's docs on Redis-backed caching and its full routing-strategy list.

LLM_GATEWAY

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README