SentinelX-Real-Time-Fraud-Intelligence

KO | EN

gitlite — search

#javascript #python #hacktoberfest #go #rust #awesome #cli #vue #pytorch #security #ios #mcp #openai #open-source

SentinelX-Real-Time-Fraud-Intelligence

(★ 16)

SentinelX is an AI-powered real-time fraud detection system that combines machine learning, deep learning autoencoders, and graph-based analysis to identify fraudulent transactions. Using Kafka streaming and SHAP explainability, it delivers fast, transparent risk scoring through an interactive web dashboard.

SentinelX-Real-Time-Fraud-Intelligence Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Task 4 — Real-time AI fraud detection platform

End-to-end demo: **FastAPI** API, **React** dashboard, **Kafka** streaming, **PostgreSQL**, an **ensemble + Keras autoencoder** stack, **NetworkX** graph risk, **SHAP** explanations, and an optional **OpenAI-compatible** narrative endpoint.

**Default login (change in production):** `admin` / `admin123` (JWT).

---

## Contents

- [Features](#features)
- [Repository layout](#repository-layout)
- [Prerequisites](#prerequisites)
- [Quick start (Docker)](#quick-start-docker)
- [Configuration (environment)](#configuration-environment)
- [Database migrations (Alembic)](#database-migrations-alembic)
- [Local development (no Docker)](#local-development-no-docker)
- [API summary](#api-summary)
- [Risk engine](#risk-engine)
- [Security notes](#security-notes)
- [Performance](#performance)

## Features

- REST + WebSocket live updates; JWT auth
- Kafka producer/consumer pipeline into Postgres
- Hybrid score: ML ensemble, reconstruction error, graph risk
- SHAP-based explanations; optional LLM “analyst” copy via compatible APIs

## Repository layout

```
task4/
├── backend/                 # FastAPI, ML, Kafka, Alembic, scripts
│   ├── app/
│   ├── migrations/          # Alembic (avoids shadowing pip `alembic`)
│   ├── scripts/
│   ├── alembic.ini
│   └── requirements.txt     # pinned Python deps (included from repo root)
├── frontend/                # Vite + React + Tailwind + Recharts + D3
├── docker-compose.yml
├── requirements.txt         # pip install -r requirements.txt → backend pins
└── README.md
```

## Prerequisites

| Area | Requirement |
|------|-------------|
| **Docker path** | [Docker Desktop](https://www.docker.com/products/docker-desktop/) |
| **Local backend** | Python **3.11+**, `pip install -r requirements.txt` from repo root (or `backend/requirements.txt` from `backend/`) |
| **Local frontend** | Node.js **20+**, `npm install` in `frontend/` |
| **Full local streaming** | PostgreSQL, Kafka (KRaft or ZooKeeper), Java 17+ if using classic Kafka scripts |

## Quick start (Docker)

1. Install [Docker Desktop](https://www.docker.com/products/docker-desktop/).
2. From the `task4` directory:

```bash
docker-compose up --build
```

3. On startup the backend runs **`alembic upgrade head`** (see `scripts/entrypoint.py`), then trains models on first run if artifacts are missing. The first start can take several minutes while **models train** (synthetic data if `creditcard.csv` is absent) and artifacts are written to the shared `modelshare` volume.
4. Open **http://localhost:3000** (dashboard) and **http://localhost:8000/docs** (OpenAPI).
5. Sign in with **`admin` / `admin123`**. Set a strong **`JWT_SECRET`** (and rotate demo credentials) before any real deployment.

### Services

| Service | Port (host) | Role |
|--------|-------------|------|
| backend | 8000 | REST + WebSocket + ML inference |
| frontend | 3000 | Static UI (Nginx) |
| postgres | 5432 | `transactions`, `predictions`, `graph_edges` |
| kafka | 9092 | External PLAINTEXT_HOST listener |
| zookeeper | 2181 | Kafka dependency |
| consumer | — | Kafka → DB pipeline |
| producer | — | Simulated transaction stream |

Internal Kafka listener for containers: **`kafka:29092`** (set automatically in Compose).

### Optional: real credit card CSV

Place [Kaggle creditcard.csv](https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud) at `backend/data/creditcard.csv` and rebuild, or mount a volume to `/app/data/creditcard.csv` and run training:

```bash
docker-compose exec backend python scripts/train_models.py --csv /app/data/creditcard.csv
```

### Optional: AI narrative (OpenAI-compatible)

Set in `.env` or your shell (see [Configuration](#configuration-environment)). `POST /ai/insight` returns a short analyst-style summary; without a key, a **template** summary is built from SHAP + scores.

## Configuration (environment)

Variables are read from the environment and optional `.env` (see `backend/app/config.py`). Docker Compose sets the main ones for `backend`; override with a root `.env` for secrets.

| Variable | Purpose | Example / default |
|----------|-----------|-------------------|
| `DATABASE_URL` | Postgres connection | `postgresql://fraud:fraud@localhost:5432/fraud_db` |
| `KAFKA_BOOTSTRAP_SERVERS` | Broker list | `localhost:9092` (local); `kafka:29092` in Compose |
| `JWT_SECRET` | HS256 signing secret | **Set in production** |
| `JWT_ALGORITHM` | Algorithm | `HS256` |
| `JWT_EXPIRE_MINUTES` | Token lifetime | `60` |
| `MODELS_DIR` | Saved model artifacts | `models` or `/app/models` in Docker |
| `DATA_DIR` | CSV / data files | `data` |
| `OPENAI_API_KEY` | Optional LLM insights | Empty → template-only |
| `OPENAI_BASE_URL` | Compatible API base | `https://api.openai.com/v1` |
| `OPENAI_MODEL` | Model id | `gpt-4o-mini` |

**Frontend:** `VITE_API_URL` points the UI at the API (e.g. `http://127.0.0.1:8000` for `npm run dev`).

## Database migrations (Alembic)

Schema is **not** created with `Base.metadata.create_all()` at runtime. Migrations live under `backend/migrations/` and are applied before the API starts in Docker.

**Common commands** (from `backend/`, with `DATABASE_URL` set):

```bash
# Apply all pending migrations
python -m alembic upgrade head

# Create a new revision from model changes (review the diff before committing)
python -m alembic revision --autogenerate -m "describe change"

# One step downgrade
python -m alembic downgrade -1
```

**Existing database** that already has the same tables from an older build (when the app used `create_all`): either apply migrations on a fresh volume, or tell Alembic the DB is already at the latest revision without running DDL:

```bash
python -m alembic stamp head
```

`migrations/env.py` reads `DATABASE_URL` first, then falls back to `app.config.Settings.database_url`.

## Local development (no Docker)

### Full stack: Postgres + Kafka on your machine

Run **PostgreSQL**, **Kafka** (KRaft without ZooKeeper is documented below; Compose-style ZooKeeper + Kafka also works), the backend, **consumer**, **producer**, and the Vite dev server.

**Kafka (Windows, KRaft)** — prerequisites: **Java 17+**, **Python 3.11+**, **Node.js 20+**, [Apache Kafka](https://kafka.apache.org/downloads) extracted to a path **without spaces** (e.g. `C:\kafka\kafka_2.13-3.7.0`). Set `KAFKA_HOME` to that folder.

**One-time KRaft setup**

1. Edit `%KAFKA_HOME%\config\kraft\server.properties`: `listeners=PLAINTEXT://localhost:9092`, `advertised.listeners=PLAINTEXT://localhost:9092`, and `log.dirs` to an existing folder (e.g. `C:/kafka-data/kraft-combined-logs`).
2. Format storage (once per new `log.dirs`):

```powershell
cd $env:KAFKA_HOME
.\bin\windows\kafka-storage.bat random-uuid
.\bin\windows\kafka-storage.bat format -t PASTE_UUID_HERE -c .\config\kraft\server.properties
```

3. Start the broker (keep running): `.\bin\windows\kafka-server-start.bat .\config\kraft\server.properties`

4. Create topic `transactions`:

```powershell
& "$env:KAFKA_HOME\bin\windows\kafka-topics.bat" --create --if-not-exists --topic transactions --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
```

**Startup order:** PostgreSQL → Kafka → topic → backend → consumer → producer → frontend.

**Backend** (from `backend/`, new venv: `python -m venv .venv` then install deps from repo root with `pip install -r requirements.txt`, or from `backend/` with `pip install -r requirements.txt`):

```powershell
$env:DATABASE_URL = "postgresql://fraud:fraud@localhost:5432/fraud_db"
$env:KAFKA_BOOTSTRAP_SERVERS = "localhost:9092"
python -m alembic upgrade head
python scripts\train_models.py
uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
```

**Consumer** (second terminal, same venv + same `DATABASE_URL` / `KAFKA_BOOTSTRAP_SERVERS`): `python -m app.kafka_app.consumer`

**Producer** (third terminal, venv + `KAFKA_BOOTSTRAP_SERVERS`): `python -m app.kafka_app.producer`

**Frontend:**

```powershell
cd ..\frontend
npm install
$env:VITE_API_URL = "http://127.0.0.1:8000"
npm run dev
```

Open the URL Vite prints (e.g. **http://localhost:5173**). Sign in: **`admin` / `admin123`**.

**Troubleshooting**

- `NoBrokersAvailable` → Kafka not running or `KAFKA_BOOTSTRAP_SERVERS` does not match broker `listeners`.
- Consumer “models not loaded” → run `python scripts\train_models.py`.
- Older Kafka + ZooKeeper installs work: start ZooKeeper and broker, then create the same topic.
- **`Activate.ps1` not found** → create the venv in `backend/` with `python -m venv .venv`, then install requirements. If PowerShell blocks scripts: `Set-ExecutionPolicy -Scope CurrentUser RemoteSigned`, or call `.\.venv\Scripts\python.exe -m app.kafka_app.consumer` without activating.
- **`No module named 'kafka'`** → use the venv’s Python after `pip install -r requirements.txt` (from root) or `pip install -r backend/requirements.txt`.

### Backend only (no Kafka)

Useful for UI + API work; omit producer/consumer. The dashboard simulator can still drive flows without a broker.

```bash
cd backend
python -m venv .venv
.venv\Scripts\activate          # Windows
# macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
set DATABASE_URL=postgresql://fraud:fraud@localhost:5432/fraud_db
python -m alembic upgrade head
python scripts/train_models.py
uvicorn app.main:app --reload --port 8000
```

### Kafka consumer / producer (broker already running)

```bash
# from backend/, venv active, KAFKA_BOOTSTRAP_SERVERS set
python -m app.kafka_app.producer
python -m app.kafka_app.consumer
```

### Frontend (custom API URL)

```bash
cd frontend
npm install
set VITE_API_URL=http://localhost:8000
npm run dev
```

## API summary (JWT required except `/health`, `/auth/login`)

| Method | Path | Description |
|--------|------|-------------|
| POST | `/auth/login` | JSON `{ "username", "password" }` → JWT |
| POST | `/predict` | Score + persist transaction |
| GET | `/transactions` | Recent rows with predictions |
| GET | `/fraud-graph` | Nodes/edges + metrics for D3 |
| GET | `/explain/{transaction_id}` | SHAP top features (RandomForest) |
| POST | `/ai/insight` | SHAP + scores → template or LLM narrative |
| POST | `/ai/score-preview` | Score features without DB graph context |
| WS | `/ws/stream?token=...` | New transactions (polls DB + push) |
| GET | `/health` | Liveness + `models_loaded` |

## Risk engine

Final score (0–1):

`0.5 × ML_ensemble_prob + 0.35 × autoencoder_norm + 0.15 × graph_risk`

Levels: **LOW** / **MEDIUM** / **HIGH** from thresholds on the final score.

## Security notes

- Replace `JWT_SECRET` and demo credentials before any real deployment.
- CORS is `*` for the template; restrict origins in production.
- Protect Kafka and Postgres with network policies and strong passwords.

## Performance

Inference is tuned for **sub-200 ms** per transaction on CPU for typical batch size 1; SHAP and optional LLM calls are **on-demand** and not counted in the streaming SLA.

SentinelX-Real-Time-Fraud-Intelligence

SentinelX-Real-Time-Fraud-Intelligence Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

SentinelX-Real-Time-Fraud-Intelligence

SentinelX-Real-Time-Fraud-Intelligence Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README