Claimguard_kafka_spark

KO | EN

gitlite — search

#javascript #python #hacktoberfest #typescript #llm #golang #go #android #machine-learning #rust #linux #nodejs #java #vue

Claimguard_kafka_spark

(★ 10)

ClaimGuard is a real-time insurance analytics pipeline that detects fraud and monitors SLA breaches from live claim streams. Built using Apache Kafka, Apache Spark, and Flask, it processes and analyzes data instantly. It flags high-risk claims and visualizes insights through a live interactive dashboard.

Claimguard_kafka_spark Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ClaimGuard: Real-Time Insurance Fraud & SLA Intelligence
**ClaimGuard** is a high-performance, real-time data engineering pipeline designed to monitor insurance claims as they happen. Using a **Lambda-style streaming architecture**, the system identifies fraudulent patterns, tracks Service Level Agreement (SLA) breaches, and visualizes operational health through a sleek, dark-mode command center.
## System Architecture
The pipeline is built on a modern Big Data stack:
 1. **Data Generation:** A Python-based producer simulating a live firehose of insurance claim events.
 2. **Message Broker:** **Apache Kafka 4.2.0 (KRaft Mode)** acting as the real-time ingestion layer.
 3. **Stream Processing:** **Apache Spark 3.5.8 (Structured Streaming)** applying stateful transformation and fraud logic.
 4. **Frontend Dashboard:** A **Flask-based SPA** using **WebSockets (SSE)** and **Chart.js** for sub-second visualization.
## Key Features
 * **Real-Time Fraud Detection:** Hard-rule triggers for high-value claims ($50k+) and high-frequency claimant monitoring.
 * **SLA Escalation:** Automatically flags claims pending for >30 days for immediate manager intervention.
 * **Live Operations Firehose:** A rolling ledger of every claim hitting the broker, categorized by risk level.
 * **Dynamic Analytics:** Real-time Doughnut and Bar charts showing claim status distribution and volume by insurance line (Auto, Home, Medical, Travel).
 * **Auto-Approval Tracking:** Dedicated "Clearance" view for claims that pass the STP (Straight-Through Processing) check.
## Setup & Installation
### 1. Prerequisites
 * Ubuntu 20.04+ VM
 * Java 11+
 * Python 3.10+
 * Apache Spark 3.5.x
 * Apache Kafka 4.2.0
### 2. Environment Setup
```bash
# Install Python dependencies
pip3 install pyspark kafka-python flask

```
### 3. Start Kafka (KRaft Mode)
```bash
cd $KAFKA_HOME
# Generate Cluster ID and Format Storage
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/server.properties

# Start Broker
bin/kafka-server-start.sh config/server.properties &

# Create Topics
bin/kafka-topics.sh --create --topic raw_claims --bootstrap-server localhost:9092
bin/kafka-topics.sh --create --topic claim_alerts --bootstrap-server localhost:9092

```
## Running the Pipeline
Run these in three separate terminal windows or tmux panes:
**Terminal 1: The UI Dashboard**
```bash
python3 app.py

```
**Terminal 2: The Spark Stream Processor**
```bash
python3 spark_job.py

```
**Terminal 3: The Live Data Generator**
```bash
python3 producer.py

```
**Access the UI:** Open your browser and navigate to http://<your-vm-ip>:5000.
## Business Logic applied
| Module | Logic | Result |
|---|---|---|
| **Fraud Engine** | Amount > 50,000 | Flags as HIGH_RISK |
| **SLA Monitor** | Status == PENDING & Days > 30 | Triggers ESCALATION |
| **STP Engine** | Status == APPROVED | Moves to CLEARED |
## Configuration Notes
 * **Spark-Kafka Connector:** This project uses org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.0 to match Scala 2.12 environments.
 * **Checkpoints:** Spark checkpoints are stored locally in ./spark_checkpoints/ using a unique timestamp to prevent HDFS connection errors and ensure a clean state on every restart.
*Developed for Big Data Analytics (BDA) Lab | CBIT Hyderabad*

Claimguard_kafka_spark

Claimguard_kafka_spark Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Claimguard_kafka_spark

Claimguard_kafka_spark Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README