Go2NetSpectra

KO | EN

gitlite — search

#python #javascript #reactjs #react #html #machine-learning #typescript #css #google #home-assistant #express #nlp

★ 68 Open GitHub ↗

Go2NetSpectra is a high-performance, distributed traffic monitoring and analysis framework designed for both real-time and offline data processing in large-scale network environments.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Go2NetSpectra

[![Go](https://img.shields.io/badge/go-1.21%2B-blue.svg)](https://go.dev/) [![gopacket](https://img.shields.io/badge/gopacket-1.1.19-blue.svg)](https://github.com/google/gopacket) [![NATS](https://img.shields.io/badge/NATS-2.11%2B-green.svg)](https://nats.io/) [![Thrift](https://img.shields.io/badge/Thrift-0.22%2B-blue.svg)](https://thrift.apache.org/) [![ClickHouse](https://img.shields.io/badge/ClickHouse-23.0%2B-yellow.svg)](https://clickhouse.com/) [![Grafana](https://img.shields.io/badge/Grafana-10.0%2B-orange.svg)](https://grafana.com/) [![Docker](https://img.shields.io/badge/docker-20.10%2B-blue)](https://www.docker.com/) [![Kubernetes](https://img.shields.io/badge/Kubernetes-1.27%2B-blue.svg)](https://kubernetes.io/) [![License](https://img.shields.io/badge/license-Apache-blue.svg)](https://raw.githubusercontent.com/Decade-qiu/Go2NetSpectra/main/LICENSE)

## 📖 Overview

**Go2NetSpectra** is a high-performance, distributed network traffic monitoring and analysis framework written in Go. It provides a powerful platform for network engineers, security analysts, and SREs to gain deep, multi-dimensional insights into network traffic in real-time.

By leveraging a high-speed data pipeline, a flexible pluggable aggregation engine, and AI-powered threat analysis, Go2NetSpectra enables everything from basic network performance monitoring to sophisticated security threat detection, all at scale.

### ✨ Key Features

- **🔀 Hybrid Analysis Engine**: Simultaneously run multiple aggregator types (Exact + Sketch) for **100% accurate accounting** and **high-performance probabilistic analysis** at the same time. Use Sketch to find anomalies, then use Exact to get precise details.

- **🤖 AI-Powered Alert Analysis**: The new `ns-ai` microservice provides intelligent analysis for triggered alerts. It enriches notifications with root cause analysis, threat assessment, and mitigation suggestions, turning raw alerts into actionable insights.

- **⚡ Real-Time Alerting**: A built-in alerting pipeline allows tasks to generate event messages. These are processed by a central `Alerter` which can trigger notifications via webhooks, providing immediate insights into network events.

- **🧩 Pluggable Aggregation Algorithms**: The `sketch` aggregator is a micro-framework supporting multiple estimation algorithms via configuration:
  - **Count-Min Sketch**: Heavy hitter detection with configurable accuracy
  - **SuperSpread**: Cardinality estimation and super-spreader detection

- **⚙️ High-Performance by Design**: Built from the ground up for performance:
  - Lock-free atomic operations in sketches
  - Worker pool concurrency with optimal goroutine scheduling
  - Efficient Thrift serialization
  - Zero-copy data handling where possible

- **📊 Full-Stack Observability**: Built-in support for:
  - Real-time Grafana dashboards
  - Structured alerting with webhooks
  - Multiple data backends (ClickHouse, File-based storage)
  - Comprehensive query APIs (Thrift RPC + HTTP/JSON)

- **🚀 Scalable & Distributed**: All components are decoupled and horizontally scalable:
  - NATS for message bus decoupling
  - Stateless microservices (Engine, API, AI)
  - Kubernetes-ready with Helm charts
  - Automatic leader election and failover

- **🔍 Intelligent Search**: Support for both exact and approximate queries:
  - Precise flow accounting with 100% accuracy
  - Fast probabilistic heavy hitter detection
  - Flexible multi-dimensional aggregations

---

## 🏗️ System Architecture

Go2NetSpectra operates as a multi-stage, distributed pipeline designed for performance, scalability, and real-time analysis.

```mermaid
graph TD
    subgraph DataPlane["📊 Data Collection Plane"]
        direction LR
        Iface["🖧 Network Interface"]
        Pcap["📁 PCAP Files"]
        Probe["🔍 ns-probe Live Capture"]
        Analyzer["🔬 pcap-analyzer Offline Analysis"]
        
        Iface -->|live traffic| Probe
        Pcap -->|offline traffic| Analyzer

style Iface fill:#fbe9e7,stroke:#d84315,stroke-width:2px,color:#222
        style Pcap fill:#fbe9e7,stroke:#d84315,stroke-width:2px,color:#222
        style Probe fill:#ffccbc,stroke:#e64a19,stroke-width:2px,color:#222
        style Analyzer fill:#ffccbc,stroke:#e64a19,stroke-width:2px,color:#222
    end

subgraph MessageBus["🚌 Message Bus"]
        direction TB
        NATS["📬 NATS Message Broker"]
        style NATS fill:#fff3e0,stroke:#fb8c00,stroke-width:3px,color:#222
    end

subgraph ProcessingPlane["⚙️ Processing & Analysis Plane"]
        direction TB
        
        subgraph Engine["🧠 ns-engine: Core Processing"]
            Manager["📋 Manager Worker Pool Orchestrator"]
            style Manager fill:#e0f2f1,stroke:#00897b,stroke-width:2px,color:#222
        end
        
        subgraph Aggregators["📈 Aggregation Tasks"]
            ExactTask["✓ Exact Task 100% Accurate"]
            SketchTask["⚡ Sketch Task Probabilistic Count-Min/SuperSpread"]
            style ExactTask fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#222
            style SketchTask fill:#bbdefb,stroke:#1e88e5,stroke-width:2px,color:#222
        end
        
        subgraph Alerting["🚨 Real-time Alerting & AI"]
            Alerter["⚠️ Alerter Rule Evaluation"]
            AI_Service["🤖 ns-ai AI Analysis Gateway"]
            Notifier["📧 Notifier Email/Webhook"]
            style Alerter fill:#fff8e1,stroke:#fbc02d,stroke-width:2px,color:#222
            style AI_Service fill:#ede7f6,stroke:#7e57c2,stroke-width:2px,color:#222
            style Notifier fill:#ffe0b2,stroke:#f57c00,stroke-width:2px,color:#222
        end
        
        subgraph Storage["💾 Storage Layer"]
            ClickHouse["📊 ClickHouse Time-Series DB"]
            style ClickHouse fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#222
        end
        
        Manager --> ExactTask
        Manager --> SketchTask
        ExactTask -->|snapshot| ClickHouse
        SketchTask -->|snapshot| ClickHouse
        ExactTask -->|generates event| Alerter
        SketchTask -->|generates event| Alerter
        Alerter -->|Thrift RPC| AI_Service
        AI_Service -->|enriched analysis| Alerter
        Alerter -->|formatted alert| Notifier
        
        %% --- 修改点 1：为 ProcessingPlane 内部的 subgraph 添加 color ---
        style Engine fill:#f1f8e9,stroke:#388e3c,stroke-width:2px,color:#222
        style Aggregators fill:#e3f2fd,stroke:#1976d2,stroke-width:2px,color:#222
        style Alerting fill:#fff8e1,stroke:#fbc02d,stroke-width:2px,color:#222
        style Storage fill:#ede7f6,stroke:#5e35b1,stroke-width:2px,color:#222
    end

subgraph QueryPlane["🔍 Query & Interaction Plane"]
        direction TB
        
        API["🌐 ns-api Multi-Query Router"]
        GrpcClient["📱 RPC Client High-Performance"]
        HttpClient["🌍 HTTP/JSON Client Grafana Integration"]
        AI_Client["💬 AI Client Interactive QA"]
        Grafana["📊 Grafana Real-time Dashboards"]
        
        style API fill:#e8f5e9,stroke:#43a047,stroke-width:3px,color:#222
        style GrpcClient fill:#c8e6c9,stroke:#388e3c,stroke-width:2px,color:#222
        style HttpClient fill:#c8e6c9,stroke:#388e3c,stroke-width:2px,color:#222
        style AI_Client fill:#c8e6c9,stroke:#388e3c,stroke-width:2px,color:#222
        style Grafana fill:#ffe0b2,stroke:#f57c00,stroke-width:2px,color:#222
    end

%% --- 修改点 2：为顶层 subgraph 添加 color ---
    style DataPlane fill:#ffebee,stroke:#d32f2f,stroke-width:2px,color:#222
    style MessageBus fill:#fff3e0,stroke:#ef6c00,stroke-width:2px,color:#222
    style ProcessingPlane fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#222
    style QueryPlane fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#222
```

### Core Components

---

## Boundary Ownership And Refactor Guardrails

- `cmd/` packages are process wiring only. Shared runtime assembly belongs in `internal/api`, `internal/ai`, or `internal/engine/app`.
- `internal/probe` and `pkg/pcap` own transport packet conversion; `internal/engine/manager` owns fan-out and lifecycle orchestration.
- `api/thrift/v1/`, `configs/config.yaml`, Docker Compose values, and Helm values are synchronized product surfaces. Any behavioral key change must update all affected runtime and deployment assets together.
- Use [module-boundaries.md](doc/refactor/module-boundaries.md) for the current ownership map and [build.md](doc/build.md) for validation entrypoints.

---

## 🚀 Quick Start

Choose the deployment option that best fits your needs.

### Prerequisites

### Environment Setup

All configuration uses environment variables via `.env` or `.docker.env` files:

```bash
# Copy the example file
cp configs/.env.example .env

# Edit with your settings
# NATS_URL=nats://localhost:4222
# CLICKHOUSE_HOST=localhost
# API_RPC_LISTEN_ADDR=localhost:50051
# AI_API_KEY=your-openai-key
# SMTP_PASSWORD=your-smtp-password
```

### Option 1: Docker Compose (Recommended - Quickest)

Perfect for quick evaluation and development. Starts all backend services in containers.

**Step 1: Prepare Configuration**

```bash
# Copy environment file for Docker Compose
cp configs/.env.example deployments/docker-compose/.docker.env

# Edit .docker.env with Docker-specific settings
# (e.g., NATS_URL=nats://nats:4222)
```

**Step 2: Start All Services**

```bash
cd deployments/docker-compose/
docker compose up --build
```

This starts:
- ✅ NATS message broker
- ✅ ClickHouse database
- ✅ ns-engine (processing)
- ✅ ns-api (query service)
- ✅ ns-ai (AI analysis)
- ✅ Grafana (visualization)

**Step 3: Run Probe & Capture Traffic**

In a **new terminal**, capture live traffic:

```bash
# Replace <interface_name> with your network interface (e.g., en0, eth0, wlan0)
sudo go run ./cmd/ns-probe/main.go --mode=pub --iface=<interface_name>
```

**Step 4: Query & Visualize**

In a **third terminal**, interact with the services:

```bash
# Query aggregated flows
go run ./scripts/query/v2/main.go --mode=aggregate --task=per_src_ip

# Query heavy hitters (top IPs)
go run ./scripts/query/v2/main.go --mode=heavyhitters --task=per_src_ip --type=0 --limit=10

# Interactive AI analysis
go run ./scripts/ask-ai/main.go "Summarize the network anomalies in the last minute"
```

Access Grafana at `http://localhost:3000` (admin/admin) to view real-time dashboards.

---

### Option 2: Local Development (Advanced)

Run services directly on your machine for debugging and development.

**Step 1: Prepare Local Configuration**

```bash
# Copy to project root
cp configs/.env.example .env

# Edit .env with local settings
# (e.g., NATS_URL=nats://localhost:4222, CLICKHOUSE_HOST=localhost)
```

**Step 2: Start Dependencies**

Open separate terminals for each:

```bash
# Terminal 1: Start NATS
docker run --rm -p 4222:4222 nats:latest

# Terminal 2: Start ClickHouse
docker run -d -p 18123:8123 -p 19000:9000 \
  -e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} \
  --name clickhouse-server \
  --ulimit nofile=262144:262144 \
  clickhouse/clickhouse-server
```

**Step 3: Run Applications Locally**

Open separate terminals for each service:

```bash
# Terminal 3: Start Engine
go run ./cmd/ns-engine/main.go

# Terminal 4: Start API Service
go run ./cmd/ns-api/v2/main.go

# Terminal 5: Start AI Service
go run ./cmd/ns-ai/main.go

# Terminal 6: Run Probe (requires sudo)
sudo go run ./cmd/ns-probe/main.go --mode=pub --iface=<interface_name>
```

Applications automatically load `.env` configuration.

---

### Option 3: Kubernetes Deployment

Deploy to a Kubernetes cluster for high availability and scalability.

#### Method A: Raw Manifests (Quick Test)

```bash
# Configure secrets
cd deployments/kubernetes/
vim go2netspectra-secret.yaml  # Fill in your credentials

# Run deployment script
chmod +x deploy-k8s.sh
./deploy-k8s.sh
```

#### Method B: Helm (Recommended for Production)

```bash
cd deployments/helm/go2netspectra/

# Customize values
cp values.yaml my-values.yaml
vim my-values.yaml  # Update sensitive config in the 'config' section

# Install the chart
helm install go2netspectra . -f my-values.yaml

# Verify deployment
helm status go2netspectra

# Uninstall when done
helm uninstall go2netspectra
```

---

## 📚 Detailed Documentation

| Document | Content |
| :--- | :--- |
| **[`doc/build.md`](doc/build.md)** | Comprehensive build, environment setup, and deployment guide. Includes Thrift generation, local development, Docker Compose, and Kubernetes instructions. |
| **[`doc/technology.md`](doc/technology.md)** | In-depth technical architecture, design decisions, performance optimizations, and algorithm implementations. Covers the hybrid analysis engine, AI integration, and performance benchmarks. |
| **[`doc/re.md`](doc/re.md)** | Requirements specification, feature list, project roadmap, and evolutionary milestones. Explains the vision and development phases. |

---

## 📊 Core Features Deep Dive

### 1️⃣ Hybrid Analysis Engine (Exact + Sketch)

Process traffic through multiple algorithms simultaneously:

```
Input Packet
    ↓
[Exact Task]     → 100% accurate per-flow accounting → ClickHouse
    ↓                     
[Sketch Task]    → Fast probabilistic heavy hitter detection → ClickHouse
    ↓
[Alert Generator] → Real-time event messages → Alerter → AI Analysis
```

**Benefits**:
- Use fast `sketch` for anomaly detection
- Use accurate `exact` for verification
- No performance trade-off - run both simultaneously
- Configurable aggregation granularity (per-source IP, per-destination, etc.)

### 2️⃣ Pluggable Aggregation Algorithms

Configure different estimation algorithms via `config.yaml`:

- **Count-Min Sketch**: Frequency estimation for heavy hitter detection
 - Memory: ~625 KB (fixed, configurable)
 - Error rate: <0.1% for large flows
 - Performance: 4.3x faster than exact mode

- **SuperSpread**: Cardinality estimation for super-spreader detection
  - Memory: ~84 MB (for 600K source IPs)
  - Use case: DDoS attack source detection, botnet tracking
  - Performance: 6.8x faster than exact cardinality

### 3️⃣ Real-Time Alerting Pipeline

```
[ns-engine]
    ↓ generates alert events
[Alerter] → evaluates against rules
    ↓ if triggered
[ns-ai] → enriches with LLM analysis
    ↓ formats as HTML
[Notifier] → sends via webhook/email/etc
```

**Alert Types**:
- Heavy hitters (top N flows by traffic)
- Anomalies (statistical deviations)
- Protocol violations (malformed packets)
- Threshold breaches (custom rules)

### 4️⃣ AI-Powered Analysis (ns-ai)

Integrates LLM capabilities for intelligent threat analysis:

```
Alert Summary → LLM Analysis → Enriched Report
   ↓                                ↓
"High traffic"  →  → "Potential DDoS from 10.0.0.x/24
                       - Recommend: rate limiting
                       - Attack pattern: SYN flood"
```

**Capabilities**:
- Root cause analysis
- Threat classification
- Mitigation recommendations
- Integration with OpenAI-compatible APIs

### 5️⃣ Performance Optimization

Benchmark results (Intel i7-14700, 30M+ packet dataset):

| Operation | Exact Mode | Sketch Mode | Speedup |
| :--- | :--- | :--- | :--- |
| Insert | 2.07s | 582.88ms | **3.6x faster** |
| Query | 358ms | 156.7ms | **2.3x faster** |
| Memory | Dynamic | 625 KB | **Fixed memory** |
| GC Overhead | 140s | 106s | **24% reduction** |

**Optimizations**:
- Lock-free atomic operations (CAS) in Count-Min
- Object pooling with `sync.Pool` for temporary allocations
- Goroutine worker pools for parallel processing
- Efficient Thrift serialization

---

## 🔧 Configuration Guide

### Key Configuration Parameters

```yaml
aggregator:
  period: 60s              # Global measurement period
  types: [exact, sketch]   # Enabled aggregators
  
exact:
  tasks:
    - name: per_src_ip
      dimensions: [SrcIP]
      writer:
        type: clickhouse
        interval: 10s

sketch:
  tasks:
    - name: per_src_ip
      skt_type: 0          # 0=CountMin, 1=SuperSpread
      countmin:
            depth: 3
            width: 8191    # 2^13
      writer:
        type: clickhouse
        interval: 10s

alerter:
  enabled: true
  rules:
    - name: high_traffic
      condition: "ByteCount > 1GB"
      action: alert

ai:
  enabled: true
  api_key: ${AI_API_KEY}
  base_url: https://api.openai.com/v1
```

For complete configuration reference, see [`doc/build.md`](doc/build.md).

---

## 📈 Use Cases

### 1. Network Performance Monitoring
Monitor bandwidth usage, latency, and packet loss across your network in real-time.

### 2. DDoS Attack Detection
Detect volumetric attacks using Sketch-based heavy hitter detection, then get precise attack details with Exact mode.

### 3. Anomaly Detection
Identify unusual traffic patterns (sudden spikes, new protocols, unusual port combinations).

### 4. Security Threat Analysis
Combine network traffic insights with AI analysis to detect and classify security threats.

### 5. Capacity Planning
Use historical data to identify trends and plan network capacity upgrades.

---

## 🐳 Docker & Container Support

### Docker Compose Quick Commands

```bash
# Build and start all services
docker compose up --build

# View logs
docker compose logs -f ns-engine

# Stop all services
docker compose down

# Clean up volumes
docker compose down -v
```

### Multi-Language Sandbox Support

Go2NetSpectra can process traffic from multiple network protocols:

```bash
# View supported analyzers
go run ./cmd/ns-probe/main.go --help

# Analyze specific protocols
go run ./scripts/query/v2/main.go --help
```

---

## 🧪 Testing & Validation

### Run Thrift Generation (If Modifying IDL Files)

```bash
# Generate code
thrift --gen go -out api/gen/thrift api/thrift/v1/traffic.thrift
thrift --gen go -out api/gen/thrift api/thrift/v1/query.thrift
thrift --gen go -out api/gen/thrift api/thrift/v1/ai.thrift
rm -rf api/gen/thrift/v1/*-remote
```

### Run Algorithm Benchmarks

```bash
# Count-Min accuracy and concurrency tests
go test -v ./internal/engine/impl/sketch/

# Sketch vs Exact performance comparison
go test -bench=. ./internal/engine/impl/benchmark/

# Representative hot-path microbenchmarks
go test -run '^$' -bench '^Benchmark(ProtocolParsePacketInto|PacketCodecRoundTrip|ExactTaskProcessPacket|CountMinTaskProcessPacket|SuperSpreadTaskProcessPacket)$' -benchmem ./internal/engine/impl/benchmark/
go test -run '^$' -bench '^BenchmarkMurmurHash3RepresentativeFlowInputs$' -benchmem ./scripts/hash/
```

### Validate Against Test PCAP

```bash
# Analyze included test file
go run ./cmd/pcap-analyzer/main.go test/data/test.pcap

# Query results
go run ./scripts/query/v2/main.go --mode=aggregate --task=per_src_ip
```

---

## 🤝 Contributing

Contributions are welcome! Here's how to get started:

1. **Fork** the repository
2. **Create a feature branch** (`git checkout -b feature/amazing-feature`)
3. **Commit your changes** (`git commit -m 'Add amazing feature'`)
4. **Push to the branch** (`git push origin feature/amazing-feature`)
5. **Open a Pull Request**

### Development Guidelines

- Follow [Effective Go](https://golang.org/doc/effective_go) coding style
- Add tests for new functionality
- Update documentation in [`doc/`](doc/) directory
- Ensure all tests pass: `go test ./...`

---

## 📝 License

This project is licensed under the **MIT License**. See [LICENSE](LICENSE) for details.

---

## 📞 Support & Community

For questions, suggestions, or issues:

- **GitHub Issues**: [Report bugs or request features](https://github.com/yourusername/Go2NetSpectra/issues)
- **Discussions**: [Join community discussions](https://github.com/yourusername/Go2NetSpectra/discussions)
- **Documentation**: Check [`doc/`](doc/) for detailed guides

---

## 🙏 Acknowledgments

Go2NetSpectra builds upon these excellent open-source projects

**⭐ If Go2NetSpectra helps your network monitoring, please give it a Star! ⭐**

**🚀 Visit our [documentation](doc/) to get started today!**

Go2NetSpectra

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README