DeukPack

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 19 Open GitHub ↗

DeukPack — AI-Ready Interface Hub & Universal IDL Gateway. Unified serializer for Mixed-IDL (Protobuf, OpenAPI, Thrift, .deuk) with AI Semantic Mapping & MCP Guardrails. Ultra-fast codegen for Python, C#, C++, TS/JS, Elixir.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DeukPack: AI-Native Universal Schema Multi-hub

> ⚠️ [NOTICE] Large-Scale Internal Refactoring in Progress
>
> During recent integration testing across various languages, 
> several defects were identified, prompting a large-scale internal refactoring. 
> I am intentionally pivoting my focus 'from extreme performance optimizations 
> to ensuring architectural stability and flexibility'. 
> My primary goal is to carefully review the specific characteristics of 
> each language and protocol to improve and fix omissions from the initial implementation. 
> Please keep in mind that there may be unexpected issues during use 
> until this work is complete.

> **Universal Schema Multi-hub for Protobuf, Thrift, OpenAPI, and .deuk. (High-Performance)**

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![npm version](https://img.shields.io/npm/v/deukpack.svg)](https://www.npmjs.com/package/deukpack)
[![npm downloads](https://img.shields.io/npm/dm/deukpack.svg)](https://www.npmjs.com/package/deukpack)
[![GitHub stars](https://img.shields.io/github/stars/joygram/DeukPack.svg?style=social)](https://github.com/joygram/DeukPack/stargazers)

**Languages / 언어:** [English](README.md) · [한국어](README.ko.md)

Turn **any IDL** (Protobuf, OpenAPI, JSON Schema, `.deuk`) into **type-safe, deterministic code** across C#, C++, TypeScript, JavaScript, Java, and Elixir — with a single, unified serialization API.

---

### 📢 [v1.9.0 Update] Official Python (Pure/Rust) Support and Industry Standard BMT Benchmark Revamp
We are pleased to announce the addition of the **official Python (3.6+) binary engine (Pure/Rust Accelerated)** to the DeukPack ecosystem. Furthermore, to ensure objective reliability in enterprise environments, we have discarded legacy measurement methods and elevated the benchmark environments across all languages to **industry-standard frameworks** (such as `BenchmarkDotNet`, `mitata`, and `pytest-benchmark`).

> You can review the newly validated, highly transparent and strict performance metrics—measured under real-world complex object conditions—in our updated [Performance Matrix White Paper](docs/DEUKPACK_GC_PERFORMANCE_MATRIX.md).

- **Unified API:** You no longer need to call verbose factory methods. All languages now intuitively share the same 2-Method struct syntax: **`Hero.Pack()`** and **`Hero.Unpack()`**. (Legacy APIs are temporarily maintained for backward compatibility).
> - **⚠️ CAUTION (C# / Unity Users):** If you manually copy generated `.cs` runtime files, you **MUST completely empty that legacy folder** prior to importing the new files to prevent duplicate declaration compile errors. (npm / UPM users are automatically updated).

---

## Why DeukPack: The AI-Ready Advantage

### 1. Universal IDL Gateway (OpenAPI, JSON Schema, Protobuf, Thrift, CSV)
Modern systems often struggle with a fragmented mix of Specs—legacy (Thrift), modern (Protobuf/gRPC), and web-native (OpenAPI/JSON Schema). DeukPack acts as a **Single Source of Truth** that aggregates diverse IDL sources into one unified, semantic model — wire-compatible with your legacy protocols.

### 2. IDL-to-AI Semantic Mapping

DeukPack extracts metadata from IDL comments and field structures, transforming them into a **Semantic Context** that AI can instantly grasp. Designers evolve into architects defining **data lineage** via a machine-readable semantic layer.

### 3. AI-Native Execution Bridge (MCP Plugin Support)

The **Model Context Protocol (MCP)** server auto-generation feature (`DeukPackMcp`) lets AI agents (Cursor, Claude, etc.) browse live documentation and execute backend methods directly.

### 4. Core Memory Optimization (In-place Reuse)

Engineered for high efficiency. **60–100% reduction in memory allocation** and **250% increase in JS parsing speed** vs. classic industry flows. See the [Performance](#-performance-the-zero-bottleneck-foundation) section for raw numbers.

---

## ⚡ Two Words. Every Language.

DeukPack v1.7.6 introduces a **universal 2-method serialization API**: **`Pack`** and **`Unpack`**.  
Regardless of language or wire format — binary, JSON, In-place — you only need to remember two verbs.

```
Pack    → Serialize (data out)
Unpack  → Deserialize (data in)
```

Pass a **format parameter** to switch protocols. Call `Unpack` on an **existing instance** for in-place overwrite.  
That's the entire API surface.

> **Unity / C# Notice (GC Spike Prevention):**
> NEVER use `var h = Hero.Unpack(bin);` (Factory method) for high-frequency network packets (Hotpath). It implicitly triggers `new` allocations, causing GC spikes and severe frame drops.
> You MUST use **`Hero.Unpack(cachedHero, bin);`** to overwrite pre-allocated (pooled) objects if you want to achieve true in-place architecture without GC stutters.

```csharp
// C# / Unity: 1.Create  2.Pack  3.Unpack (In-place)
var hero = new Dto.Hero { id = 1, name = "Deuk" };
byte[] bin = Dto.Hero.Pack(hero);          // Serialize (Static)
Dto.Hero.Unpack(hero, bin);                // In-place Update
```

```typescript
// TypeScript / JavaScript: 1.Create  2.Pack  3.Unpack (No Class Wrappers)
const hero = Dto.Hero.create({ id: 1, name: "Deuk" });
const bin = Dto.Hero.pack(hero);           // Serialize
Dto.Hero.unpack(hero, bin);                // In-place Update
```

```cpp
// C++ (Native): 1.Create  2.Pack  3.Unpack (Memory Safe)
Dto::Hero hero; hero.id = 1; hero.name = "Deuk";
auto bin = Dto::Hero::Pack(hero);          // Serialize
Dto::Hero::Unpack(hero, bin);              // In-place Deserialize
```

```java
// Java: 1.Create  2.Pack  3.Unpack (High-Performance)
Dto.Hero hero = new Dto.Hero(1, "Deuk");
byte[] bin = Dto.Hero.pack(hero);          // Serialize (Static)
Dto.Hero.unpack(hero, bin);                // In-place Overwrite (Static)
```

```elixir
# Elixir: 1.Create  2.Pack  3.Unpack (BEAM Native)
hero = %Dto.Hero{id: 1, name: "Deuk"}      # Immutable Struct
bin = Dto.Hero.pack(hero)                  # Serialize
hero_parsed = Dto.Hero.unpack(bin)         # BEAM Pattern Match
```

---

### 🚀 Quick Start

```bash
npx deukpack init
```

**1. Define your schema (or import OpenAPI / Protobuf)**

```deuk
namespace Dto

struct Hero {
    1> int32 id
    2> string name
    3> float hp
}
```

---

### 🔄 Backward Compatibility — Existing Code is Safe

All **legacy method names are preserved as deprecated aliases**. No breaking changes.

Your existing project will compile without errors. IDE will show **soft deprecation warnings** to guide migration at your own pace.

---

### 🎮 Real-World Pattern: Unity Game Client (In-place Reuse)

```csharp
Dto.Hero cachedHero = new Dto.Hero(); // Allocated exactly ONCE at startup

void OnNetworkMessage(byte[] inputData) {
    // Zero-Garbage Deserialization — NO new class allocations, NO GC spikes!
    cachedHero.Unpack(inputData);
    Debug.Log($"Hero: {cachedHero.name}, HP: {cachedHero.hp}");

// Mutate and re-serialize (Note: byte[] reuse requires Stream API)
    cachedHero.hp -= 10f;
    byte[] outputData = Dto.Hero.Pack(cachedHero);
    network.Send(outputData);
}
```

---

## 🚀 Release Roadmap

---

## ⚡ Performance: The Zero-Bottleneck Foundation

> **Notice: Benchmark Data Under Revision**
> We have identified an inconsistency between the highly-optimized protocol implementations and the aggregated result table. The performance metrics matrix is currently being revised and will be republished once the evaluation is complete.

> Figures based on decoding a 10,000-row payload. Results vary by environment.  
> [Detailed cross-language matrix](docs/DEUKPACK_GC_PERFORMANCE_MATRIX.md) · [Benchmarking Guide](docs/DEUKPACK_BENCHMARK_SCENARIO.md)

---

## Feature Matrix

| Category | Feature | TS / JS | C# / Unity | C++ | Java | Elixir | Python |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **IDL Core** | Basic Types / Aliases | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **Inheritance** | `extends` support | ✅ | ✅ | ✅ | ✅ (v1.5) | ✅ | ✅ |
| **Unified API** | `Pack` / `Unpack` (2-method) | ✅ v1.8.0 | ✅ v1.8.0 | ✅ v1.8.0 | ✅ v1.8.0 | ✅ v1.8.0 | ✅ v1.9.0 |
| **Protocols** | Native Pack (.dpk) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Pure/Rust)|
| | Protobuf Compatible | ✅ | ✅ | ✅ | ✅ | - | 🚧 |
| | Thrift Compatible (T-Series) | ✅ | ✅ | ✅ (v1.5) | ✅ (v1.5) | - | - |
| | JSON (Tagged / POJO) | ✅ | ✅ | ✅ (v1.5) | ✅ | - | ✅ |
| | YAML / CSV | ✅ | ✅ (v1.2.7) | 🚧 | 🚧 | - | - |
| **Optimizations**| In-place / JIT | ✅ (v1.6) | ✅ | ✅ (v1.4.2) | 🚧 | ✅ (BEAM) | ✅ (v1.9) |
| | `Write` Logic Overrides | ✅ | ✅ | ✅ (v1.5) | ✅ (v1.5) | - | - |
| **Data/Meta** | `tablelink` / MetaTable | ✅ | ✅ | ✅ (v1.5) | ✅ | - | - |
| **AI Integration** | MCP Tool Auto-Generation | ✅ (v1.5) | 🚧 | - | - | - | - |
| | Intelligent Context | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| | IDE IntelliSense | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

---

## Installation

```bash
npm install deukpack
npx deukpack init
npx deukpack run         # default: ./deukpack.pipeline.json
```

Tutorials and OS-specific notes: **[deukpack.app/tutorial](https://deukpack.app/tutorial/)**.

---

## 🛡️ Security & Reliability (OOM Defense / Anti-DDoS)

DeukPack implements strict defense-in-depth against **network-layer parsing vulnerabilities (OOM, Buffer Flooding, Infinite Recursion)**:

- **Universal OOM Defense (v1.7.0+)**: Enforced `MAX_SAFE_LENGTH` (10MB) and `MAX_ELEMENT_COUNT` (1,000,000) limits across all engines. Malicious packets are discarded (Fail-Fast) before any memory allocation.
- **Progressive Chunk Validation**: Replaces legacy `ReadToEnd()` with length pre-evaluation, neutralizing JSON stack bombing in Node.js and Java backends.
- **Continuous DDoS Fuzzer Suite**: CI-integrated `test-fuzz-oom.js` injects parsers with 2GB+ abnormal buffers.

---

## Documentation & Links

**Contact:** contact@deukpack.app

---

## Development

```bash
npm ci
npm run build
npm test
npm run benchmark                     # Node serialize smoke
npm run bench:format-parity           # parser comparison
npm run bench:cross-lang              # Node vs .NET pack
```

---

## ☕ Support & Contact

DeukPack is completely open-source (Apache 2.0). Built to solve fundamental GC bottleneck and model synchronization issues from 30 years of server architecture experience.

- 📩 **Contact / Technical Inquiries**: joygram@gmail.com
- ☕ **Support the Project**: [Sponsor via Ko-fi](https://ko-fi.com/joygram)

**Starring the repo** or sharing it with teams juggling Protobuf/Thrift helps immensely.

---

## Works well with (Deuk Family)

**Want agents to do more with your specs?** Use **DeukPack** — IDL in, deterministic types and serializers out.  
**Want agents to behave predictably in your repo?** Use **[DeukAgentRules](https://github.com/joygram/DeukAgentRules)** — versioned `AGENTS.md` and rule templates.

```bash
npm install -D deuk-agent-rule
npx deuk-agent-rule init --non-interactive
```

---

## ⚡ Performance & Benchmarks

> **Notice: Benchmark Data Under Revision**
> We have identified an inconsistency between the highly-optimized protocol implementations and the aggregated result table. The performance metrics matrix is currently being revised and will be republished once the evaluation is complete.

To ensure objective performance testing, we have adopted industry-standard benchmarks such as `BenchmarkDotNet`, `mitata`, and `pytest-benchmark`. Achieving **60-100% reduction in memory allocation** and **2.5x to 5x faster parsing speeds** compared to legacy industry formats. If you find any errors or have suggestions regarding the benchmark scenarios, please let us know.

- [Performance test technical document](docs/DEUKPACK_GC_PERFORMANCE_MATRIX.md)

---

## Ecosystem (Deuk Family)

## Contributing

1. Fork → feature branch → PR.
2. See [RELEASING.md](RELEASING.md) for release layout.

---

## License

**Apache License 2.0** — [LICENSE](LICENSE) · [NOTICE](NOTICE).

---

## Acknowledgments

The broader IDL / OpenAPI / schema communities; DeukPack is a **standalone pipeline** (not an Apache Thrift subproject).

DeukPack

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README