dolphin

KO | EN

gitlite — search

#javascript #hacktoberfest #llm #go #machine-learning #rust #macos #windows #awesome #awesome-list #pytorch #security #mcp #openai

(★ 24)

A Delphi/Object Pascal ML framework: GPT-style Transformers + GPU training + tabular DataFrames

dolphin Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

![WIP](https://img.shields.io/badge/status-work%20in%20progress-yellow?style=for-the-badge)

# Dolphin 🐬

**Dolphin** is a Delphi-native Pascal GPT framework for building and iterating on GPT-style models.  
It targets developers who want full control over tokenization, vocabulary, and neural network logic — implemented in Object Pascal.

![Logserver](https://raw.githubusercontent.com/magnolima/dolphin/main/logserver/html/trainmetrics.jpg)  
Logserver - this UI can be found in the logserver folder

## Features

### Core Architecture
- **GPT-like architecture** implemented entirely in Delphi
- **Modular feed-forward and transformer blocks** for rapid iteration and benchmarking
- **Pre-Norm architecture** with residual connections
- **Multiple activation functions** — GELU, ReLU, Swish, SELU, ELU, LeakyReLU

### GPU Acceleration (CUDA)
- **NVIDIA CUDA support** — Automatic CPU/GPU dispatch based on computational intensity
- **Hybrid precision** — FP64 on CPU (Delphi `Double` / MrMath default) and FP32 on GPU for performance
- **GPU Memory Pool** — Intelligent buffer reuse and memory management
- **Weight caching** — Persistent GPU-resident weights with cache hit tracking
- **Native CUDA kernels** — MatMul, Softmax, GELU, LayerNorm, RoPE, AdamW, and more
- See [CUDA Documentation](Lib/Cuda/README.MD) for detailed kernel reference

### Training Features
- **Training Objectives** — Control pipeline stop points or modes (TokenizerOnly, EmbeddingsOnly, SentenceEmbeddings, PretrainOnly, FullModel, FineTune)
- **True Batch Training** — Efficient batch processing with gradient accumulation
- **Micro Batching** — Memory-efficient training with configurable batch subdivision
- **GPU Auto Batch Size** — Automatic batch size optimization based on available VRAM
- **LLM-driven Hyperparameter Tuning (Optional)** — At epoch end, an external LLM can propose hyperparameter updates (for example `LearningRate`, `GradientClipNorm`, `Dropout`, `WeightDecay`, and micro-batch settings). Requires `LLMTUNNING` and optional `MakerAi` integration.
- **Sequence Modes**:
  - **Packed (Stream)** — Concatenates sequences for maximum throughput
  - **Chunk** — Splits long sequences with configurable stride for overlap
  - **Truncate** — Clips sequences to maximum length
- **Early Stopping** — Monitors validation loss/perplexity with patience and best-weights restoration
- **JSONL TrainingLogs** — Real-time training progress with Python console support

### Data Science & DataFrames
- **TDataFrame Component** — A columnar, Pandas-like DataFrame for data science and ML data loading
- **Native Data Ingestion** — Load directly from any `TDataSet` (e.g., FireDAC `TFDMemTable`, SQL queries) with automatic type mapping and pre-allocation, alongside multi-threaded CSV loading.
- **Structured Data Processing** — Min/Max scaling, Z-Score Normalization, and Label Encoding.
- **Data Shuffling** — Fast Fisher-Yates row shuffling via virtual indices (zero data copying).
- **GPU Integration & Batching** — Direct O(1) batch extraction to `TDeviceMatrix` without intermediate memory copies.
- **IDataProvider Architecture** — Generic `TDataFrameProvider` wraps DataFrames to yield `TTensorBatch` (separate `Features` and `Targets` matrices) for training loops, cleanly decoupling structured data from NLP token pipelines.
- **Zero-Copy Dataset Splits** — Dynamically split DataFrames into Train and Validation views using index slicing without duplicating data in memory.
- **LoRA & Fine-Tuning Ready** — Feeds purely numeric feature/target tensors directly into the network, bypassing token embeddings for structured tabular training.

### Positional Encoding
- **Sinusoidal Positional Encoding** — Classic fixed positional embeddings
- **Rotary Position Embeddings (RoPE)** — Modern relative position encoding with GPU acceleration

### Tokenization & Vocabulary
- **Custom BPE tokenizer** (Byte Pair Encoding) with vocabulary management  
- **SQLite database backend** for persistent vocabulary and merge rules
- **Hugging Face export** — Vocabulary/merges export compatible with Hugging Face tokenizers

### Precision & Memory
- **FP16 Matrix Support** — Half-precision floating point for reduced memory usage
- **Optimized memory management** — Pooled allocations, buffer reuse

### Generation & Inference
- **Advanced Generation** — Top-K, Top-P (Nucleus), and Temperature sampling
- **Greedy decoding** — Deterministic output

### Configuration
- **Flexible Configuration** — JSON-based configuration and CLI arguments
- **Offline and local-friendly** — No external runtimes required
- **Multi-GPU Support** — Specify exactly which GPU the model will use using configuration or query available GPUs at runtime.

### Upcoming features
- LoRA (Low-Rank Adaptation) support for parameter-efficient fine-tuning (train small adapter matrices while keeping base weights frozen). For more information: https://huggingface.co/docs/diffusers/training/lora

## Available Units

| Unit | Description |
| :--- | :--- |
| **PascalGPT.Attention.pas** | Implements **scaled dot-product attention** mechanisms, including Q/K/V projections, Softmax, **causal masking**, and **RoPE** support. |
| **PascalGPT.Attention.Training.pas** | Training-optimized attention with **GPU-accelerated forward/backward passes**, activation checkpointing, and memory-efficient buffer management. |
| **PascalGPT.Attention.Inference.pas** | Inference-optimized attention for **generation** and evaluation. |
| **PascalGPT.BPETokenizer.pas** | **Byte-level Byte Pair Encoding (BPE) processor** for token encoding/decoding and merge rule calculation. |
| **PascalGPT.CUDA.pas** | **NVIDIA CUDA driver core management**, handling context creation, device memory, and kernel launches. |
| **PascalGPT.CUDA.Kernels.pas** | Manages **PTX kernel loading** for GPU operations (MatMul, GELU, Softmax, LayerNorm, RoPE, AdamW, etc.). |
| **PascalGPT.CUDA.Math.pas** | High-level math interface with **adaptive CPU/GPU dispatch**, performance calibration, and **memory pool** management. |
| **PascalGPT.CUDA.BatchBuffers.pas** | **Batch buffer management** for GPU training with efficient memory allocation. |
| **PascalGPT.Cuda.DeviceAttribute.pas** | Helper unit for **CUDA device properties** and capability querying. |
| **PascalGPT.DataFrame.pas** | A **Pandas-like columnar DataFrame** for data science, handling multi-threaded CSV loading, Normalization, Label Encoding, and direct GPU batch extraction. |
| **PascalGPT.Dataset.pas** | Dataset management for processing large text files and organizing `TTrainingExample` records. |
| **PascalGPT.DatasetUtils.pas** | Utilities for text preprocessing, **sequence mode handling** (Packed/Chunk/Truncate), and corpus diversity analysis. |
| **PascalGPT.Embeddings.pas** | **Token and positional embedding layer** with **Xavier/Glorot initialization** and sinusoidal encoding. |
| **PascalGPT.FP16Matrix.pas** | Implements **Half-Precision (FP16) matrix operations** for 50% memory reduction. |
| **PascalGPT.FeedForward.pas** | Feed-forward neural network layer with multiple **activation functions** and GPU acceleration. |
| **PascalGPT.Generation.pas** | **Text generation strategies** — Greedy, Temperature, Top-K, and Top-P (Nucleus) sampling. |
| **PascalGPT.Helper.Configuration.pas** | Manages **JSON configuration files** for model, training, and runtime settings. |
| **PascalGPT.Helpers.pas** | Core utilities including matrix helpers, Softmax, **RoPE functions**, and Safetensors metadata. |
| **PascalGPT.Instructor.pas** | Specialized training loop for **instruction tuning** datasets (like Alpaca). |
| **PascalGPT.IO.SafeTensors.pas** | **Safetensors format** support for model weight import/export. |
| **PascalGPT.Linear.pas** | **Fully Connected/Dense layer** for projections and LM Head. |
| **PascalGPT.Model.pas** | Top-level GPT model structure managing layer orchestration and forward/backward passes. |
| **PascalGPT.Optimizer.pas** | **Adam and AdamW optimization** with GPU-accelerated optimizer step. |
| **PascalGPT.Storage.pas** | **SQLite database backend** for vocabulary, BPE rules, and embeddings persistence. |
| **PascalGPT.TestMatMul.pas** | Testing and benchmarking utilities for **matrix multiplication performance** validation. |
| **PascalGPT.Training.pas** | Training loop implementation with **micro-batching**, **early stopping**, and progress callbacks. |
| **PascalGPT.Training.Logger.pas** | **JSONL training logger** for real-time metrics and external monitoring. |
| **PascalGPT.Training.VRAMOptimization.pas** | **VRAM optimization strategies** including gradient accumulation and memory-efficient training. |
| **PascalGPT.TransformerBlock.pas** | Core transformer block with **Layer Normalization** and **residual connections** (Pre-Norm). |
| **PascalGPT.Types.pas** | Central repository for **common types**, configuration records, and constants. |
| **PascalGPT.Vocabulary.pas** | **Token-to-ID/ID-to-token mapping** with BPE merge rule integration. |

## Dependencies

- Required: **[MrMath](https://github.com/mikerabat/mrmath)**, a high-performance numerical library for Delphi.  
  It provides matrix operations, linear algebra, and numerical routines similar in purpose to Python's **NumPy**, serving as the computational backbone for neural and tokenization modules.
- Optional (only when using LLM Optimizer): **[MakerAi](https://github.com/gustavoeenriquez/MakerAi)**.
- LLM Optimizer usage note:
  - Enable `LLMTUNNING` only if you want LLM-driven hyperparameter proposals.
  - If you do not want the optimizer, leave `LLMTUNNING` undefined and keep `UseLLMOptimizer` disabled.

## Architecture Overview

```
           ┌─────────────────────────────┐
           │         Input Text          │
           └──────────────┬──────────────┘
                          │
                          ▼
               ┌────────────────────┐
               │  BPE Tokenizer     │
               │ (PascalGPT.BPETokenizer) │
               └────────────────────┘
                          │
                   Tokens (IDs)
                          │
                          ▼
             ┌──────────────────────────┐
             │   Embedding Layer        │
             │ (PascalGPT.Embeddings)   │
             │  + Sinusoidal / RoPE     │
             └──────────────────────────┘
                          │
                          ▼
          ┌────────────────────────────────┐
          │       Transformer Block         │
          │ (PascalGPT.TransformerBlock)    │
          │  ├─ Attention (multi-head)      │
          │  │   └─ RoPE / Causal Mask      │
          │  └─ FeedForward (GELU/ReLU)     │
          └────────────────────────────────┘
                          │
                          ▼ (N layers)
                          │
                          ▼
             ┌──────────────────────────┐
             │   Output Layer / Logits  │
             │(Prediction of next token)│
             └──────────────────────────┘
                          │
                          ▼
           ┌─────────────────────────────┐
           │     Decoded Text Output     │
           └─────────────────────────────┘  
```

## Training Configuration

![Logserver](https://raw.githubusercontent.com/magnolima/dolphin/main/logserver/html/trainsettings.jpg)  
Logserver - this UI can be found in the logserver folder
### Sequence Modes

| Mode | Description |
|------|-------------|
| **Truncate** | Clips sequences to `MaxSeqLen`. Simple but may lose data. |
| **Chunk** | Splits long sequences into overlapping chunks using `ChunkStride`. Preserves all data. |
| **Packed** | Concatenates multiple sequences into a single stream for maximum GPU utilization. Most efficient. |

### Early Stopping

```json
{
  "EarlyStopping": {
    "Enabled": true,
    "Monitor": "val_loss",
    "Mode": "min",
    "MinDelta": 0.01,
    "Patience": 3,
    "MinEpochs": 5,
    "RestoreBestWeights": true
  }
}
```

### Micro Batching

Enables training with large effective batch sizes on limited VRAM:

```json
{
  "Training": {
    "MicroBatchSize": 4,
    "MicroBatchMaxTokens": 2048
  }
}
```

### Training Objectives

Control where the training pipeline stops, enabling specialized workflows for different use cases:

| Objective | Description | Exports | Model Created |
|-----------|-------------|---------|---------------|
| **FullModel** | Default. Complete pipeline with full model training | safetensors, config.json, vocab | ✅ Yes |
| **TokenizerOnly** | Stop after BPE tokenizer training | vocab.json, merges.txt | ❌ No |
| **EmbeddingsOnly** | Stop after embedding training | embeddings.npy, embeddings.cached, vocab | ❌ No |
| **SentenceEmbeddings** | Train sentence embeddings (Contrastive Loss) | safetensors, config.json, vocab | ✅ Yes |
| **PretrainOnly** | Train model but save as checkpoint | safetensors (checkpoint) | ✅ Yes |
| **FineTune** | Same as FullModel (starts from pretrained weights) | safetensors, config.json, vocab | ✅ Yes |

**Configuration:**

```json
{
  "General": {
    "TrainingObjective": "EmbeddingsOnly"
  }
}
```

**Use Cases:**
- `TokenizerOnly` — Build vocabulary for external tools
- `EmbeddingsOnly` — Generate raw token embeddings
- `SentenceEmbeddings` — Train model for semantic search (Contrastive Learning). *Unlike Normal LLMs that predict the next word for generation, Sentence Embeddings compress full sentences into meaning vectors to measure similarity between them.*
- `PretrainOnly` — Create checkpoints for continued training
- `FullModel` — Production-ready normal LLM models with full training (predicting the next word).

If not specified, defaults to `FullModel` (no breaking changes to existing configs).

## Getting Started

1. Clone the repository  
   ```bash
   git clone https://github.com/magnolima/PascalGPT/PascalGPT.git
   ```

2. Open `Demos/pascalgpt.trainer.dpr` in Delphi 12+ and build the project (Release mode recommended).

3. Run the demo with a dataset:
   ```bash
   pascalgpt.trainer.exe [-c config.json] [-train config.train.json] [-listgpu] dataset.txt
   ```

- `-c`: Main configuration file (model params, paths).
   - `-train`: Training configuration overrides (epochs, batch size).
   - `-listgpu`: Lists available CUDA GPU devices and their IDs, then exits.
   - `dataset.txt`: Path to your training corpus.

You can select a specific GPU by defining `DefaultGPUIndex` inside the `"General"` section of `config.json`.

## GPU Requirements

For CUDA acceleration:
- **NVIDIA GPU** with Compute Capability 6.0+ (Pascal or newer)
- **CUDA Toolkit** installed (for PTX compilation)
- **nvcuda.dll** available in system PATH

The framework automatically falls back to CPU if CUDA is unavailable.

dolphin

dolphin Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

dolphin

dolphin Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README