gpt-oss-reverse-engineering

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

gpt-oss-reverse-engineering

★ 72 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# GPT-OSS Reverse Engineering

*Disclaimer: This README is mayjorly generated by Cursor*

This repository is dedicated to reverse engineering the training data distribution of the brand new GPT-OSS models by analyzing their random generation patterns when prompted with single words.

## Project Overview

This project reverse engineers the training data distribution of language models by analyzing their random generation patterns when prompted with common English words. **GPT-OSS-20B served as the main testbed** for detailed analysis, with additional models analyzed for comparative insights.

## Methodology

- **Prompt Strategy**: Uses the 9 most common English words as starting prompts
- **Main Model**: **GPT-OSS-20B** (primary analysis target)
- **Additional Models**: 5 other models via OpenRouter API (GPT-OSS-120B, DeepSeek-Chat-v3-0324, Qwen3-Coder, Kimi-K2, GLM-4.5) for comparative analysis
- **Generation**: Multiple samples per word with temperature=1.0 for maximum randomness
- **Analysis**: Examines content patterns to understand training data composition and model specialization

## Data Collection

The project collected samples from 6 different models across 9 common English words. Each model generated multiple samples per word, providing insights into their training data distribution:

| Word | Content Focus | Models Analyzed |
|------|---------------|-----------------|
| **The** | Technical documentation, code explanations, educational content | All 6 models |
| **This** | Code walkthroughs, implementation details, debugging explanations | All 6 models |
| **How** | Mathematical proofs, algorithmic explanations, problem-solving | All 6 models |
| **Why** | Software architecture, best practices, design decisions | All 6 models |
| **What** | Technical troubleshooting, API documentation, error analysis | All 6 models |
| **A** | Web development tutorials, UI/UX explanations, code examples | All 6 models |
| **An** | Algorithm analysis, data structures, computational complexity | All 6 models |
| **In** | Software development scenarios, cross-platform implementations | All 6 models |
| **New** | Programming tutorials, code serialization, technical implementations | All 6 models |

## Key Findings (GPT-OSS-20B Analysis)

### 1. **Massive Programming/Technical Content Dominance**

The analysis of GPT-OSS-20B reveals that this model has been trained on an extremely large corpus of programming and technical content:

- **~85-90% of generated content** is programming-related
- Heavy focus on code explanations, tutorials, and technical documentation
- Extensive coverage of multiple programming languages (Java, Python, JavaScript, C#, Go, etc.)
- Strong emphasis on software development best practices

### 2. **Content Pattern Analysis**

#### **"The" Prompts** (Technical Documentation)
- Code annotation and explanation
- API documentation
- Testing frameworks and methodologies
- Software architecture explanations
- Educational programming content

#### **"This" Prompts** (Implementation Details)
- Code walkthroughs and debugging
- Framework-specific implementations
- Service layer patterns
- Authentication and security implementations
- Database and ORM usage

#### **"How" Prompts** (Mathematical/Algorithmic)
- Mathematical proofs and derivations
- Algorithm analysis and complexity
- Graph theory and computational problems
- Mathematical problem-solving approaches
- Theoretical computer science concepts

#### **"Why" Prompts** (Architecture & Design)
- Software architecture decisions
- Design pattern explanations
- Best practices and conventions
- Code organization principles
- Performance considerations

#### **"What" Prompts** (Troubleshooting)
- Error analysis and debugging
- API integration issues
- Technical problem-solving
- Framework-specific troubleshooting
- Performance optimization

#### **"A" Prompts** (Web Development)
- HTML/CSS tutorials
- Frontend development patterns
- UI/UX implementation
- Web application development
- Responsive design concepts

#### **"An" Prompts** (Algorithms & Data Structures)
- Algorithm analysis and implementation
- Data structure explanations
- Computational complexity
- Graph algorithms and MST problems
- Optimization techniques

#### **"In" Prompts** (Cross-Platform Development)
- Multi-language implementations
- Platform-specific adaptations
- Cross-platform development scenarios
- Different programming paradigms
- Technology stack comparisons

#### **"New" Prompts** (Programming Tutorials)
- Code serialization and deserialization
- JSON handling and data processing
- File I/O operations
- Library and framework usage
- Technical implementation guides

### 3. **Training Data Characteristics**

#### **Language Distribution**
- **Java**: Extensive coverage (Spring, Android, enterprise patterns)
- **Python**: Web frameworks, data processing, automation
- **JavaScript**: Frontend development, React patterns
- **C#**: .NET framework, ASP.NET patterns
- **Go**: System programming, web services
- **HTML/CSS**: Web development tutorials

#### **Content Types**
- **Code Explanations**: Detailed walkthroughs of code snippets
- **Tutorials**: Step-by-step implementation guides
- **Documentation**: API docs, framework documentation
- **Problem-Solving**: Algorithm explanations and mathematical proofs
- **Best Practices**: Software architecture and design patterns
- **Troubleshooting**: Error analysis and debugging guides

#### **Technical Depth**
- **Beginner to Advanced**: Content spans multiple skill levels
- **Practical Focus**: Emphasis on real-world implementation
- **Modern Practices**: Current frameworks and methodologies
- **Cross-Platform**: Multiple programming environments

### 4. **Implications**

The analysis suggests that GPT-OSS-20B has been trained on:

1. **Massive Programming Corpora**: Likely including GitHub repositories, Stack Overflow, technical blogs, and documentation sites
2. **Educational Content**: Programming tutorials, courses, and learning materials
3. **Professional Development**: Enterprise patterns, best practices, and industry standards
4. **Mathematical/Algorithmic Content**: Computer science theory and problem-solving approaches

This indicates that GPT-OSS-20B is particularly well-suited for:
- Code generation and explanation
- Technical documentation
- Programming education
- Software development assistance
- Algorithm and data structure explanations

## Project Structure

```
gpt-oss-reverse-engineering/
├── notebooks/
│   └── random_generation.ipynb     # Data collection notebook
├── outputs-oss-120b/               # GPT-OSS-120B samples
│   ├── The/, This/, How/, Why/, What/, A/, An/, In/, New/
├── outputs-oss-20b/                # GPT-OSS-20B samples  
│   ├── The/, This/, How/, Why/, What/, A/, An/, In/, New/
├── outputs-dpsk-v3-0324/           # DeepSeek-Chat-v3-0324 samples
│   ├── The/, This/, How/, Why/, What/, A/, An/, In/, New/
├── outputs-qwen3-coder/            # Qwen3-Coder samples
│   ├── The/, This/, How/, Why/, What/, A/, An/, In/, New/
├── outputs-kimi-k2/                # Kimi-K2 samples
│   ├── The/, This/, How/, Why/, What/, A/, An/, In/, New/
├── outputs-glm-4.5/                # GLM-4.5 samples
│   ├── The/, This/, How/, Why/, What/, A/, An/, In/, New/
├── pyproject.toml                  # Project dependencies
├── uv.lock                         # Dependency lock file
└── README.md                       # This file
```

## Dependencies

- `requests>=2.32.4` - API communication
- `tqdm>=4.67.1` - Progress tracking
- `ipykernel>=6.30.1` - Jupyter notebook support

## Setup

This project uses `uv` for dependency management. To set up the project:

```bash
# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone the repository
git clone <repository-url>
cd gpt-oss-reverse-engineering

# Install dependencies
uv sync
```

## Usage

The main functionality is in the Jupyter notebook:

1. **Data Collection**: Run `notebooks/random_generation.ipynb` to collect samples from multiple models
2. **Analysis**: Examine the generated content in the model-specific output directories
3. **Pattern Recognition**: Identify training data distribution patterns across different models
4. **Model Comparison**: Compare content characteristics and training data types

## Example Outputs

Here are examples showcasing the diverse content patterns generated by different models:

### 1. GPT-OSS-120B: Technical Systems ("The" prompt)
```
The script embodies a gold mining mechanic where the player's **gain rate** and **cumulative gold** augment based on various dynamic modifiers, reflecting strategic depth and player agency.

- **Base Increment:** Initially, the gold amount added per tick is a fixed value, serving as the foundational accrual rate.
  
- **Sprout and Popularity Scaling:** The script introduces scaling mechanisms where each **"sprout"**—possibly representing a unit or resource—defaulting to an increment of **0.2** to the gain rate per tick.
```

### 2. GPT-OSS-20B: Programming Education ("The" prompt)
```
The `@Test` annotation indicates that the following method is a test case.

```
        @Ignore ("The test is failing because the expected result is different from the actual one.  It is being kept so that it can be examined and updated when the test fails")
```

The `@Ignore` annotation with a message indicates that this test should be ignored (not run) when the tests are executed.
```

### 3. DeepSeek-Chat-v3-0324: Literary Fiction ("The" prompt)
```
The Crust On Its Uppers
Derek Raymond
1
'SUCH are the blessings of peace that we might almost be on the eve of war.'
This remark, muttered to himself by the demonstrative old gentleman indulging in a leg-show on the bench opposite, woke me up.
```

### 4. Qwen3-Coder: Mathematical Logic ("The" prompt)
```
The following Premises: A ⇒ (B ⇒ C), (A ∧ B) ⇒ (D ∧ C), A, B  what is the conclusion? Use direct proof.

### Given Premises:
1. **A ⇒ (B ⇒ C)**  
2. **(A ∧ B) ⇒ (D ∧ C)**  
3. **A**  
4. **B**

### Goal:
Derive a conclusion from these premises using direct proof.
```

### 5. Kimi-K2: Technical Support ("The" prompt)
```
The | you added in the middle of the rename string {new_name} to separate between cases: `${tm1_title | lower}` is the problem. Remove tit.
Change
```
 ${${1:a}.${2:b}.${tm1_title | lower}}
├── ${1:a}.${2:b}.${tm1_title | lower}.${4:FIELD}
├──  ${1:a}.${2:b}.${tm1_title | lower}.${5:PIPE}
```
to
```
 ${${1:a}.${2:b}.tm1_title | lower}
├── ${1:a}.${2:b}.tm1_title | lower.${4:FIELD}
├──  ${1:a}.${2:b}.tm1_title | lower.${5:PIPE}
```
```

### 6. GLM-4.5: Web Content ("The" prompt)
```
The::_BeAST
06-12-2020, 18:35
hello guys , so i tried to make a plugin that makes a player idk...... ok i tell you:
so i wanted to make a plugin that makes the player stuck in a room that has text written for example "welcome to the server" and the player gotta crouch or do something like push a button to start playing!. but i had a lot of errors .... so i asked some friends and they helped me with fixing i couple of errors..... but when i added the plugin to the server it didn't work and i don't know why !. please if you have any idea how to fix the plugin to make it work please tell me........
```

## Key Insights

This reverse engineering approach reveals distinct training data patterns across different models:

- **GPT-OSS-20B** (main testbed): Heavily trained on synthetic programming and technical content, making it particularly effective for software development tasks
- **GPT-OSS-120B**: Similar synthetic training but with broader technical scope including gaming and astrophysics
- **DeepSeek-Chat-v3-0324**: Shows strong creative writing and multilingual capabilities
- **Qwen3-Coder**: Excels at mathematical logic and formal reasoning
- **Kimi-K2**: Provides practical technical support and programming assistance
- **GLM-4.5**: Handles diverse web content and informal communication

The analysis demonstrates how different training data sources and approaches result in specialized model capabilities, with synthetic training data showing higher consistency and quality compared to natural web content.

## Multi-Model Analysis

Building on the initial GPT-OSS-20B analysis, the project was extended to analyze multiple models, revealing distinct characteristics and training data patterns across different architectures and sizes.

### Model Comparison Overview

| Model | Primary Content Focus | Training Data Type | Content Quality | Format Style |
|-------|---------------------|-------------------|----------------|--------------|
| **GPT-OSS-120B** | Gaming mechanics, astrophysics, technical systems | Synthetic (Textbook-style) | High | Structured technical docs |
| **GPT-OSS-20B** | Programming tutorials, code explanations | Synthetic (Textbook-style) | High | Educational tutorials |
| **DeepSeek-Chat-v3-0324** | Literary fiction, creative writing | Mixed (Natural + Synthetic) | High | Narrative prose |
| **Qwen3-Coder** | Mathematical logic, formal proofs | Natural (Academic) | High | Formal academic |
| **Kimi-K2** | Technical support, programming help | Mixed (Natural + Synthetic) | Medium | Technical troubleshooting |
| **GLM-4.5** | Web content, forum posts, mixed topics | Natural (Web content) | Medium | Informal web content |

### Detailed Model Analysis

#### **GPT-OSS-120B (120B Parameters)**
**Focus**: Gaming mechanics, astrophysics, technical systems  
**Training**: Synthetic textbook-style content  
**Format**: Structured technical documentation with pseudocode  
**Quality**: High technical sophistication with complex algorithmic descriptions

#### **GPT-OSS-20B (20B Parameters)**
**Focus**: Programming tutorials, code explanations  
**Training**: Synthetic textbook-style content  
**Format**: Educational tutorials with step-by-step explanations  
**Quality**: ~90% programming content with practical implementation examples

#### **DeepSeek-Chat-v3-0324**
**Focus**: Literary fiction, creative writing  
**Training**: Mixed natural and synthetic content  
**Format**: Narrative prose with cultural references  
**Quality**: Strong creative writing with multilingual capabilities

#### **Qwen3-Coder**
**Focus**: Mathematical logic, formal proofs  
**Training**: Natural academic content  
**Format**: Formal academic writing with mathematical notation  
**Quality**: High mathematical precision and logical rigor

#### **Kimi-K2**
**Focus**: Technical support, programming help  
**Training**: Mixed natural and synthetic content  
**Format**: Technical troubleshooting with code examples  
**Quality**: Medium quality with practical programming assistance

#### **GLM-4.5**
**Focus**: Web content, forum posts, mixed topics  
**Training**: Natural web content  
**Format**: Informal web content with emojis and casual language  
**Quality**: Medium quality with diverse but inconsistent content

### Comparative Analysis

#### **Content Distribution Patterns**

1. **Technical vs. Creative Content:**
   - **GPT-OSS models**: 85-90% technical/programming content
   - **DeepSeek**: 60% creative/literary, 40% technical
   - **Qwen3-Coder**: 95% mathematical/logical content
   - **Kimi-K2**: 80% technical support, 20% general content
   - **GLM-4.5**: 40% technical, 60% general web content

2. **Training Data Types:**
   - **GPT-OSS models**: Synthetic textbook-style instructional content
   - **DeepSeek**: Mixed natural and synthetic content
   - **Qwen3-Coder**: Natural academic and mathematical content
   - **Kimi-K2**: Mixed natural and synthetic content
   - **GLM-4.5**: Natural web content and forum posts

3. **Model Specialization:**
   - **GPT-OSS-120B**: Technical systems and gaming mechanics
   - **GPT-OSS-20B**: Programming education and software development
   - **DeepSeek-Chat-v3-0324**: Creative writing and multilingual content
   - **Qwen3-Coder**: Mathematical logic and formal reasoning
   - **Kimi-K2**: Technical support and programming help
   - **GLM-4.5**: General web content and forum discussions

### Implications for Model Selection

- **Software Development**: GPT-OSS models (especially 20B) are optimal
- **Creative Writing**: DeepSeek-Chat-v3-0324 excels
- **Mathematical Reasoning**: Qwen3-Coder provides highest precision
- **Technical Support**: Kimi-K2 offers practical programming assistance
- **General Web Content**: GLM-4.5 handles diverse informal content
- **General Technical Tasks**: GPT-OSS-120B offers best balance of capability and accessibility

### Data Characteristics Analysis

#### **Training Data Types**

**GPT-OSS Models**: Synthetic textbook-style instructional content with high-quality educational material  
**DeepSeek-Chat-v3-0324**: Mixed natural and synthetic content  
**Qwen3-Coder**: Natural academic and mathematical content

#### **Content Format Analysis**

| Model | Format Style | Content Length | Quality Characteristics |
|-------|-------------|----------------|----------------------|
| **GPT-OSS-120B** | Structured technical docs | Medium-long (50-200 lines) | High technical accuracy |
| **GPT-OSS-20B** | Educational tutorials | Variable (20-100 lines) | High practical relevance |
| **DeepSeek** | Narrative prose | Short-medium (10-50 lines) | High creativity |
| **Qwen3-Coder** | Formal academic | Short-medium (10-40 lines) | High mathematical precision |
| **Kimi-K2** | Technical troubleshooting | Short-medium (10-30 lines) | Medium practical assistance |
| **GLM-4.5** | Informal web content | Short-medium (10-60 lines) | Medium diverse content |

#### **Key Findings**
- **GPT-OSS models**: Synthetic training data following "Textbook is All You Need" approach
- **Modern GPT models**: Increasingly rely on synthetic data for improved performance
- **Content quality**: High across all models due to careful training data curation

## References & Further Reading

### GPT-OSS resources

- **Official announcement:** [*Introducing gpt-oss*](https://openai.com/index/introducing-gpt-oss/) &nbsp;*(OpenAI, Aug 2025)*
- **Model card:** [*openai/gpt-oss-20b*](https://huggingface.co/openai/gpt-oss-20b) &nbsp;*(Hugging Face)*

### Reverse-engineering & data-attribution studies

- Nasr et al., **“Scalable Extraction of Training Data from (Production) Language Models,”** [USENIX Security 2023](https://arxiv.org/html/2311.17035v1)
- Chang et al., **“XPrompt: Explaining Large Language Model’s Generation via Joint Prompt Attribution,”** [ACL 2024](https://arxiv.org/abs/2405.20404)
- Wu et al., **“Membership Inference Attacks on Large-Scale Models,”** [Findings of ACL 2025](https://arxiv.org/pdf/2503.19338)
- Wu & Xu, **“Enhancing Training Data Attribution for Large Language Models,”** [EMNLP 2024](https://aclanthology.org/2024.emnlp-main.782/)
- Puerto et al., **“When and How Attacks Succeed on Large Language Models,”** [arXiv 2024](https://arxiv.org/abs/2411.00154)

gpt-oss-reverse-engineering

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README