KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agents #developer-tools #typescript #codex #cli #claude-code #mcp #rust #deepseek-harness #ai-agent

zero-to-ai-researcher

★ 55 Open GitHub ↗

Research on training an LLM with DeepSeek & Kimi architecture

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Zero to AI Researcher: A Complete Learning Journey

[![Discord](https://img.shields.io/badge/Discord-Join%20Community-7289DA?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/6AbXGpKTwN)

This repository contains a comprehensive course in the `_course` folder and scientific research on LLMs, including training language models from scratch. A complete learning journey that takes you from absolute beginner to conducting cutting-edge AI research.

YouTube videos and self-study materials working together to create ultimate # Zero to AI Researcher roadmap.

### 🎬 YouTube Course Coming Soon

## 🎯 What You'll Learn

- **Python Programming**: Master Python from variables to advanced OOP, preparing you for AI/ML development with NumPy, Pandas, and PyTorch
- **AI Mathematics**: Build intuition for functions, derivatives, gradients, vectors, and matrices - the mathematical foundation that powers all AI
- **Neural Networks**: Understand and implement everything from single neurons to complex multi-layer networks, including backpropagation
- **Transformers**: Build complete transformer architectures from scratch, including attention mechanisms, positional encoding, and feedforward layers
- **Research Skills**: Learn to design controlled experiments, conduct ablation studies, and analyze results like a professional AI researcher
- **State-of-the-Art Techniques**: Implement cutting-edge architectures including DeepSeek's latent attention and GLM-4's Mixture of Experts
- **Practical Implementation**: Write production-ready code, handle real datasets, and optimize model performance through systematic experimentation

## 📚 Complete Curriculum

### 🚀 Getting Started
- **[Start Here](_course/00_start_here/00_start_here.md)** - Course introduction and learning philosophy

### 📖 Module 1: Python Fundamentals
*Master the programming language that powers AI*

- **[Python Basics](_course/01_python_beginner_lessons/01_python_basics.md)** - Variables, data types, and functions
- **[Control Flow and Loops](_course/01_python_beginner_lessons/02_control_flow_and_loops.md)** - If statements, loops, and program flow
- **[Lists and Data Structures](_course/01_python_beginner_lessons/03_lists_and_data_structures.md)** - Lists, dictionaries, tuples, and sets
- **[File Handling and Modules](_course/01_python_beginner_lessons/04_file_handling_and_modules.md)** - Working with files and Python modules
- **[Error Handling and Debugging](_course/01_python_beginner_lessons/05_error_handling_and_debugging.md)** - Exception handling and debugging techniques
- **[Object-Oriented Programming](_course/01_python_beginner_lessons/06_object_oriented_programming.md)** - Classes, inheritance, and OOP concepts
- **[Advanced Python Features](_course/01_python_beginner_lessons/07_advanced_python_features.md)** - Decorators, generators, and context managers
- **[Preparing for AI/ML](_course/01_python_beginner_lessons/08_preparing_for_ai_ml.md)** - NumPy, Pandas, Matplotlib, and Scikit-learn
- **[Python Best Practices](_course/01_python_beginner_lessons/09_python_best_practices.md)** - Code quality, testing, and performance

### 🧮 Module 2: Math Not Scary
*The mathematical foundations of AI, explained simply*

- **[Functions](_course/02_math_not_scary/01_functions.md)** - Understanding mathematical functions
- **[Derivatives](_course/02_math_not_scary/02_derivatives.md)** - The foundation of optimization
- **[Gradients](_course/02_math_not_scary/03_gradients.md)** - Multi-dimensional derivatives
- **[Vectors](_course/02_math_not_scary/04_vectors.md)** - Vector operations and properties
- **[Matrices](_course/02_math_not_scary/05_matrices.md)** - Matrix operations and linear algebra

### 🔥 Module 3: PyTorch Fundamentals
*Master the deep learning framework*

- **[Creating Tensors](_course/03_pytorch_fundamentals/01_creating_tensors.md)** - The building blocks of deep learning
- **[Tensor Addition](_course/03_pytorch_fundamentals/02_tensor_addition.md)** - Basic tensor operations
- **[Matrix Multiplication](_course/03_pytorch_fundamentals/03_matrix_multiplication.md)** - The core operation of neural networks
- **[Transposing](_course/03_pytorch_fundamentals/04_transposing.md)** - Reshaping tensors for operations
- **[Reshaping Tensors](_course/03_pytorch_fundamentals/05_reshaping_tensors.md)** - Changing tensor dimensions
- **[Indexing and Slicing](_course/03_pytorch_fundamentals/06_indexing_and_slicing.md)** - Accessing tensor elements
- **[Concatenation](_course/03_pytorch_fundamentals/07_concatenation.md)** - Combining tensors
- **[Creating Special Tensors](_course/03_pytorch_fundamentals/08_creating_special_tensors.md)** - Ones, zeros, and random tensors
- **[Tokenization and Embeddings](_course/03_pytorch_fundamentals/09_tokenization_and_embeddings.md)** - Converting text to numbers

### 🧠 Module 4: Neuron from Scratch
*Understanding the basic building block of AI*

- **[What is a Neuron?](_course/04_neuron_from_scratch/01_what_is_a_neuron.md)** - The fundamental unit of neural networks
- **[The Linear Step](_course/04_neuron_from_scratch/02_the_linear_step.md)** - Weighted sum computation
- **[The Activation Function](_course/04_neuron_from_scratch/03_the_activation_function.md)** - Adding non-linearity
- **[Building a Neuron in Python](_course/04_neuron_from_scratch/04_building_a_neuron_in_python.md)** - Implementation from scratch
- **[Making a Prediction](_course/04_neuron_from_scratch/05_making_a_prediction.md)** - Using neurons for inference
- **[The Concept of Loss](_course/04_neuron_from_scratch/06_the_concept_of_loss.md)** - Measuring prediction errors
- **[The Concept of Learning](_course/04_neuron_from_scratch/07_the_concept_of_learning.md)** - How neurons improve over time

### ⚡ Module 5: Activation Functions
*The non-linear functions that make neural networks powerful*

- **[ReLU](_course/05_activation_functions/01_relu.md)** - Rectified Linear Unit
- **[Sigmoid](_course/05_activation_functions/02_sigmoid.md)** - The S-shaped function
- **[Tanh](_course/05_activation_functions/03_tanh.md)** - Hyperbolic tangent
- **[SiLU](_course/05_activation_functions/04_silu.md)** - Sigmoid Linear Unit
- **[SwiGLU](_course/05_activation_functions/05_swiglu.md)** - Swish-Gated Linear Unit
- **[Softmax](_course/05_activation_functions/06_softmax.md)** - Probability distributions

### 🕸️ Module 6: Neural Network from Scratch
*Building complete networks and understanding backpropagation*

- **[Architecture of a Network](_course/06_neural_network_from_scratch/01_architecture_of_a_network.md)** - How layers connect
- **[Building a Layer](_course/06_neural_network_from_scratch/02_building_a_layer.md)** - Implementing network layers
- **[Implementing a Network](_course/06_neural_network_from_scratch/03_implementing_a_network.md)** - Complete network implementation
- **[The Chain Rule](_course/06_neural_network_from_scratch/04_the_chain_rule.md)** - Mathematical foundation of backpropagation
- **[Calculating Gradients](_course/06_neural_network_from_scratch/05_calculating_gradients.md)** - Computing derivatives
- **[Backpropagation in Action](_course/06_neural_network_from_scratch/06_backpropagation_in_action.md)** - How networks learn
- **[Implementing Backpropagation](_course/06_neural_network_from_scratch/07_implementing_backpropagation.md)** - Code implementation

### 🎯 Module 7: Attention Mechanism
*The breakthrough that revolutionized AI*

- **[What is Attention?](_course/07_attention_mechanism/01_what_is_attention.md)** - Understanding the attention concept
- **[Self-Attention from Scratch](_course/07_attention_mechanism/02_self_attention_from_scratch.md)** - Building attention step by step
- **[Calculating Attention Scores](_course/07_attention_mechanism/03_calculating_attention_scores.md)** - Query, key, and value operations
- **[Applying Attention Weights](_course/07_attention_mechanism/04_applying_attention_weights.md)** - Weighted combinations
- **[Multi-Head Attention](_course/07_attention_mechanism/05_multi_head_attention.md)** - Multiple attention mechanisms
- **[Attention in Code](_course/07_attention_mechanism/06_attention_in_code.md)** - Complete implementation

### 🔄 Module 8: Transformer Feedforward
*The feedforward layers and Mixture of Experts*

- **[The Feedforward Layer](_course/08_transformer_feedforward/01_the_feedforward_layer.md)** - Standard MLP layers
- **[What is Mixture of Experts?](_course/08_transformer_feedforward/02_what_is_mixture_of_experts.md)** - Introduction to MoE
- **[The Expert](_course/08_transformer_feedforward/03_the_expert.md)** - Individual expert networks
- **[The Gate](_course/08_transformer_feedforward/04_the_gate.md)** - Expert selection mechanism
- **[Combining Experts](_course/08_transformer_feedforward/05_combining_experts.md)** - Weighted expert outputs
- **[MoE in a Transformer](_course/08_transformer_feedforward/06_moe_in_a_transformer.md)** - Integration with attention
- **[MoE in Code](_course/08_transformer_feedforward/07_moe_in_code.md)** - Implementation
- **[The DeepSeek MLP](_course/08_transformer_feedforward/08_the_deepseek_mlp.md)** - Advanced MLP design

### 🏗️ Module 9: Building a Transformer
*Assembling the complete architecture*

- **[Transformer Architecture](_course/09_building_a_transformer/01_transformer_architecture.md)** - High-level overview
- **[RoPE Positional Encoding](_course/09_building_a_transformer/02_rope_positional_encoding.md)** - Rotary positional embeddings
- **[Building a Transformer Block](_course/09_building_a_transformer/03_building_a_transformer_block.md)** - Attention + feedforward
- **[The Final Linear Layer](_course/09_building_a_transformer/05_the_final_linear_layer.md)** - Output projection
- **[Full Transformer in Code](_course/09_building_a_transformer/06_full_transformer_in_code.md)** - Complete implementation
- **[Training a Transformer](_course/09_building_a_transformer/07_training_a_transformer.md)** - Training process overview

### 🚀 Module 10: DeepSeek Latent Attention
*Advanced attention mechanisms from DeepSeek models*

- **[What is Latent Attention?](_course/10_deepseek_latent_attention/01_what_is_latent_attention.md)** - Understanding latent attention
- **[DeepSeek Attention Architecture](_course/10_deepseek_latent_attention/02_deepseek_attention_architecture.md)** - DeepSeek's specific design
- **[Implementation in Code](_course/10_deepseek_latent_attention/03_implementation_in_code.md)** - Building DeepSeek attention

### 🎭 Module 11: GLM-4 Mixture of Experts
*State-of-the-art MoE implementation*

- **[Revisiting Mixture of Experts](_course/11_glm4_moe/01_revisiting_mixture_of_experts.md)** - MoE fundamentals recap
- **[The GLM-4 MoE Architecture](_course/11_glm4_moe/02_the_glm4_moe_architecture.md)** - GLM-4's MoE design
- **[Implementation in Code](_course/11_glm4_moe/03_implementation_in_code.md)** - Building GLM-4 MoE

## 🔬 Research Methodology

After mastering the fundamentals, this course teaches you how to conduct real AI research through hands-on experiments.

### Research Design Principles
- **Hypothesis Formation**: Start with clear, testable hypotheses
- **Controlled Experiments**: Isolate variables to understand their effects
- **Ablation Studies**: Systematically remove components to understand contributions
- **Baseline Comparisons**: Always compare against established baselines

### Experimental Framework
Our research experiments follow a structured approach:

#### **Experiment 1: Simplified Ablation Study**
- **Purpose**: Compare different architectural components at a manageable scale
- **Models**: 5 variants (baseline, MLP, attention+MLP, MoE, attention+MoE)
- **Scale**: 512 hidden dimensions for efficient experimentation
- **Evaluation**: HellaSwag benchmark integration
- **Key Learning**: Understanding how different components contribute to performance

#### **Experiment 2: Learning Rate Search**
- **Purpose**: Find optimal learning rates for different architectures
- **Focus**: DeepSeek attention + MLP combinations
- **Method**: Systematic learning rate exploration
- **Metrics**: Validation loss, accuracy, perplexity
- **Key Learning**: How hyperparameters affect different architectures

#### **Experiment 3: Expert Configuration Search**
- **Purpose**: Optimize MoE configurations
- **Focus**: DeepSeek attention + GLM4 MoE
- **Variables**: Expert count, learning rates, top-k values
- **Method**: Grid search with validation
- **Key Learning**: How to scale MoE models effectively

### Research Skills You'll Develop
- **Experimental Design**: Creating meaningful, controlled experiments
- **Data Analysis**: Interpreting results and drawing conclusions
- **Benchmarking**: Using standard evaluation metrics
- **Reproducibility**: Writing code that others can replicate
- **Documentation**: Communicating research findings clearly

### How to Run the Research Experiments

```bash
# Experiment 1: Simplified Ablation Study
cd experiments/exp1_simplified_ablation_study
python exp1_trainer.py

# Experiment 2: Learning Rate Search
cd experiments/exp2_deepseek_attn_mlp_lr_search
python lr_search.py

# Experiment 3: Expert Configuration Search
cd experiments/exp3_deepseek_attn_glm4_moe_lr_expert_search
python expert_search.py
```

## 🚀 Getting Started

1. **Clone and install**:
```bash
git clone <repository-url> && cd zero-to-ai-researcher
pip install -r requirements.txt
```

2. **Start learning**: Begin with [Start Here](_course/00_start_here/00_start_here.md)

3. **Follow the path**: Complete modules 1-11 in order, then run the research experiments

## 🤝 Contributing

We welcome contributions to improve this course:

- **Content Improvements**: Better explanations, examples, or exercises
- **New Modules**: Additional topics or advanced concepts
- **Research Experiments**: New experimental designs
- **Documentation**: Clearer instructions or additional resources
- **Bug Fixes**: Code corrections or improvements

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- **DeepSeek**: For the advanced attention architecture
- **GLM-4**: For the MoE implementation inspiration
- **HuggingFace**: For the transformer library foundation
- **PyTorch**: For the deep learning framework
- **OpenAI**: For the transformer architecture
- **Google**: For the attention mechanism

## 📞 Support and Community

- **GitHub Issues**: Report problems or suggest improvements
- **Discussions**: Connect with other learners
- **Code Reviews**: Get feedback on your implementations
- **Research Collaboration**: Work together on experiments

---

**Ready to start your journey from zero to AI researcher?** Begin with [Start Here](_course/00_start_here/00_start_here.md) and remember: every expert was once a beginner. Take your time, practice regularly, and don't hesitate to experiment!

**Happy Learning and Researching! 🚀🧠**

zero-to-ai-researcher

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README