DeepSeek-671B-SFT-Guide - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #typescript #go #android #rust #deep-learning #linux #awesome #kubernetes #vue #pytorch #ios #openai

DeepSeek-671B-SFT-Guide

(★ 812)

An open-source solution for full parameter fine-tuning of DeepSeek-V3/R1 671B, including complete code and scripts from training to inference, as well as some practical experiences and conclusions. (DeepSeek-V3/R1 满血版 671B 全参数微调的开源解决方案，包含从训练到推理的完整代码和脚本，以及实践中积累一些经验和结论。)

파일 탐색기

최종 버전 다운로드 (.zip)

- - convert_pth_to_hf.sh
  - sft_deepseek.py
  - sft_deepseek.sh
  - sft_deepseek_pdsh.sh
  - vllm_deploy_slurm.sh
- - __init__.py
  - entry_point.py
  - registry.py
  - version.py
- README.md
- train_example.json
- - config.json
  - configuration.json
  - configuration_deepseek.py
  - generation_config.json
  - LICENSE
  - modeling_deepseek.py
  - modeling_deepseek_origin.py
  - tokenizer.json
  - tokenizer_config.json
- - config.json
  - configuration.json
  - configuration_deepseek.py
  - modeling_deepseek.py
  - tokenizer.json
  - tokenizer_config.json
.gitignore
LICENSE
log.png
README.md
README_zh.md
requirements.txt

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/ScienceOne-AI/DeepSeek-671B-SFT-Guide

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd DeepSeek-671B-SFT-Guide

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DeepSeek-V3/R1-671B Full Parameter Fine-Tuning Guide

<div align="center" style="line-height: 1;">

[![GitHub Stars](https://img.shields.io/github/stars/ScienceOne-AI/DeepSeek-671B-SFT-Guide?style=social)](https://github.com/ScienceOne-AI/DeepSeek-671B-SFT-Guide)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

[中文版](./README_zh.md) ｜ [English](./README.md)

</div>

An open-source solution for full parameter fine-tuning of DeepSeek-V3/R1 671B, including complete code and scripts from training to inference, as well as some practical experiences and conclusions, jointly launched by the Institute of Automation of the Chinese Academy of Sciences and Beijing Wenge Technology Co. Ltd.

## 🌟 Project Highlights
- Implemented modeling files containing DeepSeek-V3/R1 training logic (see `./model`, code logic completed based on Deepseek-V3 paper and Deepseek-V2 modeling files);
- Implemented full parameter fine-tuning of DeepSeek-V3/R1 671B based on data parallelism (DeepSpeed ZeRO) + sequence parallelism (SP);
- Summarized the entire process of model training and deployment, including pitfalls, encountered problems, and solutions.

## 🚀 Quick Start
### 1. Hardware Configuration

The configuration of a single server is shown in the table below. There are 32 machines with the same configuration in the cluster, sharing 100TB of storage space, mounted at `/nfs`. The operating system of the machines is Ubuntu 22.04, with IB network communication between machines, NVLink communication between GPUs, and CUDA version 12.6.

| Component  | Specification/Version       | Command to View Details  |  
|------------|-----------------------------|--------------------------| 
| GPU        | 8 x NVIDIA H100 80GB HBM3   | `nvidia-smi`             |   
| CPU        | Intel(R) Xeon(R) Platinum 8463B (96 Cores) | `lscpu` |    
| Memory     | 2.0TB DDR4                  | `free -h`                | 
| Storage    | 100TB NVMe SSD              | `df -h`                  | 
| Network    | InfiniBand 400G             | `ibstat`                 | 
| OS         | Ubuntu 22.04                | `uname -a`               | 
| CUDA       | CUDA 12.6                   | `nvcc -V`                |

### 2. Environment Setup

We extended and improved the xtuner framework to support full parameter fine-tuning of Deepseek V3/R1 (i.e., `DeepseekV3ForCausalLM` model architecture), supporting data parallelism (DeepSpeed ZeRO based DP) and sequence parallelism (SP).

Install the Python environment, install dependencies according to the `requirements.txt` in the project, and overwrite the core code related to `DeepseekV3ForCausalLM` training in `./code/xtuner` to the corresponding code in the original xtuner package.

```bash
conda create -n ds_env python=3.10
conda activate ds_env
pip install -r requirements.txt

# Overwrite core code, modify to your environment path
YOUR_ENV_PATH='/nfs/miniconda3/envs/ds_env/lib/python3.10/site-packages'
cp -r ./code/xtuner $YOUR_ENV_PATH
```

### 3. Data Preparation
We extended the OpenAI standard data format to be compatible with reasoning data. Each original training data is formatted as follows. If there is a reasoning process, the `reasoning_content` field of the assistant role is not empty.
```json
{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "User question"},
    {"role": "assistant", "content": "Final answer", "reasoning_content": "Reasoning process"}
  ]
}
```

To simplify the processing logic, we merged the `reasoning_content` and `content` into the `content` field according to the Deepseek training format. Additionally, to be compatible with multi-turn dialogue training logic, we added a `loss` field for each round of the assistant role, and only calculate the loss for `content` with `loss` value `true`.
```json
{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "User question"},
    {"role": "assistant", "content": "<think>\nReasoning process\n</think>\n\nFinal answer", "loss": true}
  ]
}
```

To clearly show the data storage format, we provide a sample file of the converted data `./data/train_example.json` for reference.

During actual training, the program will automatically convert to the following format according to the Deepseek V3/R1 training template, here for display only:
```
<｜begin▁of▁sentence｜>You are a helpful assistant.<｜User｜>User question<｜Assistant｜><think>\nReasoning process\n</think>\n\nFinal answer<｜end▁of▁sentence｜>
```

### 4. Start Training
We provide training code and training startup scripts, including:
- `./code/scripts/sft_deepseek.py`: Configuration file for sft training, including hyperparameter settings, model and tokenizer configuration, training strategy, etc. Modify model training-related configurations in this file.
- `./code/scripts/sft_deepseek.sh`: sft training startup script, which is an execution file for a single node, so it needs to be executed on each machine through slurm or pdsh. For each machine, the only difference in the training startup command is the `NODE_RANK` value. If there are 32 machines, the number ranges from 0 to 31.

Using pdsh as an example, the steps to start training are as follows:
1. Overwrite the `modeling_deepseek.py` file provided in the `./model` directory of this project to the corresponding original file downloaded from platforms like huggingface;
2. Use pdsh to start training, execute the command `pdsh -R ssh -w node[0-31] 'bash ./code/scripts/sft_deepseek.sh'` on the 0th machine to start the full parameter fine-tuning task of the model on 32 machines. Modify `node[0-31]` according to your machine hostname or IP address. During training, you can visualize the training process (loss changes, etc.) through tensorboard.

----
Below are the conclusions of several experiments we conducted, including the feasibility of model training under different parallel strategies and configurations. The training data is ~100k, and the training context length is 32k. The table reports the number of machines used (nodes), sequence parallelism (sp), data parallelism method (dp), single card batch size (bs), number of iterations (epoch), learning rate (lr), single card memory (mem), experiment records, and notes.

| nodes | sp  | dp             | bs |epoch| lr   | mem   | notes       |
|-------|-----|----------------|----|----|-------|-------|-------------|
| 16    | 8   | zero3_offload  | 2  | 1  | 2e-7  | ~30GB  | ✅ Trainable |
| 32    | 8   | zero3_offload  | 1  | 1  | 1e-5  | ~32GB  | ✅ Trainable |
| 32    | 4   | zero3_offload  | 1  | 1  | 2e-7  | ~25GB  | ✅ Trainable |
| 32    | 1   | zero3_offload  | 1  | 1  | 2e-7  | ~30GB  | ✅ Trainable |
| 32    | 4   | zero3_offload  | 2  | 1  | 2e-7  | ~74GB  | ✅ Trainable (Recommended) |
| 32    | 1   | zero3_offload  | 2  | 1  | 2e-7  | OOM    | ❌ Out of Memory |
| 32    | 4   | zero3          | 1  | 1  | 2e-7  | OOM    | ❌ Out of Memory |
| 32    | 1   | zero3          | 1  | 1  | 2e-7  | OOM    | ❌ Out of Memory |

Below is a screenshot during training. We observed that when fully fine-tuning DeepSeek V3 on our constructed reasoning data, the initial loss is usually around 3.5, and after 1 epoch of training, the loss converges to around 1.2.

![Training Log](https://raw.githubusercontent.com/ScienceOne-AI/DeepSeek-671B-SFT-Guide/main/log.png)

### 5. Model Weight Conversion

During training, it is recommended to use at least 100TB of SSD large-capacity storage, as a single pth intermediate result occupies about `7.4TB` of disk space. After training, we need to convert the pth to a huggingface format that is better compatible with mainstream inference frameworks (such as vllm). Execute `bash ./code/scripts/convert_pth_to_hf.sh` on a single machine node to complete the model weight format conversion. You can modify the pth path and weight save path in the script according to the actual situation.

Note that this process requires a large amount of CPU memory, so you can expand it through virtual memory to prevent Out-of-memory. Swap (swap partition) is Linux's virtual memory, which is used to store part of the data to the disk when the physical memory (RAM) is insufficient, freeing up RAM.

```bash
sudo fallocate -l 8192G /tmp/swapfile  # Create 8T swap file
sudo chmod 600 /tmp/swapfile
sudo mkswap /tmp/swapfile
sudo swapon /tmp/swapfile
free -h  # Check if swap has increased
```

### 6. Model Inference Deployment
According to the [Deepseek V3 Github](https://github.com/deepseek-ai/DeepSeek-V3?tab=readme-ov-file#6-how-to-run-locally) introduction, there are multiple ways to deploy the model locally. We used [vLLM](https://github.com/vllm-project/vllm) to perform a simple deployment test of the fully fine-tuned model. Here, we assume that an environment named `vllm` has been created according to the vLLM official documentation.

If using a slurm cluster, refer to our provided script and execute the sbatch command `sbatch ./code/scripts/vllm_deploy_slurm.sh` to submit the job. Half-precision (bf16/fp16) models are recommended to be deployed using 4 machines with 32 cards. If you need to configure the port number of ray or api server, you can modify the sh file yourself.

If you need to start the deployment through pdsh (assuming using node0~node3 four machines), refer to the following steps:

1. Set environment variables (node0~node3).
```bash
export HEAD_ADDR="node0"
export DASHBOARD_PORT=8265
export HEAD_PORT=6379
export RAY_TMPDIR=/tmp/ray_tmp/
export RAY_ADDRESS=$HEAD_ADDR:$HEAD_PORT
```

2. Start Ray Head (node0).
```bash
pdsh -R ssh -w node0 "source /nfs/miniconda3/etc/profile.d/conda.sh && conda activate vllm && \
ray start --block --head --port=$HEAD_PORT --dashboard-port=$DASHBOARD_PORT --temp-dir=$RAY_TMPDIR"
```

3. Start Ray Worker (node1~node3).
```bash
pdsh -R ssh -w node1,node2,node3 "source /nfs/miniconda3/etc/profile.d/conda.sh && conda activate vllm && \
ray start --block --address=$HEAD_ADDR:$HEAD_PORT"
```

4. Start vLLM (node0).
```bash
pdsh -R ssh -w node0 "source /nfs/miniconda3/etc/profile.d/conda.sh && conda activate vllm && \
vllm serve /path/of/your/deepseek_sft_ckpt \
    --tensor-parallel-size 8 \
    --pipeline-parallel-size 4 \
    --served-model-name deepseek-r1-sft \
    --max-model-len 32768 \
    --trust-remote-code \
    --enable-reasoning \
    --reasoning-parser deepseek_r1"
```

After starting, you can test whether the interface is started normally through the curl command:

```bash
curl -X POST http://node0:8000/v1/chat/completions -d '{"model": "deepseek-r1-sft", "messages":[{"role":"user", "content": "hello"}]}' -H "Content-Type: application/json"
```

After a while, if the terminal outputs the expected response, it means that the entire process from training to deployment has been successfully completed! 🎉 If there are any problems in the above steps or any suggestions for improvement, please feel free to raise an issue for feedback, and we will try to respond and answer as soon as possible.

## 🤝 Acknowledgements
- DeepSeek-V2/V3/R1: [https://github.com/deepseek-ai](https://github.com/deepseek-ai)
- Huggingface transformers: [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers)
- DeepSpeed: [https://github.com/deepspeedai/DeepSpeed](https://github.com/deepspeedai/DeepSpeed)
- Xtuner: [https://github.com/InternLM/xtuner](https://github.com/InternLM/xtuner)
- vLLM: [https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)

## 🔍 License
This project is licensed under the Apache-2.0 License.

## ⭐ Star History
[![Star History Chart](https://api.star-history.com/svg?repos=ScienceOne-AI/DeepSeek-671B-SFT-Guide&type=Date)](https://star-history.com/#ScienceOne-AI/DeepSeek-671B-SFT-Guide&Date)