op_pangu

KO | EN

gitlite — search

#python #react #ai #go #golang #machine-learning #deep-learning #linux #docker #windows #awesome #cli #openai #open-source

(★ 523)

No description available.

파일 탐색기

최종 버전 다운로드 (.zip)

- - run.py
  - S1-1.py
  - S1-2.py
  - S1-3.py
  - S2-1.py
  - S2-2.py
- crash_test_v1.py
- crash_test_v2.py
- fix.sh
- fusion_result.json
- generate.ipynb
- run.ipynb
- run.log
- run.py
- train_v1.ipynb
- train_v1.py
- upgrade.ipynb
- train-00000-of-00001-a09b74b3ef9c3b56.parquet
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Silent Inconsistency in Data-Parallel Full Fine-Tuning  
### Experimental Fine-Tuned Models (S1-1 / S1-2 / S1-3)

This repository provides **three fully fine-tuned models** corresponding to the experimental settings in the paper:

> **Silent Inconsistency in Data-Parallel Full Fine-Tuning: Diagnosing Worker-Level Optimization Misalignment**

The models were trained to reproduce and analyze the phenomenon of **worker-level optimization misalignment** under synchronous data-parallel (DDP) full-parameter fine-tuning.

---

## Keywords

data-parallel • distributed-training • full-finetuning • inconsistency-diagnosis • alpaca • openpangu

---

## Background

In synchronous data-parallel training with All-Reduce, model parameters are strictly synchronized after each update step. However, synchronization of parameters does **not** guarantee consistency in worker-level optimization dynamics *before gradient aggregation*.

The paper introduces the concept of:

> **Silent Inconsistency** — a hidden divergence in worker-level loss and gradient behavior that remains invisible in the global averaged loss curve.

To diagnose this issue, the paper proposes three lightweight monitoring metrics:

- **Loss Dispersion** — Standard deviation / range of per-worker loss  
- **Gradient-Norm Dispersion** — Standard deviation of per-worker gradient L2 norms  
- **Gradient-Direction Consistency** — Average cosine similarity between worker gradients

These metrics can be computed online with negligible overhead and without modifying the optimization algorithm.

---

## Base Model

All three models are fully fine-tuned from:

**Ascend Tribe – openPangu-Embedded-1B-V1.1 (1B parameters)**  
https://ai.gitcode.com/ascend-tribe/openPangu-Embedded-1B-V1.1

- Architecture: Causal LM  
- Parameters: ~1B  
- Precision: bf16 mixed precision during training  
- Training mode: Full-parameter fine-tuning (no LoRA / adapters)

---

## Dataset

Fine-tuned on:

**tatsu-lab / alpaca (Instruction tuning dataset)**  
https://huggingface.co/datasets/tatsu-lab/alpaca

- Template format: `Instruction – Input – Response`  
- Maximum sequence length: 1024  
- Loss computed **only on the Response tokens**  
- Prompt and template tokens are masked during loss calculation

---

## Model Repository

Hugging Face Model Hub:  
https://huggingface.co/jiujiudahaozi/op_pangu

---

## Experimental Settings

### S1-1 — Strict Consistency

- All ranks use the **same random seed**
- Deterministic `DistributedSampler`
- Identical shuffling behavior across workers

**Expected behavior**

- Low loss dispersion  
- Low gradient-norm dispersion  
- High gradient-direction consistency

---

### S1-2 — Mild Inconsistency

- Rank 0 uses a different random seed  
- Other ranks follow the baseline seed

**Expected behavior**

- Slight increase in worker-level dispersion  
- Global averaged loss remains smooth

---

### S1-3 — Significant Inconsistency

- Each rank uses a distinct seed dependent on rank ID

**Expected behavior**

- Large loss dispersion  
- Larger gradient-norm variation  
- Reduced gradient-direction consistency  
- Global loss may still appear normal

---

## Contributors

Hong Li · Zhen Zhou · Honggang Zhang · Yuping Luo · Xinyue Wang · Han Gong · Zhiyuan Liu

op_pangu

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

op_pangu

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README