Skill-Entropy-RL

KO | EN

gitlite — search

#python #hacktoberfest #llm #golang #go #machine-learning #rust #docker #java #windows #cli #kubernetes #awesome-list #security

Skill-Entropy-RL

(★ 31)

Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

Skill-Entropy-RL Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🤹 Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

**[Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning](https://arxiv.org/abs/2608.05139)**

Yinghui He, Ling Yang, Jiarui Liu, Yongjin Yang, Lechen Zhang,<br/>
Yingcheng Wu, Zhenfei Yin, Mengdi Wang, Sanjeev Arora

</div>

---

**Long-horizon tasks require models to *switch between skills*, not just execute a single skill well.** This repository contains the full pipeline behind the paper:

- **Skill²-Bench**: a benchmark of multi-step tasks that force models to switch between skills — 558 skills across 9 domains (math, science, coding, logic, information extraction, planning, creative writing, context retrieval, instruction following). Also available on Hugging Face: [Gen-Verse/Skill2-Bench](https://huggingface.co/datasets/Gen-Verse/Skill2-Bench).
- **Skill entropy**: a measure of how difficult a skill switch is: high when a reference model handles two skills well separately but fails when combining them. Model performance drops as skill entropy rises.
- **Skill-Entropy RL**: GRPO training whose reward scores the model's skill-switching plan, not just its final answers — the reward combines a final-answer reward with a skill-entropy reward on skill-annotated rollouts.

## Get Started

Python ≥ 3.10, CUDA 12.x, PyTorch 2.8 recommended.

```bash
# core trainer (installs verl fork + dependencies)
cd skill_entropy_rl && pip install -e . && cd ..

# eval extras
pip install antlr4-python3-runtime==4.11.1 word2number timeout-decorator
```

Every script resolves the repo location from one environment variable:

```bash
export SKILL_ENTROPY_RL_ROOT=/path/to/this/repo
mkdir -p $SKILL_ENTROPY_RL_ROOT/job_log   # SLURM launchers write their logs here (gitignored)
```

## Running the pipeline

### 1. Entropy calibration *(needs an API gateway)*

```bash
bash label_skills/label_skills.sh                # skill labeling (vLLM)
bash label_skills/deduplicate_skills.sh          # merge + dedupe labels → label_skills/merged_skill_labels/
cd calculate_skill_entropy && bash run.sh       # reference-model calibration → entropy map
```

### 2. Task generation

```bash
bash propose_tasks/propose_tasks_api.sh          # benchmark tasks (API proposer — produced skill2_bench/)
sbatch propose_tasks/propose_tasks.sh            # math RL task pool (vLLM proposer → propose_tasks/math/)
```

### 3. SFT warm-up

```bash
sbatch sft/sft.sh                                # cross-skill SFT (sft_math.sh for math-only)
# regenerate SFT traces from a task pool: sft/generate_skillrl_sft.sh / generate_skillrl_sft_math.sh
```

### 4. RL (GRPO / Skill-Entropy RL)

```bash
# builds train/val parquets, then launches training
sbatch skill_entropy_rl/examples/grpo_trainer/run_skill_entropy_rl.sh           # Skill-Entropy RL
sbatch skill_entropy_rl/examples/grpo_trainer/run_skill_entropy_rl_math.sh      # math RL (needs a stage-2 math pool in propose_tasks/math/)
# chained multi-day runs: skill_entropy_rl/examples/grpo_trainer/launch_chain.sh
```

Reward variants (e.g. gated / multi-gold `r_ent`) are selected with `+env.reward_mode=<mode>` and implemented in `skill_entropy_rl/agent_system/environments/env_package/skill2_bench/reward_variants.py`.

### 5. Evaluation

```bash
bash  scripts/merge_checkpoint.sh                  # first: merge a sharded RL checkpoint to HF format
sbatch evaluation/eval_checkpoint.sh               # Skill²-Bench eval of a checkpoint (vLLM)
bash  evaluation/eval_api_models.sh                # Skill²-Bench eval of API models (Claude/GPT/Gemini via gateway)
```

## 📝 Citation

```bibtex
@article{he2026skillnative,
  title={Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning},
  author={He, Yinghui and Yang, Ling and Liu, Jiarui and Yang, Yongjin and Zhang, Lechen and Wu, Yingcheng and Yin, Zhenfei and Wang, Mengdi and Arora, Sanjeev},
  journal={arXiv preprint arXiv:2608.05139},
  year={2026}
}
```

## 📄 License

MIT (see [LICENSE](LICENSE)).

Skill-Entropy-RL

Skill-Entropy-RL Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Skill-Entropy-RL

Skill-Entropy-RL Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README