KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #codex #typescript #ai-agents #claude-code #dsh #dsh-plugin #cli #deepseek-harness #macos #ai-agent #open-source

SyllableLM

★ 63 Open GitHub ↗

Official Code for SyllableLM: Learning Coarse Semantic Units for Speech Language Models

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# SyllableLM
Official Public Code for "SyllableLM: Learning Coarse Semantic Units for Speech Language Models"

Paper: [https://arxiv.org/abs/2410.04029](https://arxiv.org/abs/2410.04029)

ICLR 2025

[Demo Audios](https://syllablelmanonymous.github.io/SyllableLMAnonymous/)

## Setup:

```bash
conda create -n syllablelm python=3.9
conda activate syllablelm

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu115

pip install omegaconf
pip install timm
```

## SylBoost:

### Checkpoints

| SylBoost | Model                                                                                        | KMeans                                                                                              | Agglomerative Clustering                                                                             |
|----------|----------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------|
| 8.33Hz   | [Model](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SylBoost_833Hz.pth) | [KMeans](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SylBoost_833Hz_kmeans.npy) | [Agglom](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SylBoost_833Hz_agglom.npy) |
| 6.25Hz   | [Model](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SylBoost_625Hz.pth) | [KMeans](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SylBoost_625Hz_kmeans.npy) | [Agglom](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SylBoost_625Hz_agglom.npy) |
| 5.0Hz    | [Model](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SylBoost_500Hz.pth) | [KMeans](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SylBoost_500Hz_kmeans.npy) | [Agglom](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SylBoost_500Hz_agglom.npy) |

### Usage

SylBoost inference and efficient extraction code in ``extract_units.py``

People have had trouble setting up Data2Vec2 so I copied it and stripped it. No Fairseq reqired!

```python
sylboost_reader = SylBoostFeatureReader(
        '/path/to/model.pt'
        '/path/to/kmeans.npy',
        '/path/to/agglom.npy',
        '8.33Hz',  # '6.25Hz', '5.0Hz'
    )
```

## SyllableLM:

### Checkpoints

| SyllableLM                    | Model                                                                                                    |
|-------------------------------|----------------------------------------------------------------------------------------------------------|
| 6.25Hz Base                   | [Model](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SyllableLM_base_625Hz.pt)       |
| 6.25Hz Large                  | [Model](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SyllableLM_large_625Hz.pt)      |
| 6.25Hz Interleaved Vocoder LM | [Model](https://www.cs.utexas.edu/~harwath/model_checkpoints/syllable_lm/SyllableLM_vocoder_lm_625Hz.pt) |

### Resynthesis (and Continuation):

Located in Experiments.ipynb

### LossPred:

This is provided as-is in Experiments.ipynb

### SylBoost training:

This is provided as-is in syllablelm/SegmentalSSE
See the provided Readme for more information

### SyllableLM training:

This is provided as-is in syllablelm/fairseq
See the provided Readme for more information

SyllableLM

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README