syntheory

KO | EN

gitlite — search

#hacktoberfest #react #ai #typescript #llm #golang #machine-learning #rust #docker #java #awesome #awesome-list #vue #mcp

(★ 24)

Do Music Generation Models Encode Music Theory? (ISMIR 2024)

syntheory Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Do Music Generation Models Encode Music Theory? (ISMIR 2024)
[Paper](https://arxiv.org/abs/2410.00872) | [Website](https://brown-palm.github.io/music-theory/) | [Hugging Face Dataset](https://huggingface.co/datasets/meganwei/syntheory)

## Installation Instructions

### Virtual Environment

Run the following from the root of the repository:
```bash
conda env create -f environment.yaml
```
>Will create a conda environment named `syntheory`

The environment may be activated with:
```bash
conda activate syntheory
```

If there are problems installing from `environment.yaml`, the main dependencies we need are
- `pytorch`: (`see link`): [LINK](https://pytorch.org/get-started/locally/)
- `transformers` (`pip install transformers`): [LINK](https://github.com/huggingface/transformers)
- `jukemirlib` (`pip install git+https://github.com/rodrigo-castellon/jukemirlib.git`): [LINK](https://github.com/rodrigo-castellon/jukemirlib)
- `ffmpeg-python` (`pip install ffmpeg-python`): [LINK](https://github.com/kkroening/ffmpeg-python)
- `mido` (`pip install mido`): [LINK](https://github.com/mido/mido)
- `zarr` (`pip install zarr`): [LINK](https://github.com/zarr-developers/zarr-python)

### Dependencies

You may need to install the following dependencies, if you don't already have them.

#### Linux
Requires `ffmpeg`.

On Linux, through a package manager like `apt`:
```bash
apt install ffmpeg
```

#### MacOS

Requires `ffmpeg`.

Can be installed through homebrew (on MacOS) with:
```bash
brew install ffmpeg
```

(If running on M1 mac, must also install `libsndfile`):
```bash
brew install libsndfile
```

Note, even after installing `libsndfile`, `OSError: sndfile library not found` might be raised. To fix this, run:
```bash
conda install -c conda-forge libsndfile
```

## Dataset Generation

### Compiling the Synthesizer

First, install [cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html). Then, run

```bash
bash repo/compile_synth.sh
```

We need this to turn MIDI into a `.wav` file. A bit more information on this step can be found here: [README.md](/midi2audio/README.md)

### SynTheory Datasets

The following datasets exist in the `dataset/synthetic` folder:

- `chords`: ~18.7 GB (13,248 samples)
- `chord_progressions`: ~29.61 GB (20,976 samples)
- `intervals`: ~56.1 GB (39,744 samples)
- `notes`: ~14.02 GB (9,936 samples)
- `scales`: ~21.82 GB (15,456 samples)
- `tempos`: ~5.68 GB (4,025 samples)
- `time_signatures`: ~1.48 GB (1,200 samples)

Any of these can be generated by running the below command from the root of the repository:

```bash
python dataset/synthetic/<NAME_OF_DATASET>.py
```

where `<NAME_OF_DATASET>` is one of the following datasets: `chords`, `chord_progressions`, `intervals`, `notes`, `scales`, `tempos`, and `time_signatures`.

This will create a folder in: `data/<NAME_OF_DATASET>`.

Each folder will contain:

- `info.csv`
- `.wav` files
- `.mid` files

Some are quite large.

### SynTheory Dataset via Hugging Face
You can also download our dataset through Hugging Face [here](https://huggingface.co/datasets/meganwei/syntheory).

To download a particular concept (e.g. notes), run the following script:
```python
from datasets import load_dataset

notes = load_dataset("meganwei/syntheory", "notes")
```

You can also access our dataset in streaming mode instead of downloading the entire dataset to disk by running the following (for each desired concept):
```python
from datasets import load_dataset

notes = load_dataset("meganwei/syntheory", "notes", streaming=True)

print(next(iter(notes)))
```

### Making ✨ Custom 🔥 Datasets

We have a short guide that explains how to use this codebase to create custom datasets. We encourage the community to create more complex and diverse concept definitions.

> [!NOTE]
> We hope for SynTheory to be more than a static dataset - it is a _framework_ and procedure for creating music theoretic concept understanding datasets.

[Custom Dataset Instruction Guide](docs/README.md)

## Embedding Extraction

After creating a synthetic dataset, run 
```bash
python embeddings/extract_embeddings.py --config embeddings/emb.yaml
```

This will use a specific configuration to extract embeddings for each `.wav` file and save them to a zarr file. You can specify the models and concepts to extract embeddings from by editing `models` and `concepts` in the configuration file.

For each dataset and model combination, this will produce a csv file named `data/<NAME_OF_DATASET>/<NAME_OF_DATASET>_<MODEL_HASH>_embeddings_info.csv`. Each file contains information about each embedding for that particular model and dataset.

Due to the time it may take to extract these embeddings, the dataset is partitioned into shards, each responsible for extracting up to some constant number of embeddings. This will start a SLURM job for each shard.

## Probing Experiments
To launch probing experiments, run 
```bash
python probe/run_probes.py
```

You can specify which models and concepts you're probing by modifying the `sweep_config` argument. The sweep configurations are defined in `probe/probe_config.py`. Currently, we include configs on probing the handcrafted features, MusicGen Audio Encoder, Jukebox, and MusicGen decoder language models across all concepts.

In our implementation, we performed hyperparameter search for handcrafted features (`CHROMA`, `MELSPEC`, `MFCC`, `HANDCRAFT`) and MusicGen Audio Encoder. For Jukebox and MusicGen Decoder models, we used a fixed set of hyperparameters and employ a layer selection process. You're welcome to adapt the models, concepts, and hyperparameters towards your own needs by modifying `SWEEP_CONFIGS` in `probe/probe_config.py`.

Our implementation involves logging our probing results to a [Weights & Biases](https://wandb.ai/site/) project. Before you run the script, make sure to log into your wandb account by providing your [wandb API key](https://docs.wandb.ai/quickstart/) and running `wandb login`.

When analyzing your results on Weights & Biases, the desired probing metric is under `primary_eval_metric`.

## Running Tests

From the root of the repository, one may run tests with:
```bash
bash repo/test.sh
```

This will call [pytest](https://docs.pytest.org/en/stable/) and [pytest-cov](https://github.com/pytest-dev/pytest-cov) to produce a coverage report.

## BibTeX

If you find this useful, please cite us in your work.

```
@inproceedings{Wei2024-music,
        title={Do Music Generation Models Encode Music Theory?},
        author={Wei, Megan and Freeman, Michael and Donahue, Chris and Sun, Chen},
        booktitle={International Society for Music Information Retrieval},
        year={2024}
}
```

syntheory

syntheory Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

syntheory

syntheory Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README