singing-database-maker

KO | EN

gitlite — search

#python #react #typescript #android #deep-learning #docker #nodejs #windows #kubernetes #pytorch #security #openai #mcp #open-source

singing-database-maker

(★ 13)

AI based singing voice synthesis database generator

singing-database-maker Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# singing-database-maker

## Preface

High quality data is always a problem of singing voice synthesis. And it's really laborious to make a database from scratch. We hope AI can make the process a lot easier, so every music lover can make his own synthesised song.

Hopefully, there are just tools satisfy our needs. We combine them and a tool to ease the process.Thanks to all the contibuters of these grate researches. We list them below, so you can go and check their work:

> [1] Sangeun Kum et al. “Semi-supervised learning using teacher-student models for vocal melody extraction”. In: Proc. International Society of Music Information Retrieval Conference (ISMIR). 2020.
>
> [2] Xianming Li. XMNLP: A Lightweight Chinese Natural Language Processing Toolkit. https://github.com/SeanLee97/ xmnlp. 2018.
>
> [3] Kilian Schulze-Forster et al. “Phoneme Level Lyrics Alignment and Text-Informed Singing Voice Separation”. In: IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021), pp. 2382–2395. DOI: 10.1109/TASLP.2021. 3091817.
>
> [4] Intelligence Engineering Lab @ Korea University. mdx-net-submission: Music demixer. https://github.com/kuielab/mdx-net-submission 2021.

## Input and Output

Notice: The project now support English songs only, with Chinese support in  early development. The key is Phoneme Level Lyrics Alignment module, we assume we can deal with it this summer.

We list input and outpit of thetoolkit here, so you can have a genneral idea of whether the project suits your needs.

Input: songs and their .lrc format lyrics.

Output:

- songs devided into slices according to lyrics sentences
- phoneme and word list, with the time their appear in the slice
- a midi file generated by Semi-supervised AI network

We plan to add:

- musicXML generator
- Chinese support
- more precise midi file
- an synthesised example using the database

## Steps

### Introduction

suppose you have a song called foo. the processed database folder will be like this: (only list the tools you will use)

```shell
 - origin
 	- foo.wav
 	- foo.lrc
 - processed_data
 	- vocal
 	- slice
 		- foo00150019
 			foo00150019.wav
 			foo00150019.txt
 		- ...
 	- pitch
 		- pitch_foo00150019
 		- ...
 	- midi
 		- foo00150019.mid
 		- ...
 	- align
 		- foo00150019
 			- phoneme_onsets
 				- foo00150019.txt
 			- word_onsets
 				- foo00150019.txt
 - utils
 	- english-align
 		- phoneme_from_word
 		- make_phoe.py
 	- melodyExtraction
 		- gen_freq.py
 	- vocal-extraction
 	- config.py
    - song_cutter.py
    - demix_vocal.py
    - gen_midi.py
    - make_Midi.py
    - make_lab.py (not finished)
    - make_musicxml.py (not finished)
    - delete_useless.py (not tested)
    - missing.txt (generate after align)
```

### Project download

Download the full project, and its submodules.
   
   We use submodules to ease our development, so you must use `--recurse-submodules` to clone the full project.

```shell
$ git clone --recurse-submodules git@github.com:leavelet/singing-database-maker.git
```

### Environment preparation

When dealing with mutiple AI projects, it will make your life much easier to set up the environment properly at first step. We've had a hard time dealing with all of this, and we found you can use the project on your own pc if you set correctly.

1. install a python virtual environment manager

We use recommand conda, and we take conda as an example.

2. crate environments

```shell
    $ cd requirements
    
    # misc
    $ conda create -n singing-dealer
    $ conda activate singing-dealer
    $ pip install -r make_midi.txt
    
    # vocal extraction
    $ conda env create -f vocal-extraction/environment.yml
    #if you use arm mac, use environment-m1.yml
    $ conda activate vocal-extraction
    $ pip install -r vocal-extraction/requirements.txt
    
    # alignment & melody extraction
    $ conda env create -f vocal-extraction/maker_ai_cpu.yml
    # if you have gpu, use maker_ai_gpu.yml
    $ conda activate maker_ai
    $ pip install -r melody_extraction.txt
    ```

3. Download models
   1. Download demucs models
   
    ```shell
    $ conda activate vocal-extract
    $ python download_demucs.py
    ```

2. Download onnx models
    
    Download the models from [release page](https://github.com/leavelet/singing-database-maker/tags), put them under `utils/vocal-extracion/onnx`folder.

4. download the [make_dic tool](https://github.com/leavelet/phoneme_from_word/releases/tag/v1.0.1) and put it under project root directory

### *from now on, We assume your current dir is `utils`*

### Setup your own config

We use `config.py` to control the whole project. All the files are programmed to follow settings.

1. set the project root.

Project root is parent folder of utils. We use `..` to mark the root since we are in `utils`folder, but an absolute path is recommanded, to avoid mistakes like forgetting to change back to `utils` after operation.

2. set your thread_num.

We use parallelism to accelerate processing. Set the `thread_num` to a proper number to make full use of your processor. Default is 10.

### Seperate songs

### Demix the slices

### Generate midi notes and make midi

### Generate phoneme Level Lyrics Alignment

## License and acknowledgement

The whole project is under MIT License, all the projects we used in this project are under their own license.

We do not guarantee the quality of dataset, and before using any data, you must have the appropriate copyright permission.

singing-database-maker

singing-database-maker Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

singing-database-maker

singing-database-maker Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README