Speaker-Diarization - GitLite

KO | EN

gitlite — search

#javascript #android #machine-learning #rust #linux #nodejs #awesome #kubernetes #awesome-list #vue #openai #mcp #ios #open-source

Speaker-Diarization

(★ 502)

speaker diarization by uis-rnn and speaker embedding by vgg-speaker-recognition

File Explorer

Download Latest Version (.zip)

- arguments.html
- evals.html
- index.html
- loss_func.html
- uisrnn.html
- utils.html
- - a_1.wav
  - a_2.wav
  - a_3.wav
  - b_1.wav
  - b_2.wav
  - b_3.wav
  - x_1.wav
  - x_2.wav
  - x_3.wav
  - z_1.wav
  - z_2.wav
  - z_3.wav
- - weights.h5
- backbone.py
- generate_embeddings.py
- model.py
- predict.py
- preprocess.py
- toolkits.py
- utils.py
- saved_model.uisrnn_benchmark
- diarization.gif
- uisrnn.gif
- YouTube_lecture_screenshot.png
- evals_test.py
- integration_test.py
- uisrnn_test.py
- utils_test.py
- __init__.py
- arguments.py
- evals.py
- loss_func.py
- uisrnn.py
- utils.py
- player.py
- viewer.py
- rmdmy.wav
.coveragerc
.gitignore
.travis.yml
CONTRIBUTING.md
LICENSE
publish.sh
README.md
requirements.txt
run_pdoc3.sh
run_pylint.sh
run_tests.sh
setup.py
speakerDiarization.py
train.py

# Installation Guide

1. Get the code

git clone https://github.com/taylorlu/Speaker-Diarization

Downloads the entire project code from GitHub to your computer.

cd Speaker-Diarization

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

Runs the Python script (or module).

✅ If it runs without errors and prints output in the terminal, it worked.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Speaker-Diarization

This project contains:
* Text-independent Speaker recognition module based on VGG-Speaker-recognition
* Speaker diarization based on UIS-RNN.</br>
* *Mainly borrowed from UIS-RNN and VGG-Speaker-recognition, just link the 2 projects by generating speaker embeddings to make everything easier, and also provide an intuitive display panel*
## Prerequisites
1. pytorch 1.3.0
2. keras
3. Tensorflow 1.8-1.15
4. pyaudio (About how to install on windows, refer to [pyaudio_portaudio](https://github.com/intxcc/pyaudio_portaudio))

## Outline
### 1. Speaker recognition.</br>
    cd ghostvlad
    python predict.py

The confusion matrix of 4 persons utterances is as below

0.00  0.32  0.40  | 0.70  0.62  0.76  | 0.81  0.83  0.76  | 0.92  0.83  0.89  |

0.32  0.00  0.48  | 0.68  0.58  0.76  | 0.87  0.84  0.83  | 0.92  0.82  0.86  |

0.40  0.48  0.00  | 0.71  0.65  0.74  | 0.79  0.81  0.72  | 0.90  0.84  0.85  |

********************************************************************************

0.70  0.68  0.71  | 0.00  0.35  0.30  | 0.78  0.81  0.76  | 0.80  0.81  0.80  |

0.62  0.58  0.65  | 0.35  0.00  0.45  | 0.76  0.71  0.73  | 0.82  0.77  0.77  |

0.76  0.76  0.74  | 0.30  0.45  0.00  | 0.83  0.83  0.80  | 0.83  0.84  0.80  |

********************************************************************************

0.81  0.87  0.79  | 0.78  0.76  0.83  | 0.00  0.40  0.46  | 0.76  0.80  0.86  |

0.83  0.84  0.81  | 0.81  0.71  0.83  | 0.40  0.00  0.45  | 0.80  0.78  0.82  |

0.76  0.83  0.72  | 0.76  0.73  0.80  | 0.46  0.45  0.00  | 0.85  0.85  0.84  |

********************************************************************************

0.92  0.92  0.90  | 0.80  0.82  0.83  | 0.76  0.80  0.85  | 0.00  0.41  0.44  |

0.83  0.82  0.84  | 0.81  0.77  0.84  | 0.80  0.78  0.85  | 0.41  0.00  0.41  |

0.89  0.86  0.85  | 0.80  0.77  0.80  | 0.86  0.82  0.84  | 0.44  0.41  0.00  |

********************************************************************************

Thanks to the authors of VGG, they are kind enough to provide the code and pre-trained model.
Their paper can refer to [UTTERANCE-LEVEL AGGREGATION FOR SPEAKER RECOGNITION IN THE WILD](https://arxiv.org/pdf/1902.10107.pdf)</br>
It's a novel idea that combines `netvlad/ghostvlad` which popularly used in image recognition to speaker recognition, the state-of-the-art in the past was `i-vector` based, which depended on the `GMM` model and `pLDA`.

About VGG speaker model, I have re-implemented in tensorflow, [ghostvlad-speaker](https://github.com/taylorlu/ghostvlad-speaker) and corresponding pretrained model.

This project only shows how to generate speaker embeddings using pre-trained model for uis-rnn training in later.</br>
The training project link to [VGG-Speaker-Recognition](https://github.com/WeidiXie/VGG-Speaker-Recognition)
#### Dataset
  1. http://www.openslr.org/38 contains 855 speakers and 120 utterances of Chinese Mandarin in each, so there are 102600 utterances in total.</br>
  2. [VCTK](https://datashare.is.ed.ac.uk/handle/10283/2651) contains 109 speakers of English.</br>
  3. [VoxCeleb1](http://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1.html) contains 1251 speakers.</br>
  4. [VoxCeleb2](http://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox2.html) contains 6112 speakers.</br>
  **How to generate speaker embeddings for the next training stage:**</br>
  `python generate_embeddings.py` </br>
  You may need to change the dataset path by your own.
 
### 2. Speaker diarization.</br>
![diarization](https://github.com/taylorlu/Speaker-Diarization/blob/master/resources/diarization.gif)
#### Training
    python train.py
The speaker embeddings generated by vgg are all non-negative vectors, and contained many **zero** elements. The uis-rnn seems abnormally deal with these data somehow, shows as below

Iter: 0  	Training Loss: nan    
    Negative Log Likelihood: 7.3020	Sigma2 Prior: nan	Regularization: 0.0007
    Iter: 10  	Training Loss: nan    
    Negative Log Likelihood: nan	Sigma2 Prior: nan	Regularization: nan
    Iter: 20  	Training Loss: nan    
    Negative Log Likelihood: nan	Sigma2 Prior: nan	Regularization: nan
        
When I added an insignificate bias (e.g. 0.00001) to each element of vectors, error disappeared.

Iter: 0  	Training Loss: -581.8732    
    Negative Log Likelihood: 7.0125	Sigma2 Prior: -588.8864	Regularization: 0.0007
    Iter: 10  	Training Loss: -614.1193    
    Negative Log Likelihood: 1.7536	Sigma2 Prior: -615.8737	Regularization: 0.0007
    Iter: 20  	Training Loss: -644.9244    
    Negative Log Likelihood: 1.7123	Sigma2 Prior: -646.6375	Regularization: 0.0007

#### Clustering
`python speakerDiarization.py`

The Result is showing as below:(3 speakers)

========= 0 =========
    0:00.288 ==> 0:04.406
    0:07.699 ==> 0:16.461
    0:33.921 ==> 0:35.8
    ========= 1 =========
    0:04.406 ==> 0:07.699
    0:16.461 ==> 0:19.594
    0:30.371 ==> 0:33.921
    0:41.19 ==> 0:44.185
    ========= 2 =========
    0:19.594 ==> 0:30.371
    0:35.8 ==> 0:41.19

The final result is influenced by the size of each window and the overlap rate.
When the overlap is too large, the uis-rnn perhaps generates fewer speakers since the speaker embeddings changed smoothly, otherwise will generate more speakers.
And also, the window size cannot be too short, it must contain enough information to generate more discrimitive speaker embeddings.