Speaker-Diarization - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #react #ai #llm #go #golang #rust #java #awesome #kubernetes #awesome-list #pytorch #ios

Speaker-Diarization

(★ 502)

speaker diarization by uis-rnn and speaker embedding by vgg-speaker-recognition

파일 탐색기

최종 버전 다운로드 (.zip)

- arguments.html
- evals.html
- index.html
- loss_func.html
- uisrnn.html
- utils.html
- - a_1.wav
  - a_2.wav
  - a_3.wav
  - b_1.wav
  - b_2.wav
  - b_3.wav
  - x_1.wav
  - x_2.wav
  - x_3.wav
  - z_1.wav
  - z_2.wav
  - z_3.wav
- - weights.h5
- backbone.py
- generate_embeddings.py
- model.py
- predict.py
- preprocess.py
- toolkits.py
- utils.py
- saved_model.uisrnn_benchmark
- diarization.gif
- uisrnn.gif
- YouTube_lecture_screenshot.png
- evals_test.py
- integration_test.py
- uisrnn_test.py
- utils_test.py
- __init__.py
- arguments.py
- evals.py
- loss_func.py
- uisrnn.py
- utils.py
- player.py
- viewer.py
- rmdmy.wav
.coveragerc
.gitignore
.travis.yml
CONTRIBUTING.md
LICENSE
publish.sh
README.md
requirements.txt
run_pdoc3.sh
run_pylint.sh
run_tests.sh
setup.py
speakerDiarization.py
train.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/taylorlu/Speaker-Diarization

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd Speaker-Diarization

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Speaker-Diarization

This project contains:
* Text-independent Speaker recognition module based on VGG-Speaker-recognition
* Speaker diarization based on UIS-RNN.</br>
* *Mainly borrowed from UIS-RNN and VGG-Speaker-recognition, just link the 2 projects by generating speaker embeddings to make everything easier, and also provide an intuitive display panel*
## Prerequisites
1. pytorch 1.3.0
2. keras
3. Tensorflow 1.8-1.15
4. pyaudio (About how to install on windows, refer to [pyaudio_portaudio](https://github.com/intxcc/pyaudio_portaudio))

## Outline
### 1. Speaker recognition.</br>
    cd ghostvlad
    python predict.py

The confusion matrix of 4 persons utterances is as below

0.00  0.32  0.40  | 0.70  0.62  0.76  | 0.81  0.83  0.76  | 0.92  0.83  0.89  |

0.32  0.00  0.48  | 0.68  0.58  0.76  | 0.87  0.84  0.83  | 0.92  0.82  0.86  |

0.40  0.48  0.00  | 0.71  0.65  0.74  | 0.79  0.81  0.72  | 0.90  0.84  0.85  |

********************************************************************************

0.70  0.68  0.71  | 0.00  0.35  0.30  | 0.78  0.81  0.76  | 0.80  0.81  0.80  |

0.62  0.58  0.65  | 0.35  0.00  0.45  | 0.76  0.71  0.73  | 0.82  0.77  0.77  |

0.76  0.76  0.74  | 0.30  0.45  0.00  | 0.83  0.83  0.80  | 0.83  0.84  0.80  |

********************************************************************************

0.81  0.87  0.79  | 0.78  0.76  0.83  | 0.00  0.40  0.46  | 0.76  0.80  0.86  |

0.83  0.84  0.81  | 0.81  0.71  0.83  | 0.40  0.00  0.45  | 0.80  0.78  0.82  |

0.76  0.83  0.72  | 0.76  0.73  0.80  | 0.46  0.45  0.00  | 0.85  0.85  0.84  |

********************************************************************************

0.92  0.92  0.90  | 0.80  0.82  0.83  | 0.76  0.80  0.85  | 0.00  0.41  0.44  |

0.83  0.82  0.84  | 0.81  0.77  0.84  | 0.80  0.78  0.85  | 0.41  0.00  0.41  |

0.89  0.86  0.85  | 0.80  0.77  0.80  | 0.86  0.82  0.84  | 0.44  0.41  0.00  |

********************************************************************************

Thanks to the authors of VGG, they are kind enough to provide the code and pre-trained model.
Their paper can refer to [UTTERANCE-LEVEL AGGREGATION FOR SPEAKER RECOGNITION IN THE WILD](https://arxiv.org/pdf/1902.10107.pdf)</br>
It's a novel idea that combines `netvlad/ghostvlad` which popularly used in image recognition to speaker recognition, the state-of-the-art in the past was `i-vector` based, which depended on the `GMM` model and `pLDA`.

About VGG speaker model, I have re-implemented in tensorflow, [ghostvlad-speaker](https://github.com/taylorlu/ghostvlad-speaker) and corresponding pretrained model.

This project only shows how to generate speaker embeddings using pre-trained model for uis-rnn training in later.</br>
The training project link to [VGG-Speaker-Recognition](https://github.com/WeidiXie/VGG-Speaker-Recognition)
#### Dataset
  1. http://www.openslr.org/38 contains 855 speakers and 120 utterances of Chinese Mandarin in each, so there are 102600 utterances in total.</br>
  2. [VCTK](https://datashare.is.ed.ac.uk/handle/10283/2651) contains 109 speakers of English.</br>
  3. [VoxCeleb1](http://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1.html) contains 1251 speakers.</br>
  4. [VoxCeleb2](http://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox2.html) contains 6112 speakers.</br>
  **How to generate speaker embeddings for the next training stage:**</br>
  `python generate_embeddings.py` </br>
  You may need to change the dataset path by your own.
 
### 2. Speaker diarization.</br>
![diarization](https://github.com/taylorlu/Speaker-Diarization/blob/master/resources/diarization.gif)
#### Training
    python train.py
The speaker embeddings generated by vgg are all non-negative vectors, and contained many **zero** elements. The uis-rnn seems abnormally deal with these data somehow, shows as below

Iter: 0  	Training Loss: nan    
    Negative Log Likelihood: 7.3020	Sigma2 Prior: nan	Regularization: 0.0007
    Iter: 10  	Training Loss: nan    
    Negative Log Likelihood: nan	Sigma2 Prior: nan	Regularization: nan
    Iter: 20  	Training Loss: nan    
    Negative Log Likelihood: nan	Sigma2 Prior: nan	Regularization: nan
        
When I added an insignificate bias (e.g. 0.00001) to each element of vectors, error disappeared.

Iter: 0  	Training Loss: -581.8732    
    Negative Log Likelihood: 7.0125	Sigma2 Prior: -588.8864	Regularization: 0.0007
    Iter: 10  	Training Loss: -614.1193    
    Negative Log Likelihood: 1.7536	Sigma2 Prior: -615.8737	Regularization: 0.0007
    Iter: 20  	Training Loss: -644.9244    
    Negative Log Likelihood: 1.7123	Sigma2 Prior: -646.6375	Regularization: 0.0007

#### Clustering
`python speakerDiarization.py`

The Result is showing as below:(3 speakers)

========= 0 =========
    0:00.288 ==> 0:04.406
    0:07.699 ==> 0:16.461
    0:33.921 ==> 0:35.8
    ========= 1 =========
    0:04.406 ==> 0:07.699
    0:16.461 ==> 0:19.594
    0:30.371 ==> 0:33.921
    0:41.19 ==> 0:44.185
    ========= 2 =========
    0:19.594 ==> 0:30.371
    0:35.8 ==> 0:41.19

The final result is influenced by the size of each window and the overlap rate.
When the overlap is too large, the uis-rnn perhaps generates fewer speakers since the speaker embeddings changed smoothly, otherwise will generate more speakers.
And also, the window size cannot be too short, it must contain enough information to generate more discrimitive speaker embeddings.