pyannote-audio_speaker-diarization_cpp

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

pyannote-audio_speaker-diarization_cpp

★ 22 Open GitHub ↗

C++ version of pyannote audio speaker diarizaiton pipeline

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Speaker Diarization Prototype in C++

Convert pyannote-audio's speaker diarization pipeline to C++.

Whole pipeline is splitted into 3 stages,
- segment
- embedding
- clustering

For segment and embedding part,
1. export embedding and segmentation from python code to onnx model, which later be used in c++.
2. convert pre-embedding, post-embedding, pre-segment, post-segment to c++.

For clustering, convert all python code to C++ code.

# Sibling Project
[Overlapped Speech Detectation in C++](https://github.com/leohuang2013/pyannote-audio_overlapped-speech-detection_cpp)

# Build

```
$> cd pipline
$> mkdir build && cd build
$> cmake ..
$> make
```
Build GPU version
```
$> cmake -DGPU=ON ..
$> make
```

GPU option is for libtorch. For onnxruntime, it is hard coded to be running on GPU.

# Run
Wave file must be 16k sample rate, 1 channel and 16bit.
``` bash
./speakerDiarizer ../model/segment2.onnx ../model/emd4.onnx ../data/multi-speaker_1min.wav
```

Example output,
``` bash
----------------------------------------------------
[5.22281 -- 17.7441] --> Speaker_3
[17.8116 -- 25.2197] --> Speaker_0
[25.3041 -- 28.4428] --> Speaker_3
[28.4428 -- 39.2934] --> Speaker_1
[39.2934 -- 47.2416] --> Speaker_3
[46.4484 -- 52.7934] --> Speaker_2
[52.5066 -- 58.4634] --> Speaker_3
```

# Model

## Export model
- segmenation
segment/export2.py

- embedding
embeddings/export3.py

If you want to export model by yourself, follow steps below, otherwise you may use
pre-exported models in model folder.

Create conda evironment
```
$> conda create --name sd_env
$> conda activate sd_env
$> cd embedding
$> python -m pip install -r requirements.txt
$> cd segment
$> python export2.py
$> cd embedding
$> python export3.py
```

# Verification
Since whole project is to translate pyannote-audio speaker diarization pipleline from python to C++, strategy I adopted here is 
write input/output of each small function in python to txt file, and do same for C++, then load txt file into python to compare 
and check difference. Target is to make each input and output is same.
For this purpose, script/verifyEveryStepResult.py is created.
``` bash
$> python verifyEveryStepResult.py
```
Above command is to compare txt files generated /tmp. and command below is to delete all the txt files.
``` bash
$> python verifyEveryStepResult.py clean
```

# Dependencies
- onnxruntime
- liborch
onnxruntime is used to inference for Segmentation and Embeddings. 
liborch is used to calculate STFT. STFT is part of segmentation module, we can export segmentation as whole, including STFT, however, 
there gonna be to relative big difference. It cannot even pass the verification which compare segmentation result generated by using
original model with pytorch and exported onnx model with onnxruntime in python. This is part is done in segment/export2.py.

Note: even verification passed in python code - export2.py, there are still tiny difference between result gererated by python onnxruntime 
and onnxruntime C++. For embeddings, absolute difference is 0.02. relative difference is 0.1, since clustering stage is very senstive 
to tiny difffernce which will leads to different result compared with result generated by pyannote-audio.

# Known issue

## Performance
Performance is main problem. Even it runs on GPU for pytorch part, more percisely, STFT part, it takes 356 seconds for test 
wave file included in data folder in a machine with nvidia 2070S and 16 threads CPU.

STFT is part of second stage of pipeline: Embedding.

## Accuracy
For some audio, there will be slight discrepency between this project and pyannote-audio.

# Target

Next step is to remove dependency to libtorch, which is currently only for STFT calcualtion.

Currently all tensor operations are implemented on our own, [xtensor](https://github.com/xtensor-stack/xtensor) could be an option, to replace them
for sake of performance.

# References

## For hierichical clustering
Have tried following methods,
- hclust-cpp/fastcluster
https://github.com/cdalitz/hclust-cpp
Result is wrong, including distance of clusters and result 'fcluster'

- agglomerative-hierarchical-clustering
https://github.com/gyaikhom/agglomerative-hierarchical-clustering/tree/master
The result returned by centroid_linkage is empty

- alglib
https://www.alglib.net/dataanalysis/clustering.php
It does not support centriod which is needed in this project.

# Thanks

- [pyannote-audio](https://github.com/pyannote/pyannote-audio)
- [pyannote-onnx](https://github.com/pengzhendong/pyannote-onnx)

pyannote-audio_speaker-diarization_cpp

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README