mispronunciation-detection

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

mispronunciation-detection

★ 19 Open GitHub ↗

Mispronunciation detection code for jingju singing voice

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Mispronunciation detection
Mispronunciation detection code for jingju singing voice.

Rong Gong's thesis "Automatic assessment of singing voice pronunciation: 
a case study with jingju music" chapter 6.

This repo contains two methods:

* Baseline - forced alignment system built on Kaldi.
* Deep learning - discriminative model built using Keras and Tensorflow.

## Baseline
The main idea of the forced alignment-based mispronunciation detect is to use
two lexicons respectively for training and testing phases. The detail of this
idea is described in section 6.2.1 in the thesis.

We here only explain the general pipeline of the model training and testing. Please
write to the author if you want to know how to use the code for your own
experiment. Pipeline:

1. generate language dictionary by using `srcPy/parseLang.py`.
2. generate all the files that Kaldi need, e.g., text, wav.scp, phone.txt, by
`srcPy/parseTextRepetition.py`.
3. run the model training and decode the text for test data by `run.sh`
4. parse decoded pronunciation by `srcPy/parse_decoded_pronunciation.py`
5. evaluation `srcPy/mispron_eval.py`

## Deep learning-based discriminative model

We built discriminative models for mispronunciation detection. Two types of model
are built, one for special pronunciation, another for jiantuanzi syllables. We have
experimented several deep learning architectures, such as BiLSTM, CNN, attention, 
Temporal convolutional networks (TCNs), self-attention. The details are described in 
sections 6.3 and 6.4 in the thesis. Here, we also only describe the pipeline of model 
training and testing. Please write to the auther if you want to use the code for your own
experiment. Pipeline:

1. collecting training logarithmic Mel representation by `training_sample_collection_syllable.py`
2. train various deep learning architectures by using `train_rnn_jianzi.py` or `train_rnn_special.py` respectively for
special pronunciation and jiantuanzi models. e.g., attention var can be `feedforward` or `selfatt`.
3. train TCNs architectures by using `train_rnn_special_tcn.py` and `train_rnn_jianzi_tcn.py`
4. evaluation by `eval.py`

## Contact
Rong Gong: rong.gong\<at\>upf.edu

## Code license
GNU Affero General Public License 3.0

mispronunciation-detection

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

mispronunciation-detection

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README