soxan - GitLite

KO | EN

gitlite — search

#python #react #ai #typescript #golang #go #rust #deep-learning #nodejs #cli #kubernetes #vue #mcp #open-source

soxan

(★ 277)

Wav2Vec for speech recognition, classification, and audio classification

파일 탐색기

최종 버전 다운로드 (.zip)

- - sync_streamlit_speech_text_emotion_to_space.yml
- Eating_Sound_Collection_using_Wav2Vec2.ipynb
- Emotion_recognition_in_Greek_speech_using_HuBERT.ipynb
- Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb
- Regression_Example_Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb
- __init__.py
- collator.py
- modeling_outputs.py
- models.py
- trainer.py
.gitignore
LICENSE
README.md
requirements.txt
run_wav2vec_clf.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/m3hrdadfi/soxan

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd soxan

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

jupyter notebook

브라우저에서 노트북(.ipynb) 파일들을 열람하고 실행할 수 있는 Jupyter 화면을 켭니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Soxan

> در زبان پارسی به نام سخن

This repository consists of models, scripts, and notebooks that help you to use all the benefits of Wav2Vec 2.0 in your
research. In the following, I'll show you how to train speech tasks in your dataset and how to use the pretrained
models.

## How to train

I'm just at the beginning of all the possible speech tasks. To start, we continue the training script with the speech
emotion recognition problem.

### Training - Notebook

| Task                                     | Notebook                                                                                                                                                                                                            |
|------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Speech Emotion Recognition (Wav2Vec 2.0) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb) |
| Speech Emotion Recognition (Hubert)      | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_HuBERT.ipynb)   |
| Audio Classification (Wav2Vec 2.0)       | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Eating_Sound_Collection_using_Wav2Vec2.ipynb)             |

### Training - CMD

```bash
python3 run_wav2vec_clf.py \
    --pooling_mode="mean" \
    --model_name_or_path="lighteternal/wav2vec2-large-xlsr-53-greek" \
    --model_mode="wav2vec2" \ # or you can use hubert
    --output_dir=/path/to/output \
    --cache_dir=/path/to/cache/ \
    --train_file=/path/to/train.csv \
    --validation_file=/path/to/dev.csv \
    --test_file=/path/to/test.csv \
    --per_device_train_batch_size=4 \
    --per_device_eval_batch_size=4 \
    --gradient_accumulation_steps=2 \
    --learning_rate=1e-4 \
    --num_train_epochs=5.0 \
    --evaluation_strategy="steps"\
    --save_steps=100 \
    --eval_steps=100 \
    --logging_steps=100 \
    --save_total_limit=2 \
    --do_eval \
    --do_train \
    --fp16 \
    --freeze_feature_extractor
```

### Prediction

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchaudio
from transformers import AutoConfig, Wav2Vec2FeatureExtractor
from src.models import Wav2Vec2ForSpeechClassification, HubertForSpeechClassification

model_name_or_path = "path/to/your-pretrained-model"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
config = AutoConfig.from_pretrained(model_name_or_path)
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_name_or_path)
sampling_rate = feature_extractor.sampling_rate

# for wav2vec
model = Wav2Vec2ForSpeechClassification.from_pretrained(model_name_or_path).to(device)

# for hubert
model = HubertForSpeechClassification.from_pretrained(model_name_or_path).to(device)

def speech_file_to_array_fn(path, sampling_rate):
    speech_array, _sampling_rate = torchaudio.load(path)
    resampler = torchaudio.transforms.Resample(_sampling_rate, sampling_rate)
    speech = resampler(speech_array).squeeze().numpy()
    return speech

def predict(path, sampling_rate):
    speech = speech_file_to_array_fn(path, sampling_rate)
    inputs = feature_extractor(speech, sampling_rate=sampling_rate, return_tensors="pt", padding=True)
    inputs = {key: inputs[key].to(device) for key in inputs}

with torch.no_grad():
        logits = model(**inputs).logits

scores = F.softmax(logits, dim=1).detach().cpu().numpy()[0]
    outputs = [{"Emotion": config.id2label[i], "Score": f"{round(score * 100, 3):.1f}%"} for i, score in
               enumerate(scores)]
    return outputs

path = "/path/to/disgust.wav"
outputs = predict(path, sampling_rate)    
```

Output:

```bash
[
    {'Emotion': 'anger', 'Score': '0.0%'},
    {'Emotion': 'disgust', 'Score': '99.2%'},
    {'Emotion': 'fear', 'Score': '0.1%'},
    {'Emotion': 'happiness', 'Score': '0.3%'},
    {'Emotion': 'sadness', 'Score': '0.5%'}
]
```

## Demos

| Demo                                                     | Link                                                                                                               |
|----------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
| Speech To Text With Emotion Recognition (Persian) - soon | [huggingface.co/spaces/m3hrdadfi/speech-text-emotion](https://huggingface.co/spaces/m3hrdadfi/speech-text-emotion) |

## Models

| Dataset                                                                                                                      | Model                                                                                                                                           |
|------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|
| [ShEMO: a large-scale validated database for Persian speech emotion detection](https://github.com/mansourehk/ShEMO)          | [m3hrdadfi/wav2vec2-xlsr-persian-speech-emotion-recognition](https://huggingface.co/m3hrdadfi/wav2vec2-xlsr-persian-speech-emotion-recognition) |
| [ShEMO: a large-scale validated database for Persian speech emotion detection](https://github.com/mansourehk/ShEMO)          | [m3hrdadfi/hubert-base-persian-speech-emotion-recognition](https://huggingface.co/m3hrdadfi/hubert-base-persian-speech-emotion-recognition)     |
| [ShEMO: a large-scale validated database for Persian speech emotion detection](https://github.com/mansourehk/ShEMO)          | [m3hrdadfi/hubert-base-persian-speech-gender-recognition](https://huggingface.co/m3hrdadfi/hubert-base-persian-speech-gender-recognition)       |
| [Speech Emotion Recognition (Greek) (AESDD)](http://m3c.web.auth.gr/research/aesdd-speech-emotion-recognition/)              | [m3hrdadfi/hubert-large-greek-speech-emotion-recognition](https://huggingface.co/m3hrdadfi/hubert-large-greek-speech-emotion-recognition)       |
| [Speech Emotion Recognition (Greek) (AESDD)](http://m3c.web.auth.gr/research/aesdd-speech-emotion-recognition/)              | [m3hrdadfi/hubert-base-greek-speech-emotion-recognition](https://huggingface.co/m3hrdadfi/hubert-base-greek-speech-emotion-recognition)         |
| [Speech Emotion Recognition (Greek) (AESDD)](http://m3c.web.auth.gr/research/aesdd-speech-emotion-recognition/)              | [m3hrdadfi/wav2vec2-xlsr-greek-speech-emotion-recognition](https://huggingface.co/m3hrdadfi/wav2vec2-xlsr-greek-speech-emotion-recognition)     |
| [Eating Sound Collection](https://www.kaggle.com/mashijie/eating-sound-collection)                                           | [m3hrdadfi/wav2vec2-base-100k-eating-sound-collection](https://huggingface.co/m3hrdadfi/wav2vec2-base-100k-eating-sound-collection)             |
| [GTZAN Dataset - Music Genre Classification](https://www.kaggle.com/andradaolteanu/gtzan-dataset-music-genre-classification) | [m3hrdadfi/wav2vec2-base-100k-gtzan-music-genres](https://huggingface.co/m3hrdadfi/wav2vec2-base-100k-gtzan-music-genres)                       |