dcase2023-audio-retrieval

KO | EN

gitlite — search

#javascript #react #llm #go #docker #macos #java #nodejs #awesome #kubernetes #vue #awesome-list #mcp #ios

dcase2023-audio-retrieval

(★ 10)

Baseline system for Language-based Audio Retrieval (Task 6B) in DCASE 2023 Challenge

dcase2023-audio-retrieval 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Language-based Audio Retrieval in DCASE 2023 Challenge

This repository provides the baseline system for **Language-based Audio Retrieval** (Task 6B) in DCASE 2023 Challenge.

**2023/03/20 Update:**
Training checkpoints for the baseline system and its audio encoder are available on Zenodo:
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.7752975.svg)](https://doi.org/10.5281/zenodo.7752975).

![Language-based Audio Retrieval](https://raw.githubusercontent.com/xieh97/dcase2023-audio-retrieval/master/figs/dcase2023_task_6b.png)

# Baseline Retrieval System

![Baseline Retrieval System](https://raw.githubusercontent.com/xieh97/dcase2023-audio-retrieval/master/figs/baseline_system.png)

```
- Audio Encoder                   # fine-tuned PANNs, i.e., CNN14
- Text Encoder                    # pretrained Sentence-BERT, i.e., all-mpnet-base-v2
- Contrastive Learning Objective  # InfoNCE loss
```

# Quick Start

This codebase is developed with Python 3.9 and [PyTorch 1.13.0](https://pytorch.org/).

1. Check out source code and install required python packages:

```
git clone https://github.com/xieh97/dcase2023-audio-retrieval.git
pip install -r requirements.txt
```

2. Download the [Clotho](https://zenodo.org/record/4783391) dataset:

```
Clotho
├─ clotho_captions_development.csv
├─ clotho_captions_validation.csv
├─ clotho_captions_evaluation.csv
├─ development
│   └─...(3839 wavs)
├─ validation
│   └─...(1045 wavs)
└─ evaluation
    └─...(1045 wavs)
```

3. Pre-process audio and caption data:

```
preprocessing
├─ audio_logmel.py              # extract log-mel energies from audio clips
├─ clotho_dataset.py            # process audio captions, generate fids and cids
├─ sbert_embeddings.py          # generate sentence embeddings using Sentence-BERT (all-mpnet-base-v2)
└─ cnn14_transfer.py            # transfer pretrained CNN14 (Cnn14_mAP=0.431.pth)
```

4. Train the baseline system:

```
models
├─ core.py                      # dual-encoder framework
├─ audio_encoders.py            # audio encoders
└─ text_encoders.py             # text encoders

utils
├─ criterion_utils.py           # loss functions
├─ data_utils.py                # Pytorch dataset classes
└─ model_utils.py               # model.train(), model.eval(), etc.

conf.yaml                       # experimental settings
main.py                         # main()
```

5. Calculate retrieval metrics:

```
postprocessing
├─ xmodal_scores.py             # calculate audio-text scores
└─ xmodal_retrieval.py          # calculate mAP, R@1, R@5, R@10, etc.
```

# Examples

1. Code example for using the pretrained audio encoder:

```
example
├─ audio_encoder.py             # code example for audio encoder
├─ example.wav                  # audio segment example
└─ audio_encoder.pth            # audio encoder checkpoint (https://doi.org/10.5281/zenodo.7752975)
```

dcase2023-audio-retrieval

dcase2023-audio-retrieval 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

dcase2023-audio-retrieval

dcase2023-audio-retrieval 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README