SSPL

KO | EN

gitlite — search

#python #automation #react #typescript #javascript #hacktoberfest #django #nlp #docker #ios #cli #go

★ 32 Open GitHub ↗

PyTorch code for "Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes" (CVPR, 2022)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Self-Supervised Predictive Learning (SSPL)
This repository hosts the PyTorch code for our self-supervised sound source localization method.

## Paper
**Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes** <br/>
[Zengjie Song](https://zjsong.github.io/), 
[Yuxi Wang](https://scholar.google.com/citations?hl=en&user=waLCodcAAAAJ), 
[Junsong Fan](https://scholar.google.com/citations?user=AfK4UcUAAAAJ&hl=en&oi=ao), 
[Tieniu Tan](http://cripac.ia.ac.cn/tnt/), 
[Zhaoxiang Zhang](https://zhaoxiangzhang.net/) <br/>
In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2022 <br/>
[[Paper](https://zjsong.github.io/index_files/2022cvpr_sspl.pdf)] [[arXiv](https://arxiv.org/abs/2203.13412)] <br/>

> **Abstract:** *Sound source localization in visual scenes aims to localize objects emitting the sound in a given image. 
> Recent works showing impressive localization performance typically rely on the contrastive learning framework. 
> However, the random sampling of negatives, as commonly adopted in these methods, can result in misalignment between audio 
> and visual features and thus inducing ambiguity in localization. In this paper, instead of following previous literature, 
> we propose Self-Supervised Predictive Learning (SSPL), a negative-free method for sound localization via explicit positive mining. 
> Specifically, we first devise a three-stream network to elegantly associate sound source with two augmented views of 
> one corresponding video frame, leading to semantically coherent similarities between audio and visual features. 
> Second, we introduce a novel predictive coding module for audio-visual feature alignment. Such a module assists SSPL to 
> focus on target objects in a progressive manner and effectively lowers the positive-pair learning difficulty. 
> Experiments show surprising results that SSPL outperforms the state-of-the-art approach on two standard sound 
> localization benchmarks. In particular, SSPL achieves significant improvements of 8.6% cIoU and 3.4% AUC on SoundNet-Flickr 
> compared to the previous best.*

## Requirements
We have tested the code on the following environment:
* Python 3.8.6 | PyTorch 1.6.0 | torchvision 0.7.0 | CUDA 10.1 | Ubuntu 16.04.7

## Download & pre-process videos
We train models on respectively two video datasets: [SoundNet-Flickr](http://soundnet.csail.mit.edu/) and [VGG-Sound](https://www.robots.ox.ac.uk/~vgg/data/vggsound/), 
while testing on two corresponding benchmarks: [test set](https://github.com/ardasnck/learning_to_localize_sound_source) of SoundNet-Flickr 
and [test set](https://www.robots.ox.ac.uk/~vgg/research/lvs/) (i.e., VGG-Sound Source) of VGG-Sound. Videos are downloaded 
with [youtube-dl](https://github.com/ytdl-org/youtube-dl) if only the YouTube IDs are given. Please see main text (Sec. 4.2) 
for details of pre-processing video frames and audio signals. To improve data loading efficiency, we use [h5py](https://docs.h5py.org/en/stable/build.html) 
to respectively group frames, audios, and spectrograms, before training.

## Usage
### Training
We utilize [VGG16](https://arxiv.org/pdf/1409.1556.pdf%E3%80%82) and [VGGish](https://github.com/harritaylor/torchvggish) as backbones 
to extract visual and audio features, respectively. Before training, you need to place pre-trained VGGish weights, 
i.e., [vggish-10086976.pth](https://github.com/harritaylor/torchvggish/releases/download/v0.1/vggish-10086976.pth), 
in ```models/torchvggish/torchvggish/vggish_pretrained/```. To train SSPL on SoundNet-Flickr10k with default setting, simply run:
```
python main.py
```
**Note:** We found that learning rates have vital influence on SSPL's performance. So we suggest that using the early stopping strategy 
to select hyper-parameters and avoid overfitting.

### Test
After training, ```frame_best.pth```, ```sound_best.pth```, ```ssl_head_best.pth``` (and ```pcm_best.pth``` for SSPL (w/ PCM)) 
can be obtained, and you need to place them in ```models/pretrain/``` before testing. To test SSPL on SoundNet-Flickr 
with default setting, simply run:
```
python test.py
```

## Citation
Please consider citing our paper in your publications if the project helps your research.
```
@inproceedings{song2022sspl,
  title={Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes},
  author={Song, Zengjie and Wang, Yuxi and Fan, Junsong and Tan, Tieniu and Zhang, Zhaoxiang},
  booktitle={Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},
  pages={3222--3231},
  year={2022}
}
```

## Acknowledgement
Our code is partially based on [Attention](https://github.com/ardasnck/learning_to_localize_sound_source), 
[HardWay](https://github.com/hche11/Localizing-Visual-Sounds-the-Hard-Way), and [SimSiam](https://github.com/PatrickHua/SimSiam). 
We thank the authors for sharing their code.

SSPL

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README