SSPL

KO | EN

gitlite — search

#javascript #hacktoberfest #react #typescript #llm #golang #rust #linux #macos #java #windows #awesome #security #ios

(★ 32)

PyTorch code for "Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes" (CVPR, 2022)

SSPL 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Self-Supervised Predictive Learning (SSPL)
This repository hosts the PyTorch code for our self-supervised sound source localization method.

## Paper
**Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes** <br/>
[Zengjie Song](https://zjsong.github.io/), 
[Yuxi Wang](https://scholar.google.com/citations?hl=en&user=waLCodcAAAAJ), 
[Junsong Fan](https://scholar.google.com/citations?user=AfK4UcUAAAAJ&hl=en&oi=ao), 
[Tieniu Tan](http://cripac.ia.ac.cn/tnt/), 
[Zhaoxiang Zhang](https://zhaoxiangzhang.net/) <br/>
In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2022 <br/>
[[Paper](https://zjsong.github.io/index_files/2022cvpr_sspl.pdf)] [[arXiv](https://arxiv.org/abs/2203.13412)] <br/>

> **Abstract:** *Sound source localization in visual scenes aims to localize objects emitting the sound in a given image. 
> Recent works showing impressive localization performance typically rely on the contrastive learning framework. 
> However, the random sampling of negatives, as commonly adopted in these methods, can result in misalignment between audio 
> and visual features and thus inducing ambiguity in localization. In this paper, instead of following previous literature, 
> we propose Self-Supervised Predictive Learning (SSPL), a negative-free method for sound localization via explicit positive mining. 
> Specifically, we first devise a three-stream network to elegantly associate sound source with two augmented views of 
> one corresponding video frame, leading to semantically coherent similarities between audio and visual features. 
> Second, we introduce a novel predictive coding module for audio-visual feature alignment. Such a module assists SSPL to 
> focus on target objects in a progressive manner and effectively lowers the positive-pair learning difficulty. 
> Experiments show surprising results that SSPL outperforms the state-of-the-art approach on two standard sound 
> localization benchmarks. In particular, SSPL achieves significant improvements of 8.6% cIoU and 3.4% AUC on SoundNet-Flickr 
> compared to the previous best.*

## Requirements
We have tested the code on the following environment:
* Python 3.8.6 | PyTorch 1.6.0 | torchvision 0.7.0 | CUDA 10.1 | Ubuntu 16.04.7

## Download & pre-process videos
We train models on respectively two video datasets: [SoundNet-Flickr](http://soundnet.csail.mit.edu/) and [VGG-Sound](https://www.robots.ox.ac.uk/~vgg/data/vggsound/), 
while testing on two corresponding benchmarks: [test set](https://github.com/ardasnck/learning_to_localize_sound_source) of SoundNet-Flickr 
and [test set](https://www.robots.ox.ac.uk/~vgg/research/lvs/) (i.e., VGG-Sound Source) of VGG-Sound. Videos are downloaded 
with [youtube-dl](https://github.com/ytdl-org/youtube-dl) if only the YouTube IDs are given. Please see main text (Sec. 4.2) 
for details of pre-processing video frames and audio signals. To improve data loading efficiency, we use [h5py](https://docs.h5py.org/en/stable/build.html) 
to respectively group frames, audios, and spectrograms, before training.

## Usage
### Training
We utilize [VGG16](https://arxiv.org/pdf/1409.1556.pdf%E3%80%82) and [VGGish](https://github.com/harritaylor/torchvggish) as backbones 
to extract visual and audio features, respectively. Before training, you need to place pre-trained VGGish weights, 
i.e., [vggish-10086976.pth](https://github.com/harritaylor/torchvggish/releases/download/v0.1/vggish-10086976.pth), 
in ```models/torchvggish/torchvggish/vggish_pretrained/```. To train SSPL on SoundNet-Flickr10k with default setting, simply run:
```
python main.py
```
**Note:** We found that learning rates have vital influence on SSPL's performance. So we suggest that using the early stopping strategy 
to select hyper-parameters and avoid overfitting.

### Test
After training, ```frame_best.pth```, ```sound_best.pth```, ```ssl_head_best.pth``` (and ```pcm_best.pth``` for SSPL (w/ PCM)) 
can be obtained, and you need to place them in ```models/pretrain/``` before testing. To test SSPL on SoundNet-Flickr 
with default setting, simply run:
```
python test.py
```

## Citation
Please consider citing our paper in your publications if the project helps your research.
```
@inproceedings{song2022sspl,
  title={Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes},
  author={Song, Zengjie and Wang, Yuxi and Fan, Junsong and Tan, Tieniu and Zhang, Zhaoxiang},
  booktitle={Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR)},
  pages={3222--3231},
  year={2022}
}
```

## Acknowledgement
Our code is partially based on [Attention](https://github.com/ardasnck/learning_to_localize_sound_source), 
[HardWay](https://github.com/hche11/Localizing-Visual-Sounds-the-Hard-Way), and [SimSiam](https://github.com/PatrickHua/SimSiam). 
We thank the authors for sharing their code.

SSPL

SSPL 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

SSPL

SSPL 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README