WatchYourMouth

KO | EN

gitlite — search

#python #docker #javascript #react #deep-learning #android #json #tensorflow #php #java #csharp #reinforcement-learning

★ 23 Open GitHub ↗

Implementation for WatchYourMouth: Silent Speech Recognition with Depth Sensing presented at CHI 2024

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# WatchYourMouth: Silent Speech Recognition with Depth Sensing
![Teaser Image](https://raw.githubusercontent.com/hilab-open-source/WatchYourMouth/main/img/Teaser.png)
<div>
    <h4 align="center">
        Authors:&nbsp
        <a href='https://xueewang.github.io/' target='_blank'>Xue Wang</a>,&nbsp;
        <a href='https://www.shawnsu.net/' target='_blank'>Zixiong Su</a>,&nbsp;
        <a href='https://lab.rekimoto.org/members/rekimoto/' target='_blank'>Jun Rekimoto</a>,&nbsp;
        <a href='https://yangzhang.dev/' target='_blank'>Yang Zhang</a>
    </h4>
</div>

## Introduction
Silent speech recognition is a promising technology that decodes human speech without requiring audio signals, enabling private human-computer interactions. In this paper, we propose Watch Your Mouth, a novel method that leverages depth sensing to enable accurate silent speech recognition. By leveraging depth information, our method provides unique resilience against environmental factors such as variations in lighting and device orientations, while further addressing privacy concerns by eliminating the need for sensitive RGB data. We started by building a deep-learning model that locates lips using depth data. We then designed a deep learning pipeline to efficiently learn from point clouds and translate lip movements into commands and sentences. We evaluated our technique and found it effective across diverse sensor locations: On-Head, On-Wrist, and In-Environment. Watch Your Mouth outperformed the state-of-the-art RGB-based method, demonstrating its potential as an accurate and reliable input technique.

## Installation
#### STEP1: Create Environment
```
conda create -n <YourEnvName> python=3.8
pip install tqdm textblob editdistance einops
```
To ensure compatibility between PyTorch and CUDA, you should first determine the version of CUDA installed on your system. You can do this by running the following command in your terminal:
```
nvcc --version
```
Once you have your CUDA version, visit the [PyTorch official installation guide](https://pytorch.org/get-started/previous-versions/) to find the PyTorch version that corresponds to your CUDA version, for example:
```
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia
```
#### STEP2: Clone Repository
```
git clone https://github.com/hilab-open-source/WatchYourMouth.git
```
Compile the [Pointnet2](https://github.com/facebookresearch/votenet/tree/main/pointnet2) CUDA layers with the following code:
```
cd WatchYourMouth/modules
pip install .
```
#### STEP3: Download the Dataset
Please download the dataset [HERE](https://drive.google.com/drive/folders/174mlRrNpxAdqMASRp7cAU4d0iCTQk7SA) and store it in the directory.

#### STEP4: Start Training
Update the dataset path to the location where your dataset is stored in this [line](https://github.com/hilab-open-source/WatchYourMouth/blob/ebda2232ab9efbe4d31c941b694d9d8ca1172ff4/train_sentences.py#L210), and start training with the following code:
```
cd WatchYourMouth/
python train_sentences.py
```

## To-Do List

- [x] Code Release
- [x] Dataset Release
- [x] Tutorial Document (Released)

## References
Here are some great resources we benefit:
- P4Transformer: https://github.com/hehefan/P4Transformer
- Point Spatio-Temporal Transformer: https://github.com/hehefan/PST-Transformer
- PointNet: https://github.com/charlesq34/pointnet
- PointNet++: https://github.com/charlesq34/pointnet2
- LipNet-Pytorch: https://github.com/VIPL-Audio-Visual-Speech-Understanding/LipNet-PyTorch

## Citation
If you find our work useful in your research, please consider citing:
```
@inproceedings{wang2024watch,
  title={Watch Your Mouth: Silent Speech Recognition with Depth Sensing},
  author={Wang, Xue and Su, Zixiong and Rekimoto, Jun and Zhang, Yang},
  booktitle={Proceedings of the CHI Conference on Human Factors in Computing Systems},
  pages={1--15},
  year={2024}
}
```
## License
This project is licensed under <a rel="license" href="./LICENSE">MIT License</a>. Redistribution and use should follow this license.

WatchYourMouth

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README