SignSpeak

KO | EN

gitlite — search

#typescript #android #rust #linux #nodejs #windows #awesome #kubernetes #vue #pytorch #security #ios #mcp #open-source

(★ 11)

This repository contains the code and dataset for the paper "SignSpeak: Open-Source Time Series Classification for ASL Translation."

SignSpeak 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# SignSpeak: Open-Source Time Series Classification for ASL Translation

This repository contains the code and dataset for the paper "SignSpeak: Time Series Classification for ASL Prediction."

**[Paper](https://arxiv.org/abs/2407.12020)**

## Overview

**Abstract**: The lack of fluency in sign language remains a barrier to seamless communication for hearing and speech-impaired communities. In this work, we propose a low-cost, real-time ASL-to-speech translation glove and an exhaustive training dataset of sign language patterns. We then benchmarked this dataset with supervised learning models, such as LSTMs, GRUs and Transformers, where our best model achieved 92\% accuracy. The SignSpeak dataset has 7200 samples encompassing 36 classes (A-Z, 1-10) and aims to capture realistic signing patterns by using five low-cost flex sensors to measure finger positions at each time step at 36 Hz. Our open-source dataset, models and glove designs, provide an accurate and efficient ASL translator while maintaining cost-effectiveness, establishing a framework for future work to build on.

## Data Glove

The glove uses
- **Flex Sensors**: Five flex sensors are integrated into the glove, one for each finger. These sensors measure the bend of each finger.
- **Microcontroller**: An Arudino MEGA 2560 processes the signals from the flex sensors and sends the data verial serial ouput to a database.

All code for glove setup and data collection can be found at ```src/dataCollection```

Below is the schematic and completed glove.

## Key Features

- **Open-Source Dataset**: The SignSpeak dataset comprises 7200 samples covering 36 classes (A-Z, 1-10), collected at 36 Hz using five flex sensors.
- **High Accuracy**: Achieves 92% categorical accuracy using state-of-the-art models such as LSTMs, GRUs, and Transformers.
- **Real-World Applicability**: Designed to be a cost-effective and resource-efficient solution for seamless communication for the hearing and speech-impaired communities.

## Models

The repository includes implementations and benchmarks for various models:
- Stacked LSTM
- Stacked GRU
- Transformer-based models

All models can be found in ``` src/models/ ```

## Getting Started

1. **Clone the Repository**
   ```bash
   git clone https://github.com/adityamakkar000/ASL-Sign-Research.git
   ```
2. **Download the Dataset**

Download the dataset from this [Harvard Dataverse](https://doi.org/10.7910/DVN/ODY7GH) and place it in the ```src/experiments/data``` directory naming it ```data.csv```.

3. **Install Dependencies**

```bash
    pip install -r req.txt
    ```
4. **Run the model**
     Run the models using the following bash command inside of the ```src/experiments/``` or use the training scripts found in the directory
    ```bash
    python LightningTrain.py \
          -layers $layers \
          -model $model \
          -hidden_size $hidden_size \
          -lr $lr \
          -time_steps $time_steps \
          -batch_size $batch_size \
          -epochs $epochs \
          $dense_layer_arg \
          -dense_size $dense_size \
    ```

## Contact

For any queries, please contact:

Aditya Makkar: aditya.makkar@uwaterloo.ca
    Divya Makkar: divya.makkar@uwaterloo.ca
    Aarav Patel: aarav.patel@uwaterloo.ca

## Citation

```
@misc{makkar2024signspeakopensourcetimeseries,
      title={SignSpeak: Open-Source Time Series Classification for ASL Translation}, 
      author={Aditya Makkar and Divya Makkar and Aarav Patel and Liam Hebert},
      year={2024},
      eprint={2407.12020},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.12020}, 
}
```

## Acknowledgement

We thank University of Waterloo PhD Liam Hebert for providing invaluable guidance and unwavering support throughout the course of SignSpeak.

SignSpeak

SignSpeak 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

SignSpeak

SignSpeak 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README