Sign-Language-Recognition--MediaPipe-DTW

KO | EN

gitlite — search

#javascript #python #react #android #machine-learning #rust #deep-learning #docker #nodejs #macos #windows #awesome #cli #ios

Sign-Language-Recognition--MediaPipe-DTW

(★ 96)

No description available.

Sign-Language-Recognition--MediaPipe-DTW Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Sign Language Recognition - using MediaPipe and DTW

![License: MIT](https://img.shields.io/badge/license-MIT-green)

This repository proposes an implementation of a Sign Recognition Model using the **MediaPipe** library 
for landmark extraction and **Dynamic Time Warping** (DTW) as a similarity metric between signs.

![](https://raw.githubusercontent.com/gabguerin/Sign-Language-Recognition--MediaPipe-DTW/master/example.gif)

#### Source : https://www.sicara.ai/blog/sign-language-recognition-using-mediapipe
___

## Set up

### 1. Open terminal and go to the Project directory

### 2. Install the necessary libraries

- ` pip install -r requirements.txt `

To automatically create a small dataset of French signs:

- Install `ffmpeg` (for MacOS `brew install ffmpeg`)
- Run: ` python yt_download.py `
- Add more YouTube links in ``yt_links.csv`` if needed
> N.B. The current dataset is insufficient to obtain good results. Feel free to add more links or import your own videos

### 4. Load the dataset and turn on the Webcam

- ` python main.py `

### 5. Press the "r" key to record the sign.

___
## Code Description

### *Landmark extraction (MediaPipe)*

- The **Holistic Model** of MediaPipe allows us to extract the keypoints of the Hands, Pose and Face models.
For now, the implementation only uses the Hand model to predict the sign.

### *Hand Model*

- In this project a **HandModel** has been created to define the Hand gesture at each frame. 
If a hand is not present we set all the positions to zero.

- In order to be **invariant to orientation and scale**, the **feature vector** of the
HandModel is a **list of the angles** between all the connexions of the hand.

### *Sign Model*

- The **SignModel** is created from a list of landmarks (extracted from a video)

- For each frame, we **store** the **feature vectors** of each hand.

### *Sign Recorder*

- The **SignRecorder** class **stores** the HandModels of left hand and right hand for each frame **when recording**.
- Once the recording is finished, it **computes the DTW** of the recorded sign and 
all the reference signs present in the dataset.
- Finally, a voting logic is added to output a result only if the prediction **confidence** is **higher than a threshold**.

### *Dynamic Time Warping*

-  DTW is widely used for computing time series similarity.

- In this project, we compute the DTW of the variation of hand connexion angles over time.

___

## References

- [Pham Chinh Huu, Le Quoc Khanh, Le Thanh Ha : Human Action Recognition Using Dynamic Time Warping and Voting Algorithm](https://www.researchgate.net/publication/290440452)
 - [Mediapipe : Pose classification](https://google.github.io/mediapipe/solutions/pose_classification.html)

Sign-Language-Recognition--MediaPipe-DTW

Sign-Language-Recognition--MediaPipe-DTW Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Sign-Language-Recognition--MediaPipe-DTW

Sign-Language-Recognition--MediaPipe-DTW Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README