sign-language-translator

KO | EN

gitlite — search

#react #golang #android #rust #linux #docker #nodejs #java #awesome #cli #vue #pytorch #security #open-source

sign-language-translator

(★ 11)

Translate American Sign Language alphabet letters in real time using your webcam, MediaPipe, and a PyTorch neural network.

sign-language-translator Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🤟 Real-Time ASL Sign Language Translator

**Translate American Sign Language alphabet letters in real time using your webcam, MediaPipe, and a PyTorch neural network.**

<br/>

[![Python](https://img.shields.io/badge/Python-3.9--3.12-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/)
[![PyTorch](https://img.shields.io/badge/PyTorch-2.3.1-EE4C2C?style=for-the-badge&logo=pytorch&logoColor=white)](https://pytorch.org/)
[![MediaPipe](https://img.shields.io/badge/MediaPipe-0.10.21-0097A7?style=for-the-badge&logo=google&logoColor=white)](https://mediapipe.dev/)
[![OpenCV](https://img.shields.io/badge/OpenCV-4.10-5C3EE8?style=for-the-badge&logo=opencv&logoColor=white)](https://opencv.org/)
[![License](https://img.shields.io/badge/License-MIT-green?style=for-the-badge)](https://raw.githubusercontent.com/Eddiegah/sign-language-translator/main/LICENSE)
[![Stars](https://img.shields.io/github/stars/Eddiegah/sign-language-translator?style=for-the-badge&color=yellow)](https://github.com/Eddiegah/sign-language-translator/stargazers)
[![Accuracy](https://img.shields.io/badge/Val%20Accuracy-99.84%25-brightgreen?style=for-the-badge)](https://raw.githubusercontent.com/Eddiegah/sign-language-translator/main/results/training_log.txt)

<br/>

![ASL Demo](https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExcDd4dzdwMzVhNGo4YWlzeHlhc2JuNzFwdXBoMm4wcW5oMHFkeTVrayZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/l0MYt5jPR6QX5pnqM/giphy.gif)

> *Hold a sign for 1.5 seconds → it gets "typed" into a live text string on screen.*

</div>

---

## ✨ What It Does

| Feature | Description |
|---|---|
| 🖐 **Hand Detection** | MediaPipe Hands extracts 21 3D landmarks per frame |
| 🧠 **Neural Network** | 3-layer MLP trained to 99.84% validation accuracy |
| 🔤 **24 ASL Letters** | Covers A–X (J and Z require motion, excluded by design) |
| ⚡ **Real-Time** | Runs live on CPU — no GPU required |
| 📝 **Text Builder** | Hold a sign steady for 1.5s to "type" it into a string |
| 🎯 **Smoothing** | Flicker-free display — only updates after 7 consistent frames |

---

## 🖥️ Demo

```
┌─────────────────────────────────────────────┐
│                                             │
│   🖐  Detected: A  (97%)                    │
│                                             │
│   ████████████████░░░░  Hold to type...     │
│                                             │
│   Text: HELLO WOR_                          │
│                                             │
│   [SPACE] space  [BACKSPACE] delete         │
│   [C] clear      [ESC] quit                 │
└─────────────────────────────────────────────┘
```

> **Note:** A browser-based live demo requires a separate deployment (Hugging Face Spaces / Streamlit Cloud).
> To try it instantly, clone and run locally — setup takes under 5 minutes. ⬇️

</div>

---

## 🚀 Quick Start

### Prerequisites

- **Python 3.9–3.12** (Python 3.11 recommended — [download here](https://www.python.org/downloads/release/python-3119/))
- **Webcam** (built-in or USB)
- **Kaggle account** (free — only needed to download training data)
- **Windows / Linux / macOS**

> ⚠️ **Windows + OneDrive users:** If your project folder is inside `OneDrive\...`, move it to a plain path like `C:\Projects\sign-language-translator` first. OneDrive's file-on-demand feature breaks compiled Python packages.

---

### Step 1 — Clone the Repository

```bash
git clone https://github.com/Eddiegah/sign-language-translator.git
cd sign-language-translator
```

---

### Step 2 — Create a Virtual Environment

```bash
# Windows (Python 3.11 explicitly)
py -3.11 -m venv venv
venv\Scripts\activate

# macOS / Linux
python3.11 -m venv venv
source venv/bin/activate
```

---

### Step 3 — Install Dependencies

```bash
pip install -r requirements.txt
```

> This installs MediaPipe, PyTorch, OpenCV, scikit-learn, matplotlib and more (~500 MB).
> Grab a coffee — first install takes a few minutes. ☕

**Verify everything works:**
```bash
python -c "import mediapipe; import matplotlib; import cv2; import torch; print('✅ All imports OK')"
```

---

### Step 4 — Get Your Kaggle API Key

The training data comes from the public [ASL Alphabet dataset on Kaggle](https://www.kaggle.com/datasets/grassknoted/asl-alphabet) (87,000 hand images, free).

1. Sign up / log in at [kaggle.com](https://www.kaggle.com)
2. Go to **Account → Settings → API → Create New Token**
3. A `kaggle.json` file will download
4. Place it here:
   - **Windows:** `C:\Users\<YourName>\.kaggle\kaggle.json`
   - **Linux/macOS:** `~/.kaggle/kaggle.json`

---

### Step 5 — Build the Dataset

```bash
python src/build_dataset.py
```

This will:
- 📥 Download ~1 GB of hand images from Kaggle
- 🔍 Run MediaPipe over 300 images per letter (7,200 total)
- 💾 Save normalized landmarks to `data/landmarks.csv`
- 🗑️ Delete the raw images to save disk space

> ⏱️ Takes **15–40 minutes** depending on your internet speed and CPU.

---

### Step 6 — Train the Model

```bash
python src/train.py
```

```
Loaded 6326 samples across 24 classes.
Using device: cpu

✅ Training complete in 96s. Best val accuracy: 99.84%
```

Outputs saved to:
- `models/classifier.pt` — trained model checkpoint
- `results/confusion_matrix.png` — per-class accuracy heatmap
- `results/training_log.txt` — full epoch log

---

### Step 7 — Run Live Inference 🎉

```bash
python src/infer_live.py
```

A webcam window opens. You're live!

| Key | Action |
|---|---|
| Hold a sign for **1.5s** | Types the letter into the text string |
| `SPACE` | Insert a space |
| `BACKSPACE` | Delete last character |
| `C` | Clear the entire string |
| `ESC` | Quit |

---

## 📁 Project Structure

```
sign-language-translator/
│
├── 📂 src/
│   ├── build_dataset.py     # Download Kaggle images → extract landmarks
│   ├── collect_data.py      # Optional: collect your own webcam samples
│   ├── preprocessing.py     # Landmark normalization (shared by all scripts)
│   ├── model.py             # MLP architecture definition
│   ├── train.py             # Training loop + confusion matrix
│   └── infer_live.py        # Real-time webcam inference + text builder
│
├── 📂 data/
│   └── landmarks.csv        # Generated dataset (gitignored)
│
├── 📂 models/
│   └── classifier.pt        # Trained checkpoint (gitignored)
│
├── 📂 results/
│   ├── confusion_matrix.png # Validation accuracy per letter
│   └── training_log.txt     # Per-epoch training log
│
├── requirements.txt
├── .gitignore
└── README.md
```

---

## 🧠 How It Works

```
Webcam frame
    │
    ▼
MediaPipe Hands
    │  21 landmarks × (x, y, z)
    ▼
Normalization (preprocessing.py)
    │  Translate to wrist origin → scale by max distance
    │  Output: 63 floats, position & scale invariant
    ▼
MLP Classifier (model.py)
    │  63 → 128 → 64 → 24
    │  ReLU + Dropout(0.3) per hidden layer
    ▼
Softmax → predicted letter + confidence
    │
    ▼
Smoothing buffer (7 frames)
    │  Only update display when last 7 frames agree
    ▼
Text builder
    │  Hold same letter for 1.5s → append to string
    ▼
OpenCV overlay on live frame
```

---

## 📊 Results

| Metric | Value |
|---|---|
| Training samples | 6,326 |
| Classes | 24 (ASL A–X) |
| Best validation accuracy | **99.84%** |
| Training time | ~96 seconds (CPU) |
| Inference speed | Real-time (~30 FPS) |

![Confusion Matrix](https://raw.githubusercontent.com/Eddiegah/sign-language-translator/main/results/confusion_matrix.png)

</div>

---

## 🔡 Supported ASL Letters

| ✅ Supported | ❌ Excluded |
|---|---|
| A B C D E F G H I K L M N O P Q R S T U V W X Y | J, Z (require motion) |

</div>

---

## 🛠️ Collect Your Own Data (Optional)

If you want to improve accuracy with your own hand samples, run:

```bash
python src/collect_data.py
```

- Press a **letter key** to arm the collector
- Show the hand shape to the webcam
- Press **SPACE** to capture a sample (green flash = captured)
- Aim for 50–100 samples per letter
- Press **ESC** when done

Your samples are appended to `data/landmarks.csv` and will be used next time you run `train.py`.

---

## 🧯 Troubleshooting

<details>
<summary><b>❌ DLL load failed while importing _c_internal_utils</b></summary>

This is a missing Visual C++ Redistributable on Windows.

**Fix:**
1. Install both:
   - [VC++ Redistributable x64](https://aka.ms/vs/17/release/vc_redist.x64.exe)
   - [VC++ Redistributable x86](https://aka.ms/vs/17/release/vc_redist.x86.exe)
2. **Restart your machine** (not just the terminal)
3. Re-run the import check

</details>

<details>
<summary><b>❌ MediaPipe won't install on Python 3.13+</b></summary>

MediaPipe does not yet support Python 3.13. Use Python 3.11:

```bash
py -3.11 -m venv venv
```

</details>

<details>
<summary><b>❌ Webcam not opening</b></summary>

- Close any app using the camera (Teams, Zoom, OBS)
- Try changing `cv2.VideoCapture(0)` to `cv2.VideoCapture(1)` in `infer_live.py`

</details>

<details>
<summary><b>❌ Low prediction accuracy</b></summary>

The model was trained on studio photos. If accuracy is low on your hand:
1. Run `collect_data.py` and capture 30–50 samples of your own hand per letter
2. Re-run `train.py` — it will train on the combined data

</details>

<details>
<summary><b>❌ Project is inside OneDrive</b></summary>

Move the project to a non-OneDrive path (e.g. `C:\Projects\sign-language-translator`), delete the venv, and recreate it.

</details>

---

## 🤝 Contributing

Contributions are welcome! Feel free to open an issue or submit a pull request for:
- New features (word-level recognition, two-hand signs)
- Accuracy improvements
- Cross-platform fixes

---

## 📄 License

This project is licensed under the **MIT License** — use it freely, credit appreciated.

---

## 🙏 Acknowledgements

- [MediaPipe](https://mediapipe.dev/) by Google — hand landmark detection
- [ASL Alphabet Dataset](https://www.kaggle.com/datasets/grassknoted/asl-alphabet) by grassknoted on Kaggle
- [PyTorch](https://pytorch.org/) — model training framework

---

Made with ❤️ by [Eddie](https://github.com/Eddiegah)

⭐ **If this project helped you, give it a star!** ⭐

</div>

sign-language-translator

sign-language-translator Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

sign-language-translator

sign-language-translator Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README