ISL Accessibility and Translation System
Indian Sign Language translation and gamified learning platform using computer vision and machine learning
Project Overview
This two-module assistive system was developed to help bridge real-time communication gaps for the deaf and hard-of-hearing community through Indian Sign Language recognition, translation, and learning tools.
The project combines real-time sign alphabet and number recognition with a separate gamified learning experience. It reached the National Semi-Finals of Mastek DeepBlue Season 11 among engineering teams across India.
Live Translation Interface
Real-time sign detection with MediaPipe hand landmarks, detected sign output panel, and confidence visualization.
Bidirectional Translation Engine
Speech-to-Sign converts spoken English into ISL animations. Text-to-Sign converts typed text into gesture sequences.
Gamified Learning Platform
Interactive web-based learning with practice exercises, quizzes, achievement badges, and live leaderboards.
System Architecture Pipeline
The recognition pipeline flows from camera input through MediaPipe landmark detection, feature extraction, Scikit-learn ML classifier, to the final recognized sign and translation output.
Core Capabilities
- Real-time sign alphabet and number recognition using MediaPipe hand-landmark tracking and Scikit-learn classifiers.
- Speech-to-Sign conversion from spoken English into sign animations.
- Text-to-Sign conversion from typed text into ISL gestures.
- Interactive practice exercises, quizzes, and live score leaderboards.
Technology Stack
Python, MediaPipe, Scikit-learn, OpenCV, Streamlit, PyTorch, React, Node.js, Express.js, and Render.
Live Resources
- ISL Mastek GitHub repository
- Live Streamlit translation app
- Gamified learning web app
- Project demo video
Engineering Approach
Module 1 — Real-Time Sign Recognition
The recognition module uses MediaPipe Hands to extract 21 three-dimensional hand landmark coordinates from a live webcam feed at 30 fps. These 63 floating-point values are normalized relative to the wrist landmark, making the classifier invariant to hand scale and position. A Scikit-learn Random Forest classifier trained on a custom dataset of ISL alphabets (A–Z) and numbers (0–9) was then applied to predict the sign in real time. OpenCV handles the frame capture, rendering, and overlay of the landmark skeleton and prediction label on the live video stream.
Module 2 — Bidirectional Translation
The Speech-to-Sign module uses Python's speech_recognition library to capture spoken English and convert it to text. Each word is then matched to an ISL animation sequence stored in a pre-built dictionary. The Text-to-Sign module accepts typed input and performs the same word-to-animation mapping. Both modules are served through a Streamlit web interface that requires no installation — users access it in a browser at isl-mastek.streamlit.app.
Gamified Learning Platform
The learning platform was built with React and Node.js, deployed on Render. It features lesson cards for each ISL alphabet and number sign, live sign-practice exercises where the webcam verifies the user's gesture, timed quizzes with instant feedback, an XP scoring system, achievement badges, and a live leaderboard. The gamification elements were designed to make ISL accessible to children and first-time learners without formal training.
Impact & Recognition
The project was submitted to Mastek DeepBlue Season 11, a national-level technology innovation competition for engineering students across India. Out of hundreds of submissions, the ISL Accessibility and Translation System advanced to the National Semi-Finals, recognised for its social impact, technical depth, and accessibility-first design philosophy.
The system addresses a real communication barrier: approximately 63 million people in India have significant hearing loss, yet most software tools and public interfaces do not support Indian Sign Language. This project demonstrates how computer vision and machine learning can be applied to build inclusive technology with meaningful societal impact.
What I Learned
This project deepened my understanding of computer vision pipelines, real-time ML inference, and accessible web design. Working within the constraints of a hackathon timeline taught me to prioritise ruthlessly, make architectural decisions quickly, and communicate complex technical ideas to a non-technical judging panel. It also reinforced my commitment to building software that solves problems for underrepresented communities.