Engineers bring sign language to ‘life’ using AI to translate in real-time

For millions of deaf and hard-of-hearing individuals around the world, communication barriers can make everyday interactions challenging. Traditional solutions, like sign language interpreters, are often scarce, expensive and dependent on human availability. In an increasingly digital world, the demand for smart, assistive technologies that offer real-time, accurate and accessible communication solutions is growing, aiming to bridge this critical gap.

American Sign Language (ASL) is one of the most widely used sign languages, consisting of distinct hand gestures that represent letters, words and phrases. Existing ASL recognition systems often struggle with real-time performance, accuracy and robustness across diverse environments.

A major challenge in ASL systems lies in distinguishing visually similar gestures such as “A” and “T” or “M” and “N,” which often leads to misclassifications. Additionally, the dataset quality presents significant obstacles, including poor image resolution, motion blur, inconsistent lighting, and variations in hand sizes, skin tones and backgrounds. These factors introduce bias and reduce the model’s ability to generalize across different users and environments.

To tackle these challenges, researchers from the College of Engineering and Computer Science at Florida Atlantic University have developed an innovative real-time ASL interpretation system. Combining the object detection power of YOLOv11 with MediaPipe’s precise hand tracking, the system can accurately recognize ASL alphabet letters in real time. Using advanced deep learning and key hand point tracking, it translates ASL gestures into text, enabling users to interactively spell names, locations and more with remarkable accuracy.

At its core, a built-in webcam serves as a contact-free sensor, capturing live visual data that is converted into digital frames for gesture analysis. MediaPipe identifies 21 keypoints on each hand to create a skeletal map, while YOLOv11 uses these points to detect and classify ASL letters with high precision.

“What makes this system especially notable is that the entire recognition pipeline – from capturing the gesture to classifying it – operates seamlessly in real time, regardless of varying lighting conditions or backgrounds,” said Bader Alsharif, the first author and a Ph.D. candidate in the FAU Department of Electrical Engineering and Computer Science. “And all of this is achieved using standard, off-the-shelf hardware. This underscores the system’s practical potential as a highly accessible and scalable assistive technology, making it a viable solution for real-world applications.” 

Results of the study, published in the journal Sensors, confirm the system’s effectiveness, which achieved a 98.2% accuracy (mean Average Precision, mAP@0.5) with minimal latency. This finding highlights the system’s ability to deliver high precision in real-time, making it an ideal solution for applications that require fast and reliable performance, such as live video processing and interactive technologies.

With 130,000 images, the ASL Alphabet Hand Gesture Dataset includes a wide variety of hand gestures captured under different conditions to help models generalize better. These conditions cover diverse lighting environments (bright, dim and shadowed), a range of backgrounds (both outdoor and indoor scenes), and various hand angles and orientations to ensure robustness.

Each image is carefully annotated with 21 keypoints, which highlight essential hand structures such as fingertips, knuckles and the wrist. These annotations provide a skeletal map of the hand, allowing models to distinguish between similar gestures with exceptional accuracy.

“This project is a great example of how cutting-edge AI can be applied to serve humanity,” said Imad Mahgoub, Ph.D., co-author and Tecore Professor in the FAU Department of Electrical Engineering and Computer Science. “By fusing deep learning with hand landmark detection, our team created a system that not only achieves high accuracy but also remains accessible and practical for everyday use. It’s a strong step toward inclusive communication technologies.”

The deaf population in the U.S. is approximately 11 million, or 3.6% of the population, and about 15% of American adults (37.5 million) experience hearing difficulties.

“The significance of this research lies in its potential to transform communication for the deaf community by providing an AI-driven tool that translates American Sign Language gestures into text, enabling smoother interactions across education, workplaces, health care and social settings,” said Mohammad Ilyas, Ph.D., co-author and a professor in the FAU Department of Electrical Engineering and Computer Science. “By developing a robust and accessible ASL interpretation system, our study contributes to the advancement of assistive technologies to break down barriers for the deaf and hard of hearing population.”

Future work will focus on expanding the system’s capabilities from recognizing individual ASL letters to interpreting full ASL sentences. This would enable more natural and fluid communication, allowing users to convey entire thoughts and phrases seamlessly.

“This research highlights the transformative power of AI-driven assistive technologies in empowering the deaf community,” said Stella Batalama, Ph.D., dean of the College of Engineering and Computer Science. “By bridging the communication gap through real-time ASL recognition, this system plays a key role in fostering a more inclusive society. It allows individuals with hearing impairments to interact more seamlessly with the world around them, whether they are introducing themselves, navigating their environment, or simply engaging in everyday conversations. This technology not only enhances accessibility but also supports greater social integration, helping create a more connected and empathetic community for everyone.”

Study co-authors are Easa Alalwany, Ph.D., a recent Ph.D. graduate of the FAU College of Engineering and Computer Science and an assistant professor at Taibah University in Saudi Arabia; Ali Ibrahim, Ph.D., a Ph.D. graduate of the FAU College of Engineering and Computer Science.

- FAU -

About FAU’s College of Engineering and Computer Science:

The FAU College of Engineering and Computer Science is internationally recognized for cutting-edge research and education in the areas of computer science and artificial intelligence (AI), computer engineering, electrical engineering, biomedical engineering, civil, environmental and geomatics engineering, mechanical engineering, and ocean engineering. Research conducted by the faculty and their teams expose students to technology innovations that push the current state-of-the art of the disciplines. The College research efforts are supported by the National Science Foundation (NSF), the National Institutes of Health (NIH), the Department of Defense (DOD), the Department of Transportation (DOT), the Department of Education (DOEd), the State of Florida, and industry. The FAU College of Engineering and Computer Science offers degrees with a modern twist that bear specializations in areas of national priority such as AI, cybersecurity, internet-of-things, transportation and supply chain management, and data science. New degree programs include Master of Science in AI (first in Florida), Master of Science and Bachelor in Data Science and Analytics, and the new Professional Master of Science and Ph.D. in computer science for working professionals. For more information about the College, please visit eng.fau.edu. 

About Florida Atlantic University:
Florida Atlantic University, established in 1961, officially opened its doors in 1964 as the fifth public university in Florida. Today, Florida Atlantic serves more than 30,000 undergraduate and graduate students across six campuses located along the Southeast Florida coast. In recent years, the University has doubled its research expenditures and outpaced its peers in student achievement rates. Through the coexistence of access and excellence, Florida Atlantic embodies an innovative model where traditional achievement gaps vanish. Florida Atlantic is designated as a Hispanic-serving institution, ranked as a top public university by U.S. News & World Report, and holds the designation of “R1: Very High Research Spending and Doctorate Production” by the Carnegie Classification of Institutions of Higher Education. Florida Atlantic shares this status with less than 5% of the nearly 4,000 universities in the United States. For more information, visit www.fau.edu.