NSF grant to make coding more accessible for persons with physical disabilities

A collaborative team of researchers across the University of Southern California (USC), led by Maja Matarić, the Chan Soon-Shiong Chair and Distinguished Professor of Computer Science, Neuroscience and Pediatrics, has been awarded a $698,161 National Science Foundation (NSF) Convergence Accelerator grant to develop new technologies for persons with disabilities.

The team, which includes experts in computer science, education, kinesiology and occupational therapy, aims to develop personalized prototype interfaces, enhanced by artificial intelligence, to help persons with disabilities learn and practice programming skills.

“Our goal as a team is to break down barriers and make coding accessible to everyone, including individuals with diverse abilities,” said Matarić. “Here we focus on connecting deeply with the user community and doing a great deal of co-design to develop prototypes, all within less than a year. It’s a wonderful and highly demanding opportunity, and a prime example of USC’s ability to unite multidisciplinary teams focused on addressing societal needs.”

USC is one of 16 multidisciplinary Phase 1 teams selected by the NSF for the 2022 cohort, Track H: Enhancing Opportunities for Persons with Disabilities. The co-principal investigators are Gisele Ragusa, a professor of engineering education; Stephen Aguilar, an assistant professor of education; Sook-Lei Liew, an associate professor of occupational science and occupational therapy; and Jesse Thomason, an assistant professor of computer science.

They are joined by USC senior personnel Heather Culbertson, an assistant professor of computer science, Mark Redekopp, a professor of electrical and computer engineering and computer science practice, and partners from the University of Colorado Boulder, University of Washington, Microsoft, Regional Center Agencies, Rancho Los Amigos Rehabilitation Center, New York University, Tufts University, Northwestern University, Children’s Hospital Los Angeles and Los Angeles-area community colleges.

“This research effort has expansive meaning for me,” said Ragusa. “As a longstanding practitioner in, and researcher of, access and inclusion for individuals with diverse abilities, I believe that the work we will engage in with community partners has tremendous potential to transform the lives and employment opportunities of people with disabilities.”

ADAPTING TO PHYSICAL ABILITIES

One billion people, or 15% of the world’s population, have experienced a disability. Yet disabilities present major barriers to entering the workforce, note the researchers in their proposal. In fact, approximately 80% of persons with disabilities were excluded from the 2021 workforce.

Meanwhile, programming jobs continue to grow explosively but are largely inaccessible to persons with disabilities. For example, an individual with a disability may find it difficult to use a keyboard and mouse to code but may be able to use speech recognition to dictate code, or a touch screen to select commands.

“We still ask persons with physical disabilities to use these antiquated technologies that cannot adapt to their physical abilities when they learn how to code. Our grant addresses this issue head-on,” said Aguilar, an expert in educational technologies and educational psychology.

“In partnership with persons with physical disabilities, we will design a platform that adapts to the physical abilities of individuals with physical disabilities. This work will enable people with physical disabilities to learn how to code using off-the-shelf, currently available programming courses by increasing the type of peripherals they can use to learn.”

As part of Phase 1, the team will gather data-driven insights for technology designers, develop machine learning methods to customize interfaces, and use large-scale language models to help with programming. Together, they will develop and evaluate prototypes of interfaces—operated using speech, eye tracking or pedals, for instance—that enable persons with disabilities to learn, practice, and use programming skills.

Thomason, an expert in natural language processing and robotics, will examine ways to reduce the number of keystrokes needed for efficient programming without keyboards and mice, using input from speech recognition, gaze tracking, gesture recognition and other inputs like pressure peddles.

“Programming involves transcribing lots of characters like ‘open brace’ that can be tedious with a speech recognition system,” said Thomason. “Additional modalities like gaze tracking, gesture recognition, and pressure peddles could join with speech acts, such as ‘brace’ together with raising the left shoulder for ‘open brace.'”

The project will also create a pathway for persons with disabilities to train and enter the programming workforce, profoundly impacting available work opportunities.

“As an occupational therapist, I am eager to build new tools with and for people with disabilities, so that they can interface with computers more comfortably to ultimately achieve their employment goals,” said Liew.

“I look forward to drawing on the past decade of my neuroscience research building brain–computer interfaces and virtual reality tools that enable people with stroke to better engage in meaningful activities. I’m thrilled to work with such a fantastic interdisciplinary team focusing on the everyday occupation so many people with disabilities say they want most — jobs.”

The Convergence Accelerator program and awarded grant are part of NSF’s new Directorate for Technology, Innovation and Partnerships. After the first year, the team will participate in a formal NSF pitch and Phase 2 proposal. Selected teams will then advance to Phase 2, which provides additional funding of up to $5 million.