KU research team develops virtual reality, AI-boosted system to help students with autism improve social skills

LAWRENCE — For more than a decade, University of Kansas researchers have been developing a virtual reality system to help students with disabilities, especially those with autism spectrum disorder, to learn, practice and improve social skills they need in a typical school day. Now, the KU research team has secured funding to add artificial intelligence components to the system to give those students an extended reality, or XR, experience to sharpen social interactions in a more natural setting.

The U.S. Office of Special Education Programs has awarded a five-year, $2.5 million grant to researchers within KU’s School of Education & Human Sciences to develop Increasing Knowledge and Natural Opportunities With Social Emotional Competence, or iKNOW. The system will build on previous work and provide students and teachers with an immersive, authentic experience blending extended reality and real-world elements of artificial intelligence.

iKNOW will expand the capabilities of VOISS, Virtual reality Opportunity to Integrate Social Skills, a KU-developed VR system that has proven successful and statistically valid in helping students with disabilities improve social skills. That system contains 140 unique learning scenarios meant to teach knowledge and understanding of 183 social skills in virtual school environments such as a classroom, hallway, cafeteria or bus that students and teachers can use via multiple platforms such as iPad, Chromebooks or Oculus VR headsets. The system also helps students use social skills such as receptive or expressive communication across multiple environments, not simply in the isolation of a classroom.

IKNOW will combine the VR aspects of VOISS with AI features such as large language models to enhance the systems’ capabilities and allow more natural interactions than listening to prerecorded narratives and responding by pushing buttons. The new system will allow user-initiated speaking responses that can accurately transcribe spoken language in real-time. AI technology of iKNOW will also be able to generate appropriate video responses to avatars students interact with, audio analysis of user responses, integration of in-time images and graphics with instruction to boost students’ contextual understanding.

“Avatars in iKNOW can have certain reactions and behaviors based on what we want them to do. They can model the practices we want students to see,” said Amber Rowland, assistant research professor in the Center for Research on Learning, part of KU’s Life Span Institute and one of the grant’s co principal investigators. “The system will harness AI to make sure students have more natural interactions and put them in the role of the ‘human in the loop’ by allowing them to speak, and it will respond like a normal conversation.”

The spoken responses will not only be more natural and relatable to everyday situations, but the contextual understanding cues will help students better know why a certain response is preferred. Rowland said when students were presented with multiple choices in previous versions, they often would know which answer was correct but indicated that’s not how they would have responded in real life.

IKNOW will also provide a real-time student progress monitoring system, telling them, educators and families how long students spoke, how frequently they spoke, number of keywords used, where students may have struggled in the system and other data to help enhance understanding.

All avatar voices that iKNOW users encounter are provided by real middle school students, educators and administrators. This helps enhance the natural environment of the system without the shortcomings of students practicing social skills with classmates in supervised sessions. For example, users do not have to worry what the people they are practicing with are thinking about them while they are learning. They can practice the social skills that they need until they are comfortable moving from the XR environment to real life.

“It will leverage our ability to take something off of teachers’ plates and provide tools for students to learn these skills in multiple environments. Right now, the closest we can come to that is training peers. But that puts students with disabilities in a different box by saying, ‘You don’t know how to do this,’” said Maggie Mosher, assistant research professor in KU’s Achievement & Assessment Institute, a co-principal investigator for the grant.

Mosher, a KU graduate who completed her doctoral dissertation comparing VOISS to other social skills interventions, found the system was statistically significant and valid in improving social skills and knowledge across multiple domains. Her study, which also found the system to be acceptable, appropriate and feasible, was published in high-impact journals Computers & Education and Issues and Trends in Learning Technologies.

The grant supporting iKNOW is one of four OSEP Innovation and Development grants intended to spur innovation in educational technology. The research team, including principal investigator Sean Smith, professor of special education; Amber Rowland, associate research professor in the Center for Research on Learning and the Achievement & Assessment Institute; Maggie Mosher, assistant research professor in AAI; and Bruce Frey, professor in educational psychology, will present their work on the project at the annual I/ITSEC conference, the world’s largest modeling, simulation and training event. It is sponsored by the National Training & Simulation Association, which promotes international and interdisciplinary cooperation within the fields of modeling and simulation, training, education and analysis and is affiliated with the National Defense Industrial Association.

The research team has implemented VOISS, available on the Apple Store and Google Play, at schools across the country. Anyone interested in learning more can find information, demonstrations and videos at the iKNOW site and can contact developers to use the system at the site’s “work with us” page.

IKNOW will add resources for teachers and families who want to implement the system at a website called iKNOW TOOLS (Teaching Occasions and Opportunities for Learning Supports) to support generalization of social skills across real-world settings.

“By combining our research-based social emotional virtual reality work (VOISS) with the increasing power and flexibility of AI, iKNOW will further personalize the learning experience for individuals with disabilities along with the struggling classmates,” Smith said. “Our hope and expectation is that iKNOW will further engage students to develop the essential social emotional skills to then apply in the real world to improve their overall learning outcomes.”