Shakespeare in sign language, seen through AI

A new study uses co-creation with reference communities to develop an app for sign language machine translation (SLMT). The research team designed a theatrical performance in sign language, seen through the eyes of artificial intelligence (AI), as one of the methodologies. “Historically, deaf people have been excluded from the development of automatic translation technologies,” says Shaun O’Boyle, Research Fellow in the School of Inclusive and Special Education (Dublin City University DCU). “This has often caused backlash and resistance from deaf communities, as the projects were designed and developed without any input from the very end-users they intended to serve—resulting in a technology no one wanted to use and a big waste of money,” adds Davy Van Landuyt, Project Manager at the European Union of the Deaf (EUD). The team decided to reverse the standard approach. The results of their work are described in a new paper published in the Journal of Science Communication JCOM.

O’Boyle, Van Landuyt, and the other partners of the European project SignON—including the Vlaams GebarentaalCentrum (Flemish Sign Language Centre)—designed an innovative co-creation methodology based on this idea: If we were to introduce an AI to Shakespeare texts in Irish Sign Language, which extracts would we choose first? This engagement with the AI allowed them to connect with the audience and gather their opinions about the technology. “We looked hard at what deaf communities really wanted (and equally important didn’t want),” explains Van Landuyt.

“It was a group effort, and it was a lot of fun,” comments O’Boyle, an expert in science communication and community engagement, who co-authored the paper with Van Landuyt, Elizabeth Mathews (who works with O’Boyle at DCU) and others. The co-creation activity of the project, which contributed to the development of an SLMT app prototype, mixed live theatrical performance with artificial intelligence tools in a setting that reversed the stereotypical perspective between deaf/hard of hearing people and hearing people. “We combined performance with an audience discussion, so to have that element of a focus group technique but within the context of culture and art and seeing a show, and being out for the night,” explains O’Boyle. 

The event, titled All the World’s a Screen, was held during Dublin’s Science Week and consisted of a theatrical performance based on the works of William Shakespeare, performed in Irish Sign Language by two deaf actors, Lianne Quigley and Alvean Jones, who are also co-authors of the research and who translated Shakespeare's texts into Irish Sign Language. The audience was composed of deaf, hard of hearing, and hearing people. “For deaf people, it was really important to have a performance and science engagement in their own language rather than in English. In our performance, an interpreter provided access to hearing people,” explains O’Boyle. “It shifted the dynamic, but it also gave people an opportunity to imagine the future of these technologies because we asked people to view the performance with their AI ‘co-spectator’.” 

During the performance, in fact, spectators could access tools for real-time analysis of the performers' movements, object recognition, and real-time transcription of the spoken English translation by the interpreters. “We got so much interesting feedback and the great thing about doing it within the context of a research project was that all of that feedback went straight to our team. It informed the use cases for the app, it informed how we spoke about it, how we thought about it, and different aspects of the technology that was developed too,” says O’Boyle.

“The ethical side is also important,” adds Van Landuyt. 

“A major frustration of deaf communities are attitudes or motivations of some hearing researchers working on these technologies. Too often ‘hearing’ decision-makers or researchers think that they know better than deaf communities and try to create something which is then rejected by the actual end-users of those technologies, simply because it is not what they want or even utterly useless in practice. I think about, for example, technologies such as signing gloves,” says Van Landuyt, who also explains that this leads to a waste of time and resources. “Just including deaf people in a project is not a structural solution. It does not mean that they are leading the research. Deaf researchers are often only asked to collaborate after the initial idea has already been conceived, the team built, the research conducted, or even near the end of the project,” he adds.

By engaging in authentic co-creation and science communication with deaf communities, this can be mitigated. “We cannot stop the evolution of these technologies, so we need to figure out how to deal with them. It’s essential that deaf people can decide for themselves,” concludes Van Landuyt. “Nothing about us without us.”

The paper A deaf-centered art-science approach to community engagement with sign language technologies can be read for free in JCOM.

Link to Video: That is the question