The idea that artificial intelligence will someday be able to understand and even generate narratives has inspired and motivated researchers for years. A question inextricably bound to both lines of research remains unresolved, however: Can AI recognize a good story if it sees one?
In the first large-scale attempt to tackle this challenging problem, scientists at Disney Research and the University of Massachusetts Boston have developed neural networks that can evaluate short narratives. These AIs don't try to mimic the professional judgment of literary critics, but attempt to predict which stories will appeal to a large population.
"Our neural networks had some success in predicting the popularity of stories," said Boyang "Albert" Li, a research scientist at Disney Research. "You can't yet use them to pick out winners for your local writing competition, but they can be used to guide future research."
The team will present its findings Aug. 23 at the International Joint Conference on Artificial Intelligence in Melbourne, Australia.
"The ability to predict narrative quality impacts on both story creation and story understanding," said Markus Gross, vice president at Disney Research. "To evaluate quality, the AI needs some level of understanding of the text. And if AIs are to create narratives, they need to be able to judge the quality of what they are producing."
One of the challenges in developing automated evaluation of story quality is the lack of large databases of stories that have been evaluated by humans and can be used to train AIs, noted Tong Wang, a computer science Ph.D. student at UMass Boston and a lab associate at Disney Research. The scientists found the question and answering website Quora, to be a good data source, as many of its answers are in the form of stories, he added. Reader upvotes, likewise, measure a story's popularity and serve as a proxy for narrative quality.
After gathering almost 55,000 answers, the researchers developed an algorithm to classify them as either stories or non-stories, yielding more than 28,000 stories with an average of 369 words.
To understand the complex semantics of the stories, the research team looked for ways to represent the influence of story structures in neural networks because a sequence of events can interact to reveal character intentions.
The researchers developed an AI that evaluated separate regions of each story -- including the question that prompted each. They created a network that looked at the regions interdependently and another network that took a holistic view, looking for how the meaning of the events and story regions emerged from the entire story. In each case, the AIs made predictions of which texts would prove most popular with readers.
In experiments, the neural nets showed improvement over a baseline text evaluation system, with the holistic neural network registering an 18 percent improvement.
In addition to Li and Wang, Ping Chen, Wang's PhD adviser and associate professor of computer science and engineering at UMass Boston, contributed to this research.
Combining creativity and innovation, this research continues Disney's rich legacy of leveraging technology to enhance the tools and systems of tomorrow.
###
For more information on this study, visit the project web site at https://www.disneyresearch.com/publication/quality-prediction-for-short-narratives/.
About Disney Research
Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice President Markus Gross manages Disney Research facilities in Los Angeles, Pittsburgh and Zurich, and works closely with the Pixar and ILM research groups in the San Francisco Bay Area. Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human-computer interaction, displays, behavioral economics, and machine learning.
Website: http://www.disneyresearch.com
Twitter: @DisneyResearch
Facebook: http://www.facebook.com/DisneyResearch
About UMass Boston
The University of Massachusetts Boston is deeply rooted in the city's history, yet poised to address the challenges of the future. Recognized for innovative research, metropolitan Boston's public university offers its diverse student population both an intimate learning environment and the rich experience of a great American city. UMass Boston's 11 colleges and graduate schools serve nearly 17,000 students while engaging local and global constituents through academic programs, research centers, and public service. To learn more, visit http://www.umb.edu.