image: TWIZ Team members including João Magalhães and CMU Portugal Ph.D. students Diogo Silva and Diogo Tavares
Credit: TWIZ team
The Portuguese team TWIZ from NOVA School of Science and Technology (FCT NOVA) secured 1st Place in the Alexa TaskBot Challenge 2 and won a $500,000 prize.
The winning team was led by João Magalhães, a researcher at NOVA LINCS, Professor at the Department of Informatics at FCT NOVA, and PI at CMU Portugal Farfetch Chat R&D project, and included CMU Portugal Affiliated Ph.D. students Diogo Tavares and Diogo Silva.
Under Farfetch Chat R&D, the FCT NOVA team supported the development of a multimodal conversational agent for the online fashion marketplace with the company Farfetch, a growing area of research where users and the conversational agent communicate by natural language and visual data. Farfetch Chat R&D project was led by Ricardo G. Sousa at Farfetch, and other PIs, besides João Magalhães, include João Paulo Costeira from Instituto Superior Técnico and Alexander Hauptmann from the Languages Technology Institute at CMU.
In this year’s edition, Alexa TaskBot Challenge 2, participants worked to address one of the most complex problems in conversational AI — creating next-generation conversational AI experiences that delight customers by addressing their changing needs as they complete complex tasks.
The challenge was also expanded to include more hobbies and at-home activities. Teams were asked to find exciting ways to incorporate visual aids into every conversation turn when a screen was available. Innovative ideas on improving the presentation of visual aids and the coordination of visual and verbal modalities were part of the judging criteria.
João Magalhães talks about this year’s experience at the competition and its challenges: “I’m extremely happy about the team’s creativity in designing the groundbreaking TWIZ LLM for Alexa. TWIZ LLMs make the Alexa TaskBot conversational experience fun, rich, and multimodal. It plays an important role in features like “What’s in your fridge?”, “fun facts”, and making task execution easier. TWIZ’s cherry on top of the cake is the video dialog feature. Conversations about video content take CX to an all-new level, and I’m very proud of pioneering video dialog in the Alexa Prize. I think there’s much to explore here .”
Diogo Silva, CMU Portugal Ph.D. affiliated student, adds that: “Participating in the Alexa Prize Taskbot Challenge was a very hectic experience, in a good way, of course. The team was highly motivated, which led to many exciting ideas that came with some tough decisions on which ones to pursue. Overall, I’d say that our biggest challenge was improving our visual interface, a crucial part of the interaction, as our team didn’t have anyone focused on HCI”.
This is the team’s second time participating in the Alexa TaskBot Challenge. Diogo Tavares, another CMU Portugal Affiliated Ph.D student, participated in both editions. He highlights the importance of the past experience: “The team and I are extremely proud of what we achieved. I think we built something that’s not only cutting-edge but also useful and fun to use! I think a lot of our success came from having last year as a learning experience. We gained a lot of insight into how users interact with chatbots in the real world. It was also the first time most of us had seen an Alexa! This year, we got to apply all our new knowledge to push the boundary“.
The opportunity to participate in the first Alexa TaskBot Challenge was possible thanks to the research already being developed on conversational assistants under the CMU Portugal large-scale collaborative project Farfetch Chat R&D. At the time, João Magalhães highlighted the relevance of CMU Portugal to Alexa TaskBot Challenge: “the Program played a key role as an enabler, both through the Farfetch Chat R&D project and CMU Portugal funded Ph.D. students. The Farfetch Chat R&D team at NOVA University already had a solid research program in multimodal conversational agents, so this was a natural step. This outstanding achievement was mainly due to several groundbreaking AI advances invented by our students: the robust dialog understanding Transformer algorithm and the language generation algorithms capable of talking about curious facts and algorithms to answer questions about the recipe and its video“.
Anyone can try the added functionality by trying “Alexa, let’s work together.”
Between this and last year’s prizes and awards for the students, the TWIZ team’s support from Amazon has totaled so far about $1.3 Million.
TWIZ work is now featured in the Alexa TaskBot Proceedings and featured at Amazon news article.