Chat-GPT designs tomato picking robot in collaboration with researchers

With their ability to process vast amounts of text data, and to use this information to answer prompts, neural networks known as large language models (LLMs) like Chat-GPT have been making headlines for their potential to change the way we write, learn, and even make art. Now, EPFL researchers have applied the technology to a new sphere: robotic design.

In a case study published today in Nature Machine Intelligence, Josie Hughes, head of the Computational Robot Design & Fabrication Lab in the School of Engineering, EPFL PhD student Francesco Stella, and Cosimo Della Santina of TU Delft used Chat-GPT to design a working robotic tomato-harvester. The study provides a framework for humans and LLMs to design such devices collaboratively. Based on their experience, the researchers describe opportunities and risks of applying artificial intelligence (AI) tools to robotics, which they argue “could change the way we design robots, while enriching and simplifying the process.”

“Even though Chat-GPT is a language model and its code generation is text-based, it provided significant insights and intuition for physical design, and showed great potential as a sounding board to stimulate human creativity,” says Hughes.

Potential and pitfalls of AI as ‘inventor’

In a first phase, the researchers and LLM engaged in an ‘ideation’ discussion to define their robot’s purpose, design parameters, and specifications. A second phase was devoted to realizing the robot in the real world, which involved refining the LLM-generated code, fabricating the device, and troubleshooting its functioning.

For the first phase, the researchers started at a high conceptual level, conversing with the LLM on future challenges to humanity, and identifying robotic crop harvesting as a solution to the challenge of global food supply. They then drew upon the LLM’s access to global data from academic publications, technical manuals, books, and media to provide the “most probable” answer to prompts such as “what features should a robot harvester have?”

Once a basic robotic format was identified (a motor-driven gripper for grasping ripe tomatoes), the researchers could then pose more specific questions, like “what shape should the gripper have?”, and ask the LLM to make technical suggestions including materials and computer code for controlling the device.

“While computation has been largely used to assist engineers with technical implementation, for the first time, an AI system can ideate new systems, thus automating high-level cognitive tasks. This could involve a shift of human roles to more technical ones,” Stella says.

In addition to assigning Chat-GPT the role of ‘inventor’, the researchers outlined other possible human-LLM collaboration modes in their paper. For example, ‘collaborative exploration’ uses AI to augment researchers’ expertise by contributing wide-ranging knowledge beyond their own fields. AI can also act as a ‘funnel’, helping to refine the design process and providing technical input, with humans retaining creative control.

As there are logical and ethical risks associated with each collaboration mode, the researchers caution that the role of LLMs must be carefully evaluated going forward. For example, the use of LLMs raises questions of bias, plagiarism, and intellectual property, as it’s unclear if an LLM-generated design can be considered novel.

“In our study, Chat-GPT identified tomatoes as the crop ‘most worth’ pursuing for a robotic harvester. However, this may be biased towards crops that are more covered in literature, as opposed to those where there is truly a real need. When decisions are made outside the scope of knowledge of the engineer, this can lead to significant ethical, engineering, or factual errors.” Hughes says.

Despite these cautions, Hughes and her team conclude, based on their experience, that LLMs have great potential to be a force for good, if well managed: “The robotics community must therefore identify how to leverage these powerful tools to accelerate the advancement of robots in an ethical, sustainable, and socially empowering way.”