Artificial intelligence brings a virtual fly to life

By infusing a virtual fruit fly with artificial intelligence, Janelia and Google DeepMind scientists have created a computerized insect that can walk and fly just like the real thing.

The new virtual fly is the most realistic simulation of a fruit fly created to date. It combines a new anatomically accurate model of the fly’s outer skeleton, a fast physics simulator, and an artificial neural network trained on fly behaviors to mimic the actions of a real fly.

Along with walking and flying across complex trajectories, the virtual fly can use its eyes to control and steer its flight.  

“You take real fly data – how real flies fly and how real flies walk – and train the network to mimic these motions, and then we let this network that we have trained control the fly and tell the fly how to move,” says Roman Vaxenburg, a machine learning researcher in the Turaga Lab at Janelia who led the project. “It is like a little brain that controls the fly’s movements.”

The new model is the first iteration of the team’s virtual fly, which they plan to make even more lifelike with additional anatomical and sensory features and a real neural network. It is also the first in what they hope will be a series of realistic animal models that they and other researchers can now develop using this general-purpose, open-source framework.

These models could help scientists more fully understand how the nervous system, body, and environment work together to control behavior. While researchers have probed these questions with real animals in the lab for decades, realistic virtual models will enable scientists to understand how all these components interface and how factors that can’t be measured in the lab – like the force exerted on the body when flying – affect behavior.

“The simulation of the body can tell you how commands from the nervous system get translated into actions and behavior and that ‘how’ has to do with the shape of the body and the physics of how the body interacts with the world,” says Janelia Group Leader Srinivas Turaga, a senior scientist on the project. “All of that is codified in this physics simulation.”

Building a fly

The new model builds on previous work to simulate fruit fly behavior, including the Grand Unified Fly that uses a simplified fly body and a manual control system to simulate flight. The more recent NeuroMechFly uses a realistic body model and a manual control system with learned components to simulate walking.

In the new work, Janelia researchers and DeepMind scientists, led by Senior Research Scientist Yuval Tassa and Josh Merel (now at Fauna Robotics), set out to improve the fly model’s anatomy, biomechanics, physics, and behavior to create a more realistic fly simulation that could carry out multiple behaviors. The effort is one of several collaborations between Janelia and DeepMind using their combined expertise in neuroscience and artificial intelligence to address scientific questions.  

“Research in computational neuroscience had rarely sought to simulate things at this global level, despite widespread acknowledgement that understanding brain function depends on understanding the body and its interactions with other physical objects,” says Matthew Botvinick, senior director of research at DeepMind, which leverages AI to help advance science by partnering with experts in the research community. “In brainstorming sessions with Srini and the rest of this team, we realized there was an exciting opportunity to bring all of the pieces together in the context of fruit fly research.”

To start, Janelia Research Specialist Igor Siwanowicz imaged various parts of an adult female fruit fly with a microscope and used computer software to construct an anatomically accurate virtual model of the outside body of the fly that includes the movements of the fly’s joints and appendages.

DeepMind researchers, including Tassa, Merel, and Research Engineer Guido Novati, translated this virtual model into code that they fed into the MuJoCo simulator, a fast, open-source physics simulator created for robotics and biomechanics. The tool enables researchers to virtually simulate how something might move and interact in the real world.

To support the fruit fly model, the researchers made significant upgrades to the simulator, including adhesion actuators that model the forces generated by insect feet gripping a surface. The team was also lucky to have Novati design a new fluid-force model, which describes the forces acting on the fly as it moves through the air. The model can support all sorts of aerodynamic behaviors, including winged flight, says Tassa, a senior author on the project.  

Because the forces acting on the fly are so miniscule, “modeling such a small insect was very challenging,” Tassa says.       

Next, Vaxenburg built an artificial neural network and trained it on real fly behavior by feeding it information taken from videos recorded by fly behavior experts, including Janelia Senior Group Leaders Kristin Branson and Michael Reiser, HHMI Investigator Gwyneth Card and Caltech professor Michael Dickinson.

The fast speed of the MuJoCo simulator enabled the team to infuse the fly model with Vaxenburg’s neural network, which had learned how to move the fly in a realistic way to carry out an action. When the model is given a high-level command, like to walk forward at a certain pace, the neural network translates this command into low-level movements that an actual fly would use to carry out the action.

“The goal was to upgrade the level of realism and that’s what we did on two fronts: one was improving the capturing of the anatomical details – how the fruit fly is built – and the other is capturing the behavior – how it moves and reacts,” Vaxenburg says.

A first step

The new model is just the beginning. Next, the team hopes to create an even more realistic fly model by incorporating other parts of the fly’s anatomy, like its muscles and tendons, and a realistic sensory system into their virtual insect. They also want to be able to use a real neural network, like the fruit fly ventral nerve cord connectome, to power the model.

Now that the team has shown that they can create these types of realistic virtual models, they also want to create a virtual mouse and zebrafish, two organisms widely studied by neuroscientists. The pipeline they used to create the virtual fly is also freely accessible to researchers worldwide, enabling others to also create their own realistic models.

“We’ve shown how to do it, and we can do it again for another organism,” Turaga says.