Textbooks come alive with new, interactive AI tool

With just an iPad, students in any classroom across the world could soon reimagine the ordinary diagrams in any physics textbook—transforming these static images into 3D simulations that run, leap or spin across the page.

These new, living textbooks are the brainchild of a team of computer scientists led by Ryo Suzuki at the University of Colorado Boulder. 

“Usually, those diagrams are fixed. We have to imagine what happens,” said Suzuki, assistant professor in the ATLAS Institute and Department of Computer Science. “But what if we could take any static diagram from any textbook and make it interactive?”

He and his colleagues recently took home a “best paper” award for their work at the 37th Annual ACM Symposium on User Interface Software and Technology this October in Pittsburgh. 

Picture this: You crack open your high school physics textbook to a drawing of a skier teetering at the top of a jump. The diagram is supposed to teach you about topics like kinetic energy, such as the energy of a skier zooming down a hill. But the skier never actually moves. He just sits there. 

Until now.

For the new study, Suzuki’s team designed a tool that allows students to record that diagram with their iPads. Then, after users adjust some settings, the tool draws on artificial intelligence to get the skier to move across the screen. He zooms down the hill, hits the jump and soars. It’s scientifically accurate, too: Students can adjust the friction of the snow, making their skier zoom faster or slower with a few swipes.

Suzuki sees the project as a chance to reinvent the textbook—giving learners the opportunity to not just read about physics but see topics like momentum and electrical currents in action.

“We believe the future of education and learning should be more interactive and personalized,” said Suzuki, who began the project as a researcher at the University of Calgary in Canada.

It’s alive!

Aditya Gunturu, first author of the study, remembers his own days learning about physics. He used to sit in science class in high school, thinking there must be a better way to visualize topics to convey motion.

“I was so frustrated,” said Gunturu, a master’s student in computer science at the University of Calgary. “I wanted to visualize these concepts.”
He’s not alone: Gunturu’s mother is a biology teacher in India and often draws on a blackboard to show students, for example, how cells divide—a phenomenon textbooks can only capture in snapshots.

The team's invention, Augmented Physics, could help. 

The tool relies on a model called Segment Anything from the tech company Meta. It’s a computer visualization tool that allows users to click on a photo to isolate particular objects—a dog, or maybe a face. Similarly, through Augmented Physics, students and teachers select various objects inside a diagram, such as the skier and the ski jump, and assign those objects roles. The AI then applies some basic physics, such as the force of gravity, to make those objects move.

It works for a several different kinds of diagrams, too. They include optics diagrams—so students can simulate how a prism refracts sunlight, for example—and electrical circuits—so they can see currents zip along wires. In a full circle moment, the researchers tested their tool out on some of those same diagrams in Gunturu’s physics textbook from years ago. 

Beyond physics

The team also designed its product with feedback from real students studying physics and with science teachers. One educator was impressed by the tool’s ability to help students lead their “own little experiments.”

The tool isn’t perfect yet, Suzuki noted. So far, it can only turn a brand new diagram into an effective simulation about 60% of the time. But the team is working to improve those success rates.

Ultimately, the researcher and his colleagues would like to move beyond physics—using similar technology to transform any diagram in any kind of document into a vivid, 3D world that students can step into with the aid of augmented reality glasses.

“That’s our goal—to escape from static 2D into the interactive 3D world,” Suzuki said.
Gunturu’s mom, meanwhile, can’t wait for him to design a tool that works on biology textbooks.

“That would be her dream come true,” he said.

Other co-authors of the new study included Rubaiat Habib Kazi at Adobe Research; Nandi Zhang and Jarin Thundathil at the University of Calgary; and Yi Wen at Texas A&M University.