A simple way to boost math progress

American students have been falling behind in math for decades—with test scores that consistently rank in the bottom 25% globally compared to students in other developed countries—and the COVID-19 pandemic made the situation worse.

Previous research has shown that interventions grounded in behavioral science that target student motivation have been effective at increasing math scores, suggesting that taking a similar “behaviorally informed” approach with teachers might have a comparable effect.

Now, a collaborative study published in the Proceedings of the National Academy of Sciences, and led by researchers at the Behavior Change for Good Initiative (BCFG) at the University of Pennsylvania has found that behaviorally informed email messages slightly improved students’ math progress compared to control messages.

“Our results showed that simple, low-cost nudges can help teachers support student progress in math,” says Angela Duckworth, Rosa Lee and Egbert Chang Professor in Penn’s School of Arts & Sciences and the Wharton School, who led the study and co-directs BCFG. “These nudges worked across different school contexts, with effects persisting eight weeks after teachers stopped receiving the nudges.”

The key to this megastudy was the partnership with Zearn Math, a nonprofit educational platform. “Large-scale studies on teacher-focused interventions have been rare due to the high cost and logistical challenges involved. Thanks to our partnership with Zearn Math, we were able to overcome these challenges,” says co-author Dena Gromet, executive director of BCFG.

A megastudy is “a large-scale experiment in which multiple interventions are tested simultaneously on the same outcome, a tournament approach, if you will,” says co-author Katy Milkman, James G. Dinan Endowed Professor and professor of operations, information and decisions at the Wharton School and co-director of BCFG. “Because all interventions run concurrently and are compared to a common control group, this method allows for direct comparisons of their effectiveness.”

In one of the largest studies of its kind—involving more than 140,000 teachers and nearly 3 million elementary students—the researchers compared the impact of 15 different interventions to a reminder-only message.

“These messages were behaviorally informed, meaning they were based on prior insights from behavioral science. For instance, one intervention asked teachers to make a specific plan for how they would use Zearn Math that week, an approach backed by research showing that people are more likely to follow through when they make detailed plans. Another intervention appealed to teachers’ empathy for their students, which previous research has demonstrated is supportive of student success,” Duckworth says.

View large image

Co-authors Katy Milkman (left) and Angela Duckworth are committed to dig deeper into what makes these kinds of interventions work and how to make them even more effective over time.

Specifically, the research team found that, compared to standard email reminders, behaviorally informed email messages improved students’ math progress during the four-week intervention period by 1.89%. The most effective intervention, which increased student math progress by about 5.06%, encouraged teachers to log into Zearn Math weekly for an updated, personalized report on their students’ progress.

“One especially promising takeaway is that personalized nudges—those that referenced progress updates about a teacher’s own students—were more effective than nonpersonalized ones,” Duckworth says.

The researchers note that though they are promising, the effects were small. “These results suggest the need for more intensive support than the light-touch email nudges we tested,” Milkman says. “And they underscore how hard it is to change human behavior.”

These findings, Milkman says, suggest several additional valuable avenues for future research, including “more random-assignment field experiments to confirm the causal benefits of teacher-targeted nudges and studies to probe the longer-term effects of behaviorally-informed interventions.”

Additional research is also needed, Duckworth says, “to confirm and explain the benefits of referencing personalized data when nudging teachers. It may be that capitalizing on teachers’ intrinsic motivation to help their students is a distinct and potentially cost-effective approach that can complement other interventions, such as offering performance bonuses and other extrinsic incentives.”

Next steps for researchers are to dig deeper into what makes these kinds of interventions work and how to make them even more effective over time. Future studies are needed to look into the long-term effects of nudges and explore why some interventions are more effective than others.

“The better we understand why something works, the more powerfully we can use it to create positive change,” Duckworth says. “Ultimately, this line of research could help shape smarter, more effective education policies.”

Angela L. Duckworth is the Rosa Lee and Egbert Chang Professor in the Department of Psychology in the School of Arts & Sciences and in the Department of Operations, Information, and Decisions in the Wharton School at the University of Pennsylvania and faculty co-director of the Penn-Wharton Behavior Change for Good Initiative.

Katherine L. Milkman is the James G. Dinan Endowed Professor in the Department of Operations, Information, and Decisions in the Wharton School of the University of Pennsylvania and faculty co-director of the Penn-Wharton Behavior Change for Good Initiative.

Dena M. Gromet is the Executive Director of the Behavior Change for Good Initiative at the University of Pennsylvania.

Other authors of the new study are Ron Berman, Eugen Dimant, Ahra Ko, Joseph S. Kay, Youngwoo Jung, Madeline K. Paxson, Ramon A. Silvera Zumaran, and Christophe Van den Bulte of the University of Pennsylvania; Aden Halpern of the University of Pennsylvania and the University of Pittsburgh; Nina Mazar of Boston University; Colin F. Camerer and Marcos N. Gallo of the California Institute of Technology; Amy Lyon of Colby-Sawyer College; Mary C. Murphy of Indiana University; Kathryn M. Kroeper of Sacred Heart University; Benjamin S. Manning of the Massachusetts Institute of Technology; Ilana Brody, Hengchen Dai, and Hal E. Hershfield of the University of Los Angeles; Ariel Kalil, Michelle Michelini, and Susan E. Mayer of the University of Chicago; Matthew D. Hilchey, Philip Oreopoulos, Renante Rondina, and Dilip Soman of the University of Toronto; Elizabeth Canning of Washington State University; and Sharon E. Parker of Philadelphia.

Research reported in this article was supported in part by an anonymous donor to Zearn Math. Support for this research was also provided in part by the AKO Foundation, J. Alexander, M. J. Leder, W. G. Lichtenstein, and A. Schiffman and J. Schiffman.