Online Math Help That Works

At a kitchen table somewhere in America, a seventh grader is doing her math homework. She works through a set of problems on paper, one by one, entering the answers into a computer program that instantly tells her whether those answers are correct. When she gets a problem wrong, the computer may offer a hint, it may explain the math behind the problem, or it may walk her through the problem, step by step (a process called scaffolding).

The next morning, when she takes her seat in class, her teacher has already viewed a report that shows how she and her 22 classmates did on the previous night’s work. Seeing that nearly everyone struggled with the same two problems, the teacher will take the time to discuss the underlying concepts of those problems before launching into a new lesson.

These scenarios may not seem revolutionary, but they represent a significant upgrade in the way middle-school math instruction has traditionally unfolded, with students completing homework largely unaware of whether their answers are right—or why they may be wrong—and teachers not fully tapped into what their charges have and have not mastered. The key difference for students and teachers is simple: feedback. When students get immediate feedback and help, they learn more. When teachers get feedback on the performance of the whole class, they can focus their efforts where it is most needed.

From an experimental web-based program rolled out in a small number of Central Massachusetts communities in the early 2000s, ASSISTments has grown into a robust educational and research platform that is made available at no cost to any teacher or school that wants to use it. So far, it has helped more than a million students around the country, earned top honors in a U.S. Department of Education online resource, served as a lifeline to teachers whose students have grappled with the challenges of remote learning during the COVID-19 pandemic, and won more than $67 million in funding from federal agencies and philanthropic organizations, more support than nearly any other research program at WPI.

Neil Heffernan’s interest in making education work better began at an early age. As a high school student in Florida, he read books on education reform and browsed copies of Phi Delta Kappan, a journal on K-12 education. “That’s what principals read,” he says, “and I was geeking out on it in high school. I was clearly a weird outlier.”

At Amherst College, where he majored in computer science and history, he organized 75 students and transported them to nearby Holyoke, Mass., to tutor elementary school students. At Amherst he also recruited students to Teach for America (TFA), which sends college graduates on two-year stints to teach in under-resourced schools. After signing up about five percent of his class, he also joined TFA and found himself teaching math in a predominantly Black section of Baltimore.

“I was totally convinced,” he says, “that I wanted to be Jaime Escalante, the teacher portrayed in the movie Stand and Deliver—the guy who was personally responsible for one-quarter of all Mexican Americans who passed AP calculus. Turns out, I was no Jaime Escalante. My classroom management was really bad. I set a record for the most students sent to the principal’s office.”

The experience did prove to be the start of a career, just not the one he had envisioned. Recognizing the challenges teachers face as they seek to move an entire class of students along a learning path without leaving anyone behind, and drawing on his knowledge of artificial intelligence, Heffernan wondered if educational technology might offer a solution.

Then three things happened: He enrolled at Carnegie Mellon University to pursue a PhD under the mentorship of some of the leaders in the development of intelligent tutoring software; he met Cristina Lindquist, the future Cristina Heffernan, who was teaching middle school math after returning from the Peace Corps; and he was diagnosed with brain cancer.

For his dissertation, he built an algebra tutor, in part by recording Cristina as she conducted one-on-one tutoring sessions with students. “I just watched what she did,” he says, “and made the computer do some of her ‘moves.’” Heffernan released the program, called Ms. Lindquist, on the Web for free, and it quickly became the most widely used intelligent tutoring system then available.

Several years into his PhD program, he received his brain tumor diagnosis and was given no more than three years to live. Fortunately, he found a surgeon who could remove the tumor and Heffernan remains cancer free to this day.

The near-death experience made him rethink his priorities. He took a break from his PhD program and began teaching in Sudbury, Mass. To help his students prepare for the MCAS (Massachusetts Comprehensive Assessment System), the state’s mastery test for public schools, he printed out sample problems for students to complete on paper. “The obvious thing,” he told the WPI Journal in 2006, “was to put those problems in the computer and to give students immediate feedback when they got one wrong.”

“Feedback is so important to cause learning,” he says today. “You don’t learn from just doing more problems. Or, at least, that’s not the most effective way. For all the kids who do the problems and get them all wrong, that’s clearly detrimental.”

Infused with a new sense of purpose, he returned to finish his PhD and joined the faculty at WPI, where he pursued a better way for students to learn via technology. That was the beginning of ASSISTments.

From the beginning, education and research have combined in a virtuous cycle that has kept the platform continually evolving. When students and teachers use ASSISTments, their experience becomes data that can be mined for clues to potential new features and approaches, as well as insights into education, itself—particularly online learning.

“I’m proud of that,” Heffernan says. “I’ve made WPI known as one of the homes of open science. And we recently received $2 million from the U.S. Department of Education (DOE) just to make our platform have better capabilities to do this type of research.”

Heffernan and his team are also in the running for the $1 million Digital Learning Challenge hosted by the X-Prize Foundation. Sponsored by the DOE, the challenge seeks to “modernize, accelerate, and improve the ways in which we identify effective learning tools and processes that improve learning outcomes.”

“They want to reward the organization that can best run experiments at scale,” he says. “And it is probably true, if I may be so immodest, that no one comes close to what we are doing.”

Of all the research conducted about ASSISTments, one study looms particularly large, for it sought to answer a question to which Heffernan had long been able to provide only an educated guess (along with ample anecdotal evidence): Does ASSISTments really improve student learning? Funded by a $3.5 million award from DOE’s Institute of Education Sciences, SRI International ran an independent, objective study in Maine where every middle school student receives a laptop to use in class and to take home to do homework.

The study took 43 public schools, with 2,850 seventh graders, and paired them according to a variety of demographic factors. In half the schools, students did their homework in the traditional fashion. In the other half, they used ASSISTments and teachers received daily reports on their work. As reported in the journal AERA Open, when measured by scores on standardized testing, students who used ASSISTments realized a 75 percent improvement in learning over what would have been expected in a typical year. The effect was greatest for students in the bottom half of their classes; their learning improved 100 percent.

The study yielded significant benefits. For one, the DOE included the study in its What Works Clearinghouse, giving the highest possible rating: highest rating: “Meets WWC Standards Without Reservations.” It noted that the study “provides the highest degree of confidence that the intervention caused the observed effect.” For another, the Office of Science and Technology Policy asked Heffernan and the authors to present the study results at the White House.

The What Works listing opened new pathways to funding from the government and from public charities, including the Gates Foundation. For example, it helped gain the attention of the Schmidt Futures Foundation, started by Eric Schmidt, former CEO of Google, and his wife, Wendy. A $2 million grant from the foundation and other philanthropic support helped launched the ASSISTments Foundation and scale up the platform.

A similar study in North Carolina aimed at answering that question ended before it could be completed when school closures during the pandemic preempted a required post-test. Schools are currently being recruited for a new nationwide study that may finally show what the Heffernans have long believed: when teachers use their daily ASSISTments reports to continually refine their teaching in response to what students are and are not learning, the whole class benefits. But students who most need a helping hand benefit the most.

When the COVID-19 pandemic led schools across the country to shut their doors and switch to online learning, millions of teachers, students, and parents needed an immediate helping hand. Some sought help from Web-based educational programs; for many, that search led them to ASSISTments. In early 2020, the creation of new ASSISTments accounts rose from the typical rate of about 300 a month to 6,000 in March and 7,000 in April. By September, more than 25,000 new teachers in all 50 states, and across all grade levels, were using the platform.

In response, the ASSISTments team, which has grown to 17 full-time employees, revamped the ASSISTments website to make it easier for new teachers to dive in, expanded the help center with weekly webinars for new and experienced teachers, and added a new Teacher Corner to share best practices.

The rapid expansion of ASSISTments was aided by the fact that the team had already entered the content of the most commonly used free middle school math curriculums into the platform and integrated it with Google Classroom and Canvas, the two most widely used K-12 educational delivery systems.

“It is now so easy for someone to adopt ASSISTments,” Heffernan says. “I’m a seventh-grade teacher and just taught module 5, and I ask, ‘Can I assign module 6?’  Here are the exact problems. Just hit the button to assign them.”

In future work, Heffernan, using cutting-edge AI, psychology, education theory, and sheer determination, will continue to seek to make ASSISTments even easier to use, more effective, and smarter (although he says he steers clear of using the label “intelligent tutor,”  which some educators associate with overhyped and ineffective products). “I like to tell teachers I get grants to make slightly less-dumb educational software,” he says.

Many of his current projects focus on crowdsourcing. Already, Heffernan is encouraging teachers to submit hints, explanations, scaffolding items, and best practices to benefit other teachers.  “I want to build an ecosystem where teachers are writing content because they want to help their own kids,”  he says. “We can then test that content at scale and see what works.”

As with Wikipedia and Stack Overflow, a resource for programmers, this altruistic output, through testing and iteration, will find its way out into the world to make a difference in the lives of students and teachers. “That’s what it has always been all about for Cristina and me,”  Heffernan says.  “Figuring out how to properly motivate children and cause better learning.”