A new chatbot raises questions about the utility of education research
The Trump administration has gutted the Institute of Education Sciences. It’s yanked contracts and zeroed out staff without explanation. This could be a big problem. Federal data like the National Assessment of Educational Progress (NAEP), data on school staff and spending, and measures of college outcomes are vital if we’re to sensibly discuss K–12 and higher education.
But, while I’m very concerned about what’s in store for collecting these statistics, I’m far less sympathetic to the complementary claim that these cuts are costing us valuable insights into “what works.” I just don’t feel like I encounter much education research that provides credible, concrete guidance when it comes to pedagogy, instruction, or classroom management.
Am I unduly skeptical? Well, spurred by fears that the knowledge in ED’s What Works Clearinghouse might be lost, The Learning Agency (TLA) stepped in and released an AI chatbot that helps users navigate the “Doing What Works Library,” which is a federally funded effort designed to make the body of research in the What Works Clearinghouse more accessible and actionable for researchers. This offers a terrific opportunity to gauge the practical value of the accumulated research.
For once, I am not making satirical use of chatbots. This is real. You can try the chatbot yourself here. I appreciate TLA’s moxie in standing it up and was curious to see how helpful the results really were. So, I asked it 10 questions and then judged each answer as Informative, Obvious, or Unhelpful. You can score them yourself as we go along. Agree or disagree, let’s keep track and then compare notes at the end.
(Now, it’s important to keep in mind that the value of the responses will speak both to the research itself and to the caliber of TLA’s chatbot, meaning that any conclusions should be appropriately hedged.)
Here we go:
Q: How many hours of homework should I assign to eighth graders in math each day?
The Learning Agency (TLA) Chatbot: Assigning a moderate amount of homework, such as around 30 minutes to an hour, for eighth graders in math each day can be beneficial.
Q: I’m a new teacher trying to craft a seventh-grade classroom policy for student behavior. My education professors told me I should be positive, not strict. But veteran teachers tell me there are times I’ll need to be strict. What does the research suggest?
TLA Chatbot: Research suggests that positive reinforcement is more effective in promoting positive behaviors in students compared to using negative consequences or punishment. By focusing on positive behaviors and providing incentives for students to exhibit those behaviors, they are more likely to continue displaying positive behaviors in the long run. This approach encourages students to understand that their positive actions will lead to rewards, creating a more conducive and motivating learning environment.
Informative, but also problematic in that it promotes a contested view as conclusive fact.
Q: I try to be positive with my middle school students, but they keep ignoring me, looking at their phones, and breaking the rules. What does the research suggest I do to ensure classroom management?
TLA Chatbot: Provide specific feedback to students, focus on understanding why they may be off task, and consider whether their basic skills need strengthening before moving on to the next step.
Obvious, and not really about classroom management per se. But not bad instructional advice.
Q: How big are the benefits of phonics instruction for students in K–3?
TLA Chatbot: The benefits of phonics instruction for students in K–3 are significant, as it helps them develop essential reading skills such as word attack and recognition. Typically, students who receive phonics instruction experience substantial growth, often progressing several grade levels in their reading abilities by the time they complete the program.
Get the latest from Rick, delivered straight to your inbox.
Q: What’s the best way to help students build phonics skills in K–3?
TLA Chatbot: The best way to help students build phonics skills in K–3 is through a multisensory approach that incorporates activities such as manipulating soundcards, encoding, speaking sounds, writing sounds, and working with syllable cards. This approach helps students engage with the material in various ways, reinforcing their learning and understanding of phonics rules.
Q: I’ve heard it’s better for students to read printed materials rather than on screens. What does the research say?
TLA Chatbot: Research indicates that reading printed materials can be more beneficial for students compared to reading on screens.
Unhelpful, no specifics or clarity provided.
Q: I’ve heard it’s better for students to read on screens rather than printed materials. What does the research say?
TLA Chatbot: Research suggests that students benefit from discussions around text to improve comprehension.
Unhelpful, irrelevant—and really should just reverse the prior response, right?
Q: At what grade level should I introduce my students to fractions?
TLA Chatbot: Introduce students to fractions at a grade level where they can start to understand the concept of fractions as numbers and their role within the number system.
Unhelpful, though appealing in a certain Zen-like way.
Q: What does the research say are effective strategies for promoting student engagement in high school?
TLA Chatbot: Effective strategies for promoting student engagement in high school include implementing reading strategies across the curriculum, fostering interdisciplinary programs where teachers collaborate, sharing resources and strategies among teachers, providing targeted support for students with specific needs such as English language learners, and involving parents in the college-going process to increase college access for students.
Informative, if pretty vague.
Q: What does the research say is the best way to help students learn geography?
TLA Chatbot: The research suggests that promoting students’ 3-D spatial skills through hands-on experiences, combining classroom manipulation with computer exercises, and offering elective courses focused on spatial abilities is an effective way to help students learn and excel in math and science, preparing them for potential future fields like engineering.
Unhelpful, and oddly ignores the actual question.
What’d you get? I judged five answers to be informative, one to be fairly obvious, and four to be unhelpful. But of the five informative answers, one struck me as problematic and another as pretty nebulous.
Bottom line: Only three out of ten answers struck me as clearly useful—and two of those had to do with early reading. The other responses ranged from vague folk wisdom to problematic answers that offered dubious advice. When it came to homework or discipline, it felt mostly akin to asking a medical chatbot how often I should administer aspirin to a four-year-old and being told to give them a “moderate amount in a reasonable timeframe.”
I don’t want to be overly critical. The results reflect on a few different things: the chatbot, the Doing What Works Library, and the underlying research. The chatbot is fairly new, and the answers we get from AI are a function of the question, which is why it’s useful to rephrase queries or ask follow-ups. (In this case, the real takeaway may be the limits of my interrogation skills). And getting any useful responses this easily is certainly better than nothing. So, there are plenty of reasons to treat this exercise with appropriate caution.
That said, I’m not sure the results here were any more useful or compelling than what I could’ve found on Pinterest. That can’t help but raise hard questions about the utility of education research, the accessibility of results, and how findings are being translated into practice. As much as I’m concerned about the fate of NAEP or IPEDS, this little exercise reminds me why I’m leery of emphatic claims that cuts to education research are going to have devastating consequences for teaching and learning.
Last Updated
NEWSLETTER
Notify Me When Education Next
Posts a Big Story
Copyright © 2025 President & Fellows of Harvard College
