Randomized university trial finds course-integrated AI tutor lowered grades and sharply reduced participation

A new working paper is likely to complicate colleges’ rush to embed AI tutors inside regular courses. In a randomized controlled trial involving 2,379 undergraduates and 30 instructors at a large U.S. public university, students in course sections given access to a course-integrated generative AI tutor earned final grades that were 0.37 standard deviations lower than comparable sections without access. Their participation in the learning management system, or LMS, fell even more sharply: by 0.90 standard deviations. The paper also reports larger estimated academic losses for first-generation students. (edworkingpapers.com)

The study, “The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial,” was posted on EdWorkingPapers in October 2026 and republished by the University of Maryland’s Center for Educational Data Science and Innovation. That timing matters because higher education leaders are now moving from ad hoc student use of ChatGPT toward institution-backed AI helpers embedded in courses and campus systems. This paper does not settle whether AI tutoring works in general. But because access was randomized among sections of the same course, it offers unusually strong causal evidence that one real-world deployment can backfire. It is also still a working paper: EdWorkingPapers says papers on the site are circulated for comment before rigorous peer review, and authors can revise them. (edworkingpapers.com)

What the study tested — and what it did not

The paper evaluates a specific implementation choice: adding an AI tutor to live undergraduate courses across multiple disciplines at one institution. That is different from testing whether students can learn a discrete lesson from a custom-built AI tutor, or whether AI can help a human tutor do better work. The intervention here sat inside ordinary course routines, where students already had assignments, course pages, discussions, and other instructor-designed activities competing for their attention. The authors’ central interpretation is that the AI tool may have displaced engagement with those existing learning activities without improving performance. (edworkingpapers.com)

That displacement argument is what makes the paper more than a simple “AI failed” headline. The participation result is the clearest signal in the study, and in standardized terms it is much larger than the grade decline. In other words, the first thing the AI tutor seemed to change was not what students knew, but what they did: they showed up less in the platform behaviors their courses were already built around. For higher education operators, that is a practical warning. A course assistant can look additive on a product sheet while functioning subtractively in a real class if students treat it as a shortcut around discussion boards, practice activities, review materials, or instructor contact. (edworkingpapers.com)

The subgroup finding should also get attention. The paper says estimated academic losses were greater for first-generation students. Because the paper is not yet peer reviewed and the abstract does not fully unpack mechanism, institutions should be careful not to overstate what explains that pattern. But the equity implication is hard to ignore: if an AI tutor encourages some students to swap away from the routines that actually drive course success, the damage may be greatest for students who have less tacit knowledge about which routines matter most. (edworkingpapers.com)

A useful correction to the “AI tutoring” catchall

One reason this paper matters is that it pushes back on a flattening habit in education debate: treating “AI tutoring” as if it were one intervention. A Stanford SCALE brief published in August 2026 argued almost the opposite — that AI tutoring is “not a monolith.” The brief distinguishes among human-led tutoring augmented by AI, AI-led tutoring software used directly by students, and other hybrid models, and says current evidence supports using AI to strengthen tutor or educator capacity more than replacing high-impact tutoring with software alone. It also argues that the effectiveness of direct-to-student AI tools depends heavily on how they are integrated. (scale.stanford.edu)

That framing helps explain why the Maryland-linked university trial can coexist with more positive findings elsewhere. In August 2026, another EdWorkingPaper on Khan Academy’s Khanmigo reported modest math gains in a two-year experiment in 18 Tennessee middle schools, but the authors said the “binding constraint” was engagement: students had access, yet used the tutor infrequently and rarely in substantive mathematical dialogue. A different randomized study, Tutor CoPilot, found better results when AI supported human tutors in real time rather than interacting with students on its own; students working with tutors assigned to use the tool were 4 percentage points more likely to master math topics, with larger benefits for lower-rated tutors. And a 2025 Scientific Reports paper on a custom AI tutor in college physics found higher learning gains than an active-learning class in that tightly designed setting. (edworkingpapers.com)

Those studies are not contradictions so much as reminders that the implementation question comes first. Was the AI replacing a human, supporting a human, or substituting for existing student behaviors that already mattered? Was it used intensely in a designed lesson, or merely made available inside a semester-long course? Was the core challenge instructional quality, tutor expertise, or student follow-through? The new university trial suggests colleges should worry less about whether an AI tutor sounds helpful in isolation and more about what it changes in the surrounding workflow. (edworkingpapers.com)

What higher-ed teams should do with this finding now

The practical lesson is not that campuses should ban AI helpers from courses. It is that they should stop treating course-integrated AI as a low-stakes feature add. If the likely failure mode is substitution, then pilots need to be instrumented around substitution. Colleges should want to know not just how often students open the tutor, but whether tutor access coincides with less LMS activity, lower completion of course tasks, weaker discussion participation, or lower use of human support channels. Where possible, they should test within the same course across sections, as this study did, rather than relying on voluntary adoption or end-of-term satisfaction surveys. (edworkingpapers.com)

They should also watch subgroup patterns early. The first-generation finding in this paper is only one result from one institution, but it points to a broader governance problem for AI in higher education. A tool can appear efficient on average while quietly widening gaps if it changes study habits differently across student groups. That means product evaluation cannot stop at overall usage or average grades. It has to include who changed behavior, who benefited, who disengaged, and whether the tool altered the mix of human and platform supports that students were already using. (edworkingpapers.com)

What colleges should not do is overread this paper into a universal verdict on all AI tutoring. It is a single-institution study of one course-integrated implementation, and the paper’s own publication venue emphasizes that it has not yet gone through rigorous peer review. But that limitation cuts both ways. If leaders have been making deployment decisions based mostly on vendor claims, small pilots, or student enthusiasm, this paper raises the evidentiary bar. It suggests the right question is no longer, “Can we give students an AI tutor?” but, “What existing learning behavior will this tool replace, and how will we know quickly if that replacement is hurting students?” (edworkingpapers.com)

That is the development worth watching next: whether the findings replicate after peer review and at other campuses, and whether institutions respond by running tighter, better-monitored pilots instead of broad rollouts. For now, the strongest new evidence in the field is not a story of effortless personalization. It is a warning that in live college courses, an AI helper can cannibalize the very participation patterns faculty and institutions depend on. (edworkingpapers.com)