Published 9 September 2026 · Updated 10 October 2026 · By Marie Castelli, Partner, Mentivis
Learning with AI: what the research really says
Research shows that AI improves learning when it makes learners produce, recall and explain, and that it can harm learning when it hands over the answers. Intelligent tutoring systems achieve on average 0.66 to 0.76 standard deviations of progress. For generative AI, a trial published in PNAS in 2025 measured a 17% drop in exam results among high school students who had used ChatGPT without guardrails.
Pillar: Adaptive learning and personalised learning paths: the guide
What does research show about intelligent tutoring systems?
Intelligent tutoring systems (ITS) have been studied for decades. Kurt VanLehn's review (2011) compares human tutoring, computer tutoring and no tutoring. It measures an effect of 0.79 standard deviations for human tutoring and 0.76 for systems that intervene at each step of the learner's reasoning, against 0.31 for those that only respond to the final answer.
These figures are often presented as the solution to Bloom's "2 sigma problem" (1984), which attributed an effect of two standard deviations to one-to-one tutoring. That is inaccurate: 0.76 standard deviations is a little over a third of two sigma. VanLehn's finding is of a different kind: a good tutoring system comes close to the effectiveness of a human tutor, which is itself well below what Bloom reported.
The meta-analysis by Kulik and Fletcher (2016) of 50 controlled evaluations finds a median effect of 0.66 standard deviations, i.e. moving a student from the 50th to the 75th percentile. Effects are much smaller on standardised tests than on tests designed by the project teams, and small when implementation was flawed.
What do recent studies show about generative AI?
The first controlled trials of language models confirm that the design of the tool determines the effect.
| Study | Population | Main result |
|---|---|---|
| Bastani et al., PNAS, 2025 | Nearly 1,000 high school students in Turkey, mathematics | With unrestricted GPT-4: +48% on practice problems, then -17% on the exam without AI compared with the control group. With a guided version: +127% on practice, no significant difference on the exam |
| Kestin et al., Scientific Reports, 2025 | 194 physics students at Harvard | AI tutor designed according to teaching principles: median learning gains more than twice those of an active-learning class, in a median 49 minutes against 60 |
| Fan et al., British Journal of Educational Technology, 2025 | 117 students, writing task | The group helped by ChatGPT improved its essay more, without greater knowledge gain or transfer |
Bastani et al. describe AI being used as a "crutch": students get the answer without doing the reasoning, then perform worse on their own. Fan et al. speak of "metacognitive laziness". Conversely, Kestin et al. show that a tutor guiding learners step by step, with answers and prompts written by experts, can outperform a good classroom session; the authors note that the result concerns introductory material and does not necessarily hold for complex reasoning.
Why does an AI that gives the answers lead to less learning?
The mechanism is well known: the testing effect. Roediger and Karpicke (2006) showed that students tested on a text remembered it much better two days and one week later than students who had reread it, even though the latter felt more confident. The practice guide of the Institute of Education Sciences (What Works Clearinghouse, 2007) rates using quizzes to re-expose learners to key content among its recommendations with "strong" evidence.
Soderstrom and Bjork (2015) distinguish performance during learning, which can be observed immediately, from lasting learning, which is measured later. The two can diverge, and learners often mistake their ease in the moment for real learning. Reading a well-written answer produced by an AI provides that ease without the recall effort that consolidates memory.
How should you read a study on AI in training?
- The control group. A before-and-after comparison on the same group does not isolate the effect of AI. The randomised controlled trial remains the benchmark.
- The measure. A result obtained with the tool says nothing about what the learner can do without it; the study by Bastani et al. shows this clearly.
- The delay. Immediate effects and effects after several weeks can diverge.
- The test used. Kulik and Fletcher (2016) observe larger effects on locally developed tests than on standardised tests.
- The effect size. Kraft (2020) points out that effects considered "small" by Cohen's thresholds are in fact large compared with most education interventions evaluated in the field.
- The population. Results obtained with high school or university students do not transfer automatically to adults in vocational training.
- The funding. A study run by a vendor on its own product is not equivalent to an independent, peer-reviewed study.
The practical consequences for designing a learning path are covered in How AI personalises and adapts a training path and What AI cannot do in training.
Frequently asked questions
Sources
- The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems, VanLehn, Educational Psychologist, 2011
- The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring, Bloom, Educational Researcher, 1984
- Effectiveness of Intelligent Tutoring Systems, Kulik and Fletcher, Review of Educational Research, 2016
- Generative AI without guardrails can harm learning: Evidence from high school mathematics, Bastani et al., PNAS, 2025
- AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting, Kestin et al., Scientific Reports, 2025
- Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance, Fan et al., British Journal of Educational Technology, 2025
- Test-Enhanced Learning, Roediger and Karpicke, Psychological Science, 2006
- Organizing Instruction and Study to Improve Student Learning, Pashler et al., What Works Clearinghouse, Institute of Education Sciences, 2007
- Learning Versus Performance, Soderstrom and Bjork, Perspectives on Psychological Science, 2015
- Interpreting Effect Sizes of Education Interventions, Kraft, Educational Researcher, 2020
Further reading
- Adaptive learning: definition, mechanisms and evidence
- Personalised learning paths in workplace training: myth or reality in 2026?
- How AI personalises and adapts a training path
- Individualised learning paths for French training providers and CFAs: rules and methods
Related products: MentivisOS Entreprise, MentivisOS Open