Mentivis OS OpenMentivis OS EntrepriseMentivis OS Edu
Contact usDemo
All articles
M · AI & TrainingGeneral

Published 9 September 2026 · Updated 10 October 2026 · By Marie Castelli, Partner, Mentivis

Learning with AI: what the research really says

Research shows that AI improves learning when it makes learners produce, recall and explain, and that it can harm learning when it hands over the answers. Intelligent tutoring systems achieve on average 0.66 to 0.76 standard deviations of progress. For generative AI, a trial published in PNAS in 2025 measured a 17% drop in exam results among high school students who had used ChatGPT without guardrails.

Pillar: Adaptive learning and personalised learning paths: the guide

What does research show about intelligent tutoring systems?

Intelligent tutoring systems (ITS) have been studied for decades. Kurt VanLehn's review (2011) compares human tutoring, computer tutoring and no tutoring. It measures an effect of 0.79 standard deviations for human tutoring and 0.76 for systems that intervene at each step of the learner's reasoning, against 0.31 for those that only respond to the final answer.

These figures are often presented as the solution to Bloom's "2 sigma problem" (1984), which attributed an effect of two standard deviations to one-to-one tutoring. That is inaccurate: 0.76 standard deviations is a little over a third of two sigma. VanLehn's finding is of a different kind: a good tutoring system comes close to the effectiveness of a human tutor, which is itself well below what Bloom reported.

The meta-analysis by Kulik and Fletcher (2016) of 50 controlled evaluations finds a median effect of 0.66 standard deviations, i.e. moving a student from the 50th to the 75th percentile. Effects are much smaller on standardised tests than on tests designed by the project teams, and small when implementation was flawed.

What do recent studies show about generative AI?

The first controlled trials of language models confirm that the design of the tool determines the effect.

Study Population Main result
Bastani et al., PNAS, 2025 Nearly 1,000 high school students in Turkey, mathematics With unrestricted GPT-4: +48% on practice problems, then -17% on the exam without AI compared with the control group. With a guided version: +127% on practice, no significant difference on the exam
Kestin et al., Scientific Reports, 2025 194 physics students at Harvard AI tutor designed according to teaching principles: median learning gains more than twice those of an active-learning class, in a median 49 minutes against 60
Fan et al., British Journal of Educational Technology, 2025 117 students, writing task The group helped by ChatGPT improved its essay more, without greater knowledge gain or transfer

Bastani et al. describe AI being used as a "crutch": students get the answer without doing the reasoning, then perform worse on their own. Fan et al. speak of "metacognitive laziness". Conversely, Kestin et al. show that a tutor guiding learners step by step, with answers and prompts written by experts, can outperform a good classroom session; the authors note that the result concerns introductory material and does not necessarily hold for complex reasoning.

Why does an AI that gives the answers lead to less learning?

The mechanism is well known: the testing effect. Roediger and Karpicke (2006) showed that students tested on a text remembered it much better two days and one week later than students who had reread it, even though the latter felt more confident. The practice guide of the Institute of Education Sciences (What Works Clearinghouse, 2007) rates using quizzes to re-expose learners to key content among its recommendations with "strong" evidence.

Soderstrom and Bjork (2015) distinguish performance during learning, which can be observed immediately, from lasting learning, which is measured later. The two can diverge, and learners often mistake their ease in the moment for real learning. Reading a well-written answer produced by an AI provides that ease without the recall effort that consolidates memory.

How should you read a study on AI in training?

  • The control group. A before-and-after comparison on the same group does not isolate the effect of AI. The randomised controlled trial remains the benchmark.
  • The measure. A result obtained with the tool says nothing about what the learner can do without it; the study by Bastani et al. shows this clearly.
  • The delay. Immediate effects and effects after several weeks can diverge.
  • The test used. Kulik and Fletcher (2016) observe larger effects on locally developed tests than on standardised tests.
  • The effect size. Kraft (2020) points out that effects considered "small" by Cohen's thresholds are in fact large compared with most education interventions evaluated in the field.
  • The population. Results obtained with high school or university students do not transfer automatically to adults in vocational training.
  • The funding. A study run by a vendor on its own product is not equivalent to an independent, peer-reviewed study.

The practical consequences for designing a learning path are covered in How AI personalises and adapts a training path and What AI cannot do in training.

Frequently asked questions

Is AI more effective than traditional training?

It depends on its design. An AI tutor that guides without giving the answer outperformed an active-learning class in the trial by Kestin et al. (2025); unrestricted access to ChatGPT reduced exam results in the trial by Bastani et al. (2025).

Do intelligent tutoring systems reach Bloom's "two sigma"?

No. The best syntheses measure about 0.66 to 0.76 standard deviations, a large effect but far below the two standard deviations Bloom reported in 1984.

Where can you find reliable studies?

In peer-reviewed journals (Educational Psychologist, Review of Educational Research, PNAS, Psychological Science), in the practice guides of the What Works Clearinghouse of the Institute of Education Sciences, and in OECD publications. Vendor white papers without peer review are marketing documents.

Should learners be banned from using generative AI?

The research does not point that way. It shows that a tool designed to make learners think (questions, hints, explanations to produce) limits the negative effects observed with unrestricted access.

Sources

Further reading

Related products: MentivisOS Entreprise, MentivisOS Open

What learners expect from an AI learning platform in 2026