Ai2 推出 TutorMoments：AI 家教知道何时该帮、何时该放手吗？

August 7, 2026

Ai2

Today we're introducing a preview of TutorMoments , a framework to measure whether cutting-edge LLMs can balance one of the hardest trade-offs in education: when to step in and help a student and when to hold back and let the student do more of the work.

Told only to “tutor well,” we find that models tend to over-help by giving too much support and rarely pushing students to do deeper thinking. Spelling out the trade-off (when to help versus when to hold back) in the tutor's prompt improves performance, but it doesn't close the gap to human tutoring that consistently fits the moment, and LLMs still differ widely in how reliably they make that call.

As part of our commitment to open research, we're releasing a dataset of de-identified tutoring transcripts , the code for running our replay pipeline , and the model tutor replays of the key moments we evaluated in those transcripts for reproducibility. We hope TutorMoments gives educators, researchers, and the teams building AI tutors a sharper way to ask how a model handles the pedagogical decisions that matter most—and helps the field build tutors that adapt to each student instead of doing the work for them.

Ask a good math tutor for help and you’ll likely get a question back like, “What do you know about what the problem is asking?” That isn't unhelpfulness–part of strong teaching is diagnosing what students do know and providing the right support for them in the moment. Immediately volunteering support would rob a student of the intellectual work that helps them learn. Sometimes support is needed; other times what’s most effective is a push to solidify understanding by explaining a correct answer.

Language models, though, are trained to be helpful, and a helpful assistant tends to do the hard part for you—explaining the concept, laying out the steps, and guiding you to the answer. In a tutoring session, that can cut short the productive struggle—the effortful, sometimes frustrating problem-solving that learning research has long tied to stronger understanding.

Most benchmarks for language models acting as tutors don't capture this tension. They tend to reward one behavior in particular – never giving away the answer to a problem, say, or always offering a hint – without accounting for whether that was the right move for where the student actually was in their understanding. But good tutoring isn't a single fixed behavior you can identify across the board. It's a judgment call: what does this student need, right now, on this problem?

TutorMoments is built on real tutoring data. The dataset we’re releasing, TutorMoments-Preview , is 462 de-identified, text-only transcripts of real one-on-one math tutoring with U.S. students in grades 2-7, with more than 1,500 teacher-annotated key moments and several thousand free-text annotations from 27 U.S.-based teacher annotators. The transcripts come from a high-dosage tutoring program whose students mostly attend Title I schools, shared under a research clause agreed to by parents and guardians; all data was stripped of identifying details, first by the provider and then through an additional math-aware pipeline.

All annotations came from experienced math teachers, whom we asked to read the transcripts and mark key learning moments—noting what was going on, what the tutor did, and how it landed for the student. Each key moment is a decision point where the tutor had to weigh scaffolding (making a problem more accessible) against pushing for rigor (encouraging the student to do harder thinking).

TutorMoments runs by pausing a transcript at one of those key moments and handing the session to a language model, which takes over as the tutor for five turns with a simulated student. We call each of these model-generated continuations a replay . An LLM-based scoring pipeline then rates each replay on three things: whether the model (1) scaffolded when the student needed support, (2) pushed for rigor when the student was ready for more challenge, and (3) avoided over-scaffolding (reducing the challenge more than the moment called for).

The scoring pipeline starts from a teacher-defined ground truth: for each key moment, whether it called for scaffolding or for a push for rigor. Several teachers annotated each moment, and when they disagreed we took the majority label—if three teachers annotated a moment and two called for rigor while one called for scaffolding, the ground truth is rigor. A separate LM classifier validated against teacher annotations then decides whether the tutor's actual move matches what the moment called for—an "appropriate" turn means the tutor's classified action (scaffold, push for rigor, or over-scaffold) lines up with what teachers judged the moment to call for.

Long form tutoring sessions between a professional tutor and student in grades 2–7. The average session lasted 50 minutes and included hundreds of turns.

Teachers select Key Learning Moments where the tutor faced an important pedagogical decision about scaffolding or pushing for rigor, and set a cut point for the model to take over.

Teachers annotate why the Situation is a Key Moment, what pedagogical Actions the tutor took, and describe how effective the Result was.

A Language Model uses the teacher's annotations to classify the moment as appropriate to push for rigor or appropriate to scaffold .

Language Model tutors get the entire transcript up to the cut point and generate 5 turns by interacting with a synthetic student.

The judge reads the replayed exchange and labels the move the model actually made — then checks it against the expert-annotated moment type .

Across every annotated moment: the share where the model's judged move matches the expert's.

We also deconstruct both human and AI tutor actions into a taxonomy of tutor moves , letting us compare which moves are used more by humans and AI in the same situations.

Back in the replay, the same taxonomy names what the LM tutor did with the synthetic student : recasting 4/4 as a pizza is an Alternative Representation — the same idea offered in a different form.

We ran seven LLMs through TutorMoments using two prompts: a plain prompt that gives no real guidance – it only tells the model to use what it knows about good tutoring to respond to the student – and an evaluation-aware prompt that spells out the trade-off between scaffolding, over-scaffolding, and pushing for rigor. Each model was scored over key moments drawn from the tutoring transcripts, split evenly between moments where scaffolding was the right approach and moments that called for rigor.

Every number in the table is a rating between 0 and 1 – the share of the relevant moments where the model did the appropriate thing – so a higher score means the model made the right call more often. A 0.50 on appropriate rigor, for instance, means the model pushed for rigor in half of the moments that called for it.

A few things to keep in mind when reading the scores:

Human tutors are a naturalistic reference, not a ceiling. We don't treat human tutors as a model of ideal practice—even experienced tutors make less-than-optimal choices in the moment. Scored the same way at the same decision points, the human tutors in our transcripts get 0.458 (appropriate scaffolding), 0.182 (appropriate rigor), and 0.496 (avoids over-scaffolding)—all below the models' evaluation-aware scores and around the range of their plain-prompt scores. But this isn’t a claim that AI tutors outperform human teachers. Annotators specifically looked for moments where tutoring could have gone better, so the dataset concentrates on missed opportunities rather than ideal practice.

The scores measure tutor behavior, not learning. Replays use a simulated “oracle” student, so the numbers reflect how a model acts at a decision point—not whether a real student learned.

Rigor is noisier than scaffolding. The scoring pipeline detects rigor pushes less reliably, and there are fewer rigor moments (260) than scaffolding moments (738) in the underlying annotations.

The clearest pattern in the table is how much the prompt matters: every model scores higher under the evaluation-aware prompt than under the plain one. That suggests a model's default "helpful assistant" behavior isn't enough on its own to tutor well. But spelling out the trade-off in the prompt only goes so far—while it lifts every score, models still differ widely in how they interpret the enhanced prompt and even the best scorers have plenty of room to improve.

We also break down the moves that tutors made under each scenario. While prompting encourages models to push for rigor, they use fewer strategies than humans do, often relying on asking students to explain their answers. In contrast, human tutors employ more varied strategies and are much more likely to step back and let students work independently.

TutorMoments is still early in its development, and it has several limitations at this stage. The biggest is that automated evaluation gives us signal about how a model behaves at a decision point, but it can’t stand in for studies with real students and real learning outcomes. The dataset is also narrow: U.S.-based, mostly elementary and middle-school math, annotated by a single pool of educators. Our findings may not generalize to other subjects, grade levels, or settings.

We’re sharing this preview to gather feedback as we build toward a larger, multimodal dataset, a stronger scoring pipeline, and deeper analysis.

Acknowledgments

This project has been made possible in part through support from the Gates Foundation and Learning Commons.

At Ai2 we’re building the future of transparent, open-source AI — built in the open to empower scientific progress and fundamental understanding of this world changing technology. We’re not here to make profits, we’re here to make sure benefits of AI are shared widely and for the benefit of humanity. If this appeals to you, please take a look at our open roles.