Epi2Diff uses large model reasoning traces to quantify question difficulty, achieving 73.0% classification accuracy on SAT Math

How difficult is a test question for humans, and can clues be found in the problem-solving process of large models?

A research team from Virginia Tech, the University of Maryland, MBZUAI, and the University of Pittsburgh proposed Epi2Diff: it splits the reasoning traces generated by large models into "cognitive segments" and, combined with the question content, predicts the difficulty of human answering.

On four real assessment datasets—SAT Math, SAT Reading and Writing, Cambridge, and USMLE—Epi2Diff ranked first in all metrics in the single main experiment. Among them, the SAT Math difficulty classification accuracy reached 73.0%, 3.2 percentage points higher than the strongest baseline.

The related paper has been accepted by the machine learning journal Transactions on Machine Learning Research (TMLR).

Research background: beyond question semantics, there is also the problem-solving process

Judging the difficulty of new questions is a fundamental task in test assembly and item bank construction. Traditional educational measurement relies on calibrating with examinee response data, which is reliable but has high pretesting costs; analyzing only the semantics of the question stem also makes it difficult to determine where the problem-solving process is actually effortful.

Epi2Diff extends the observation window to the problem-solving process: how long did the model reason, was it more analyzing or executing, did it repeatedly switch strategies? These statistically measurable behaviors provide new process signals for difficulty prediction.

△

It should be emphasized that what the study aims to predict is human question difficulty. Model traces are proxy evidence that can be obtained at scale, and the final prediction is still supervised by existing human difficulty labels; it is not a direct measurement of human cognition.

Method design: turning reasoning traces into structured features

Eight types of cognitive segments: what is the model doing?

A reasoning passage often contains different activities such as understanding the question, formulating a strategy, executing calculations, and checking the answer. Drawing on Schoenfeld's episode theory of problem solving and the ThinkARM extension, the study classifies each sentence in the trace into eight categories: reading, analysis, planning, execution, exploration, verification, monitoring, and answering.

These "segments" are sentence function labels, not a fixed eight-step problem-solving procedure. A RoBERTa-based classifier achieved 79.9% accuracy and 0.761 macro-average F1 on ThinkARM's 7,067-sentence human-annotated mathematical reasoning test set; accuracy on supplementary annotation evaluations in the target domains was 62.9%–74.2%, indicating that cross-domain automatic classification still has errors.

83-dimensional process features: how does the model solve the problem?

Epi2Diff further extracts 83-dimensional features from the traces: 3 length statistics, the token counts of the 8 segment types, the token proportions of the 8 segment types, and the counts of 64 types of adjacent segment transitions, characterizing respectively "how much is invested," "where it is invested," and "how the process unfolds."

For example, two traces may have similar lengths and similar proportions of analysis and execution, but one first concentrates on analysis and then executes continuously, while the other frequently switches between analysis and execution. Transition features preserve this kind of order difference.

△

In the experiments, traces were generated through QwQ-32B and Qwen3-32B under direct-answer and prompt conditions labeled "weak, medium, strong," and then the features were averaged. These prompts were used to induce different problem-solving paths and do not represent validated student ability levels.

Finally, the process features are concatenated with the question semantic vectors obtained by Sentence-BERT, and difficulty is predicted through a supervised model. After selection on the validation sets for the four datasets, XGBoost was ultimately used as the predictor for all of them.

Experimental results: four types of real assessment tasks

Classification and regression performance

The evaluation includes 1,075 SAT Math questions, 1,338 SAT Reading and Writing questions, 793 Cambridge English Reading questions, and 667 USMLE medical questions. The two SAT tasks predict three classes—"easy, medium, hard"; the other two predict continuous difficulty values. Baselines cover fine-tuning of small models such as BERT, zero/few-shot prompting of LLMs, and full-parameter and LoRA fine-tuning.

△

SAT Math accuracy increased from 69.8% for the strongest baseline to 73.0%, and macro-average F1 increased from 0.705 to 0.728. The improvement on SAT Reading and Writing was smaller: accuracy rose from 62.3% to 63.1%, and macro-average F1 rose from 0.622 to 0.626.

In the regression tasks, Cambridge's RMSE dropped from 8.200 to 7.612, and USMLE's from 0.298 to 0.291; the corresponding R² rose from 0.299 to 0.396 and from 0.076 to 0.121, respectively.

Repeated experiments and boundaries of the results

In five supplementary runs, Epi2Diff still had the highest average accuracy and macro-average F1 on the two SAT tasks. However, the lead on the Reading and Writing task was smaller than the run-to-run variation, so the advantage should be interpreted cautiously. The repeated experiments fixed the traces and segment features and changed only the random seeds of the downstream predictor.

Ablation analysis: longer reasoning does not mean a harder question

The SAT Math ablation results show that the performance gain does not come only from trace length. With question semantics alone, macro-average F1 was 0.607; after adding length features, it was 0.671; using the full representation reached 0.728. Even when other features are retained, removing segment transition information reduces it to 0.676.

△

A further order-shuffling experiment retained trace length and segment frequencies and changed only the arrangement of the intermediate reasoning sentences and labels. The original order performed better, supporting that local transition order carries predictive information.

In SAT Math, harder questions are also accompanied by more intermediate reasoning, higher execution investment, and more pronounced "analysis→execution," "execution→execution," and "execution→analysis" transition signals. However, these patterns have limited reproducibility in other domains; they are process correlations and cannot be regarded as direct observations of human cognition or causal proof.

Application value: providing interpretable clues for item bank construction

Epi2Diff attempts to turn "how a question is solved" into comparable features, providing auxiliary clues for pre-assessing the difficulty of new questions and for manual item review. Practical use still requires labeled data for training, validation on the target population, and expert review. Its value is not to replace educational measurement, but to add a set of evidence from the reasoning process to judgments of question difficulty.

English text is being prepared.