FirstSlip results

Can open AI models find the first wrong step in a student's algebra, and is their feedback about that step? Three Qwen3.5 models (2B, 4B, 27B) were prompted on 2,000 synthetic solutions with one injected mistake each and on 802 real, teacher-labelled student solutions (VtG). Hypotheses and thresholds were pre-registered before any test run.

Hypotheses

	Prediction	Result
H1	Finding the error is harder than solving the equation	Not supported (0 of 3)
H1b	Feedback often discusses a different step than the one flagged	Supported (2B, 4B)
H2	Being told the error's location improves the explanation	Not supported (1 of 3)
H2b	Models copy a wrong location most of the time	Not supported (1 of 3)

E1 · Finding the first mistake

The 27B model finds it 95% of the time on synthetic algebra and 87% on mistake types it was never shown examples of, but only 61% on real student work. The 2B model is barely above guessing.

Bar chart: first-error accuracy. Seen mistakes: 2B 35%, 4B 68%, 27B 95%. New mistake types: 27%, 49%, 87%. Real student work: 31%, 46%, 61%. Random guessing is 18 to 25%. — Bars: % of incorrect solutions where the model named exactly the first wrong step, with 95% confidence intervals. Dashed: random guess. Dotted: always guessing the most common position.

Dot plot: on the same equations, 27B solves 48% but finds the error 95%; 4B 11% vs 68%; 2B 6% vs 35%. — H1: every model finds errors far more often than it solves the same equation correctly, the opposite of the prediction.

Is the feedback about the flagged step?

An LLM judge, validated against 100 hand labels (97% agreement), read every feedback sentence. The 2B model flags one step and explains another about a quarter of the time; the 27B model almost never does. That is why this app shows the checker's location separately from the model's sentence.

Stacked bars: share of feedback about a different step. 2B 19 to 25%, 4B 5 to 17%, 27B 0.4 to 2.3%. — Blue: about the flagged step. Orange: about a different step. Grey: no clear step.

E2 and E3 · Giving the model a location

Telling the model the true wrong step does not make its explanation better (only 2B improves, and it copies any hint). Given a wrong step, 27B and 4B mostly ignore it; 2B copies it every time.

Diagnosis accuracy, not told vs told the location: 27B 47 to 49% (seen), 4B 18 to 18%, 2B 10 to 14%. — Feedback that names the real mistake (%), hollow = not told the location, filled = told.

Copying a wrong location: 2B about 100%, 4B 9 to 17%, 27B 2 to 7%. — How often each model repeats a wrong location it was given. The dashed line is the 50% bar for H2b.

What this means for a tutor

A symbolic checker can say where a step goes wrong with certainty; a model is needed to say what went wrong, and even the best model here gets that right only about half the time. So this app marks the checker's findings as verified and the model's sentence as a guess.

Prompt p2-v1, judge j-v1, greedy decoding. Full tables and pre-registration are in the project repository.