We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
Last week's briefs asked whether AI tools can show they work and whether students use them to think. This week's evidence makes a narrower point: the tool's design matters, but whether students use it at all matters more. The organizations buying these tools are starting to write that lesson into policy and contracts.
AI tutors: good design helps, but students have to use them
A tutoring research panel hosted by Justin Reich offered the clearest data so far this fall.
- Khanmigo in Tennessee. A randomized study added Khan Academy's AI tutor to remedial math practice. The gains, around 0.15 standard deviations on state tests by the second year, were roughly the same as Khan Academy produced before AI. The researcher's explanation was that students rarely used the tutor: the median student opened it after only about 17% of mistakes . In a follow-up one-hour trial with 7,000 students, every mistake triggered a required AI walkthrough. Students moved more slowly but were more likely to get the next question right, and slightly more likely a week later. The researcher called this a proof of concept, and concluded that evidence AI adds value on top of these platforms is "not yet" there .
- An AI tutor that starts from a diagnosed misconception. The ED platform runs a diagnostic question first, then opens a chat aimed at the specific error the student made . In a 165-student trial with a human reviewing every AI message, 76% of AI drafts went out unchanged and no harmful messages reached students . Students got the next question right 66% of the time with the supervised AI, 60% with a human tutor, and 56% with a static hint . A trial ten times larger, with about 1,200 students, reports on October 29. A full-year UK trial with 15,000 students is now running .
- Why answer-giving hurts. Panelists again pointed to the Wharton study in Turkey. Open ChatGPT access raised practice scores 48% but lowered unaided exam scores 17%, while a version that gave hints instead of answers avoided that harm . There are limits on the other side too. Breaking a problem into too many tiny steps is another way of offloading the thinking , and tutors that withhold answers can frustrate students enough that they stop engaging .
- Most students barely used it. Stanford researchers ran two randomized trials with about 350 students in grades 1–5 on an AI literacy platform that called for 30 minutes twice a week. Just over half of the students working alone ever logged in, and they averaged 2–5 minutes a week . The students who did log in tended to be higher-achieving and less likely to have a disability . Adding a human tutor for accountability raised the number of stories read by 70–80%, but that came to less than two hours of extra practice per student, with no detectable reading gains . The researchers' advice to schools: plan AI tutor rollouts as seriously as human tutoring, with protected time, a caring adult, and weekly checks on usage .
For buyers: A well-designed tutor is necessary but not enough on its own. The deciding factors are things schools control: scheduled time, adult support, and usage monitoring.
The research base is weaker than the headlines suggest
Patrick O'Neil presented the audit Reich previewed last week. He judged all 14 AI-in-education meta-analyses he examined "severely problematic." Both "AI" and "learning" were loosely defined. One meta-analysis pooled elementary PE interventions with medical-school simulations, and three counted mental-health chatbots as evidence of learning . Of 46 primary studies he sampled from them, 28 were flawed or mismatched, some citing references that don't exist. Twelve of the 14 meta-analyses failed to account for dependent effects, and none adequately checked for publication bias . He also found that 65.8% of calls for papers in the field explicitly asked for research showing AI's benefits .
O'Neil also ran his own meta-analysis of 44 studies, applying strict criteria. The average effect was effectively zero, but the range of plausible effects ran from strongly negative (g = −0.92) to strongly positive (+1.03). He says that range, not the average, is the real finding . Susanna Loeb added that of about 880 papers in her repository, only 20 met the bar for credible causal analysis in the first review. Almost all were in higher education, and none studied U.S. students using AI . O'Neil's practical advice is to read individual studies and treat promising approaches as experiments in your own setting .
Buyers start writing outcomes into contracts
Edtech Insiders' takeaways from EDTECH WEEK in New York show where this is going. Twenty-four states now vet early-reading curricula against approved lists. Arizona, Florida, Missouri, and others are building ESSA evidence tiers into procurement and funding rules, which makes evidence a condition of entering the market . South Bend, Indiana, signed a middle-school literacy contract with Really Great Reading that sets requirements for dosage, attendance, and staffing, and ties 40% of the contract's value to students reaching proficiency on the 2027 state test . Speakers named the same "5% problem" the tutoring researchers found: most students never get near the dosage a study assumed. One proposed fix is usage targets that step up over time, owned jointly by the district and the vendor . For now, vendors bear most of the cost of producing evidence, which favors larger companies .
NYC's AI restrictions hit more than 38 products, and some vendors don't know why
Chalkbeat obtained New York City's list of more than 38 flagged products. It includes widely used reading and math curricula, assessment tools, and a chatbot the Education Department built itself . The rule targets student-facing generative AI features, not necessarily whole products, but the department hasn't explained how it decided which tools qualify, including in cases where vendors dispute that their tools use AI . Amira's AI reading tutor, used in more than 200 schools, was banned outright . Other vendors are adjusting feature by feature. All In Learning turned off AI-assisted grading, and Magma Math disabled AI hints on September 4 and was cleared to operate again on October 5 .
In Washington, policy is pulling in opposite directions. NTIA head Arielle Roth reportedly compared ed-tech to tobacco in a private meeting; she says she doesn't remember the specific analogies . She has also helped prompt a review of the $3 billion E-Rate fund . Meanwhile, the Education Department has issued relatively favorable guidance on ed-tech .
Teachers push back on mandated AI essay grading
A co-teacher reports that an administrator wants to require AI scoring for ELA essays. The teacher says the AI gave 80s to 100s to essays missing major components, including essays apparently written by AI . In a meeting, staff regraded five AI-scored papers and found the AI had inflated grades and written feedback that didn't match what students wrote, such as critiquing a thesis that didn't exist. The administration still wants a department-wide approach . Another teacher says their department dropped its mandate after the tool rated strong writing 1–2 out of 4 and gave copied paragraphs a 4 . These are individual accounts, but they show that AI grading can be wrong in both directions, too generous and too harsh, which makes it hard to plan reteaching around its scores.
Teen safeguards and student self-reports
Common Sense Media ran 4,000 prompts against ChatGPT for teens and rarely triggered its protection mode. Study mode was easy to exit: students could ask for the answer or delete "@study" from their prompt . Common Sense's August survey also found that 70% of U.S. teens use AI for schoolwork. Among teens who use AI, 38% said it leads them to come up with fewer ideas of their own, and 39% said they miss out on learning when they use it to finish assignments .
Higher ed keeps scaling access
Jisc signed a framework agreement with Google that makes Google AI Pro and Workspace for Education easier to procure for UK and Irish further- and higher-education institutions. It comes with on-campus training and free certifications . As the evidence above shows, giving students access does not guarantee they will use these tools in ways that help them learn.
Worth watching
- The ED platform's 1,200-student trial report on October 29, and later results from its full-year UK trial with 15,000 students.
- Whether NYC publishes its criteria for deciding which products count as student-facing AI.
- Whether more districts copy South Bend's outcomes-based contract, especially for AI tools whose results depend on how much students use them.
