We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
Last week's briefs asked whether AI tools can show they work and whether students use them to think. This week's evidence makes a narrower point: the tool's design matters, but whether students use it at all matters more. The organizations buying these tools are starting to write that lesson into policy and contracts.
AI tutors: good design helps, but students have to use them
A tutoring research panel hosted by Justin Reich offered the clearest data so far this fall.
- Khanmigo in Tennessee. A randomized study added Khan Academy's AI tutor to remedial math practice. The gains, around 0.15 standard deviations on state tests by the second year, were roughly the same as Khan Academy produced before AI. The researcher's explanation was that students rarely used the tutor: the median student opened it after only about 17% of mistakes . In a follow-up one-hour trial with 7,000 students, every mistake triggered a required AI walkthrough. Students moved more slowly but were more likely to get the next question right, and slightly more likely a week later. The researcher called this a proof of concept, and concluded that evidence AI adds value on top of these platforms is "not yet" there .
- An AI tutor that starts from a diagnosed misconception. The ED platform runs a diagnostic question first, then opens a chat aimed at the specific error the student made . In a 165-student trial with a human reviewing every AI message, 76% of AI drafts went out unchanged and no harmful messages reached students . Students got the next question right 66% of the time with the supervised AI, 60% with a human tutor, and 56% with a static hint . A trial ten times larger, with about 1,200 students, reports on October 29. A full-year UK trial with 15,000 students is now running .
- Why answer-giving hurts. Panelists again pointed to the Wharton study in Turkey. Open ChatGPT access raised practice scores 48% but lowered unaided exam scores 17%, while a version that gave hints instead of answers avoided that harm . There are limits on the other side too. Breaking a problem into too many tiny steps is another way of offloading the thinking , and tutors that withhold answers can frustrate students enough that they stop engaging .
- Most students barely used it. Stanford researchers ran two randomized trials with about 350 students in grades 1–5 on an AI literacy platform that called for 30 minutes twice a week. Just over half of the students working alone ever logged in, and they averaged 2–5 minutes a week . The students who did log in tended to be higher-achieving and less likely to have a disability . Adding a human tutor for accountability raised the number of stories read by 70–80%, but that came to less than two hours of extra practice per student, with no detectable reading gains . The researchers' advice to schools: plan AI tutor rollouts as seriously as human tutoring, with protected time, a caring adult, and weekly checks on usage .
For buyers: A well-designed tutor is necessary but not enough on its own. The deciding factors are things schools control: scheduled time, adult support, and usage monitoring.
The research base is weaker than the headlines suggest
Patrick O'Neil presented the audit Reich previewed last week. He judged all 14 AI-in-education meta-analyses he examined "severely problematic." Both "AI" and "learning" were loosely defined. One meta-analysis pooled elementary PE interventions with medical-school simulations, and three counted mental-health chatbots as evidence of learning . Of 46 primary studies he sampled from them, 28 were flawed or mismatched, some citing references that don't exist. Twelve of the 14 meta-analyses failed to account for dependent effects, and none adequately checked for publication bias . He also found that 65.8% of calls for papers in the field explicitly asked for research showing AI's benefits .
O'Neil also ran his own meta-analysis of 44 studies, applying strict criteria. The average effect was effectively zero, but the range of plausible effects ran from strongly negative (g = −0.92) to strongly positive (+1.03). He says that range, not the average, is the real finding . Susanna Loeb added that of about 880 papers in her repository, only 20 met the bar for credible causal analysis in the first review. Almost all were in higher education, and none studied U.S. students using AI . O'Neil's practical advice is to read individual studies and treat promising approaches as experiments in your own setting .
Buyers start writing outcomes into contracts
Edtech Insiders' takeaways from EDTECH WEEK in New York show where this is going. Twenty-four states now vet early-reading curricula against approved lists. Arizona, Florida, Missouri, and others are building ESSA evidence tiers into procurement and funding rules, which makes evidence a condition of entering the market . South Bend, Indiana, signed a middle-school literacy contract with Really Great Reading that sets requirements for dosage, attendance, and staffing, and ties 40% of the contract's value to students reaching proficiency on the 2027 state test . Speakers named the same "5% problem" the tutoring researchers found: most students never get near the dosage a study assumed. One proposed fix is usage targets that step up over time, owned jointly by the district and the vendor . For now, vendors bear most of the cost of producing evidence, which favors larger companies .
NYC's AI restrictions hit more than 38 products, and some vendors don't know why
Chalkbeat obtained New York City's list of more than 38 flagged products. It includes widely used reading and math curricula, assessment tools, and a chatbot the Education Department built itself . The rule targets student-facing generative AI features, not necessarily whole products, but the department hasn't explained how it decided which tools qualify, including in cases where vendors dispute that their tools use AI . Amira's AI reading tutor, used in more than 200 schools, was banned outright . Other vendors are adjusting feature by feature. All In Learning turned off AI-assisted grading, and Magma Math disabled AI hints on September 4 and was cleared to operate again on October 5 .
In Washington, policy is pulling in opposite directions. NTIA head Arielle Roth reportedly compared ed-tech to tobacco in a private meeting; she says she doesn't remember the specific analogies . She has also helped prompt a review of the $3 billion E-Rate fund . Meanwhile, the Education Department has issued relatively favorable guidance on ed-tech .
Teachers push back on mandated AI essay grading
A co-teacher reports that an administrator wants to require AI scoring for ELA essays. The teacher says the AI gave 80s to 100s to essays missing major components, including essays apparently written by AI . In a meeting, staff regraded five AI-scored papers and found the AI had inflated grades and written feedback that didn't match what students wrote, such as critiquing a thesis that didn't exist. The administration still wants a department-wide approach . Another teacher says their department dropped its mandate after the tool rated strong writing 1–2 out of 4 and gave copied paragraphs a 4 . These are individual accounts, but they show that AI grading can be wrong in both directions, too generous and too harsh, which makes it hard to plan reteaching around its scores.
Teen safeguards and student self-reports
Common Sense Media ran 4,000 prompts against ChatGPT for teens and rarely triggered its protection mode. Study mode was easy to exit: students could ask for the answer or delete "@study" from their prompt . Common Sense's August survey also found that 70% of U.S. teens use AI for schoolwork. Among teens who use AI, 38% said it leads them to come up with fewer ideas of their own, and 39% said they miss out on learning when they use it to finish assignments .
Higher ed keeps scaling access
Jisc signed a framework agreement with Google that makes Google AI Pro and Workspace for Education easier to procure for UK and Irish further- and higher-education institutions. It comes with on-campus training and free certifications . As the evidence above shows, giving students access does not guarantee they will use these tools in ways that help them learn.
Worth watching
- The ED platform's 1,200-student trial report on October 29, and later results from its full-year UK trial with 15,000 students.
- Whether NYC publishes its criteria for deciding which products count as student-facing AI.
- Whether more districts copy South Bend's outcomes-based contract, especially for AI tools whose results depend on how much students use them.
- Sal Khan cautions against blanket classroom AI bans, favoring use tied to a pedagogical purpose and treating distraction as a classroom-management issue . He recommends identifying math and writing skills students must perform unaided and assessing those skills in class; he warns students without AI access may be less prepared for college or internships where AI use is expected . He allows that a one-year ban experiment may be reasonable .
- Khan Academy is developing assessments that ask students to explain their reasoning rather than select multiple-choice answers . Khan also reports early signs that supervised Socratic AI helps students who engage, while many students are uninterested or frustrated when it presses them to think .
- Khan says Khan Academy benchmarks its AI against human-edited content: he reports hallucinations are “pretty close to zero” when responses are anchored to that content and math errors occur in about 1–2% of full conversations; users are still told to verify outputs .
- The discussion presents mixed equity evidence: a panelist said the limited studies available suggested broadly similar outcomes when students had comparable resources, but also described greater harms for disadvantaged students, including more homogenized thinking and weaker long-term retention despite short-term test scores . Khan argued that AI can also help curious learners in places such as Afghanistan and rural Alaska explore subjects more deeply .
- Evidence cited by Mollick is mixed: studies from Harvard, Stanford, and Kenya suggest AI can work well as a tutor, while a Turkey study and a large-scale China study suggest using AI merely for homework help can undermine learning. His proposed direction is AI tutoring outside class and more flipped, evaluative, active learning in class; he cautions that the transition will be slow and messy.
- Wharton’s Gerald R. Lab offers free, modifiable open-source or Creative Commons teaching resources. Examples include students teaching a “naive” AI, AI simulations and tutors, AI-assisted business cases, evaluation of student work, and after-action questions; Mollick advises choosing pedagogically valuable uses rather than deploying AI for its own sake.
- Mollick argues universities may become more important because they teach and evaluate both deep expertise—needed to work with autonomous agents—and broad knowledge for spotting opportunities as fields blur; he says companies are less good at teaching these.
- The interviewer reported that law schools, especially in the U.S., had banned AI for faculty and students. Mollick said bans may be acceptable briefly while institutions work things out, but should accompany a serious effort to rethink teaching rather than substitute for it.
- AI’s effect on learning depends on how it is used: Mollick says offloading work to AI—even accepting its explanation instead of working through a problem—does not teach, while using AI as a tutor has positive effects. He says the lower brain activity reported in the MIT essay-recall study concerned recalling essays students had not written, which he considered unsurprising.
- For practice, Mollick recommends configuring AI as individualized, adaptive tutoring rather than letting it simply do homework; he cites positive results from meta-analyses and individual RCTs, including studies in Stanford and Kenya, and suggests using AI tutors outside class for flipped learning. He recommends active-learning activities such as case discussions, exercises and debates in class—usually without AI—and in-class blue-book tests for assessment.
- Mollick says AI systems can produce mathematical proofs at scale; although Lean can verify their formal correctness, researchers still do not know how to digest large volumes of proofs or what to learn from them.
- Patrick O’Neil’s forensic audit found serious problems in the 14 AI-in-education meta-analyses he examined, including poorly defined or inconsistently applied definitions of AI and learning. Of 46 randomly selected primary studies, 28 were flawed or mismatched; 12 of 14 meta-analyses failed to account for dependent effects, all 14 inadequately assessed publication bias, and some subgroup claims rested on only one or two studies. O’Neil also reported that 65.8% of AI-in-education calls for papers explicitly sought research showing benefits and 85% appeared to favor positive results, raising concerns about bias in the evidence base.
- In O’Neil’s separate, stricter meta-analysis of 44 studies and 80 effect sizes, the average effect was close to zero and not statistically distinguishable from zero, but the prediction interval for Hedges’ g ran from −0.92 to +1.03. He emphasized that this wide range reflects substantial variation: it neither establishes that AI has no effect nor supports a consistent benefit, and he considered his own analysis suspect.
- Susanna Loeb reported that an initial review found 20 compelling causal analyses among about 880 repository papers; the evidence was concentrated in higher education, with very few K–12 studies, and she recalled no U.S. student-use studies. An update had about 52 qualifying papers, but variation in tools, implementation, adult support and outcomes still made a broad meta-analysis premature.
- For practice, O’Neil advised educators to approach research aggregations skeptically, examine individual studies and treat promising approaches as tentative experiments in their own settings. Reich stressed testing hypotheses locally and gathering evidence about how students learn.
- Mollick argues universities may become more valuable by developing broad and deep knowledge and certifying learning; he expects AI tutoring outside class alongside active learning in class, while acknowledging that the best teaching approach is not yet settled.
- He warns AI is disrupting workplace apprenticeship—the path by which graduates become specialists—because people are using AI for work that traditionally taught them on the job. He urges employers to prioritize learning, even if that means some work is done without AI, or to partner more deeply with universities for formal training.
- He calls for a focused effort to build and certify effective AI tutors and make them available to every student worldwide.
- In his Wharton entrepreneurship course, Mollick uses AI simulations and case-study exercises, including having students teach a deliberately limited AI. He says students are building more and launching companies, while he has shifted assessment toward in-class assignments and discussions because AI-written work is difficult to identify; adoption varies across instructors and subjects.
- For learning, Mollick’s practical distinction is that an AI that simply gives answers is less conducive to learning than one that challenges the learner. For younger children, he recommends adult-mediated explanations and shared activities such as creating stories, programs, or games; he advises monitoring use and notes that teen and tutor modes can reduce answer-giving.
- Mollick warns that AI could erode informal workplace learning: juniors may delegate work to AI while managers use it instead of teaching interns. He argues universities can respond through teaching, assessment, verification, AI tutors, and active learning.
- He sees potential for AI tutors to support education at scale across languages and ability levels, adapt to learners’ interests, and follow their progress, while emphasizing the need for effort, testing, and safeguards.
- For younger children, Mollick advises against letting AI handle learning independently because they may shortcut it; parents can ask AI how to explain a homework problem to a child and then work through it together. For high-school students, he recommends using study mode so AI tutors rather than gives answers, describing AI tutors as effective learning tools.
- Mollick argues that learning requires productive challenge and discomfort; he says homework can improve test performance, while noting that assignments should be designed thoughtfully. He also warns that AI produces fairly similar ideas, making exposure to different ideas important.
- Higher-education adoption is uneven: Mollick says universities change slowly and that past technologies such as MOOCs did not transform education partly because improving education is difficult; faculty range from using overhead projectors to having courses fully modified with AI.
- Sal Khan argued against blanket school AI bans: educators should identify what students must demonstrate independently, such as math and writing, and assess or proctor those skills in class. He said AI can be used where it serves a pedagogical purpose, including prompting students to explain their reasoning; Khan Academy is developing assessments that use written explanations rather than multiple-choice answers.
- Khan described free online learning and AI as an opportunity for curious learners to explore subjects more deeply, but said awareness gaps can limit who benefits. He reported early signs of benefit among students engaging with Khan Academy’s supervised Socratic AI, while noting many students disengage or resist the challenge.
- Jacob Ward warned that unstructured, open-ended AI access for children can worsen test scores and retention and encourage cheating; he cited a study in which students could not quote from papers they had produced with AI, and argued for supervision.
- Khan said Khan Academy benchmarks its AI against human-created, edited content: he described the hallucination rate as “pretty close to zero” when answers are anchored to that content, and the math-error rate as about 1–2% per conversation. He said users are still warned to verify answers and should develop critical thinking that applies to any source.
- In a study of students using AI chatbots on a math-learning website, performance improved on problems students could copy into chatbots, but scores on tests taken away from computers fell below pre-AI levels—evidence that better AI-assisted task performance does not necessarily mean learning.
- Mollick recommends AI tutors that push students rather than give them answers; he cites positive learning effects in studies and meta-analyses, including RCTs at Harvard, Stanford, and in Kenya. His proposed approach is AI tutoring outside class and active, laptop-free in-class discussions, cases, and tests; examples include students teaching a deliberately weak AI or using AI to build case studies and simulations. He cautions that not every course needs to be an AI class and that prompting techniques taught the prior year may already be obsolete.
- AI may erode apprenticeship-based expertise formation: junior work and feedback from managers traditionally help people learn, but juniors may turn in AI work they cannot assess while managers may assign work to AI instead. Mollick argues that universities and companies will need to decouple efficiency from learning and deliberately preserve skill development.
- Ng said students using AI may score better on homework but worse on closed-book exams; he cited a recent University of Pennsylvania/Wharton study reporting 17% lower grades among students using AI. He warned that cognitive offloading can also undermine adult learning: AI completing work is not the same as the user learning it.
- He argued that upskilling must prepare workers for jobs beyond 2028, and that university computer science programs need to teach practical AI skills: he said students can graduate without using AI coding assistance or calling a frontier AI model API, and that he would not hire a software engineer who could not use an AI coding agent.
- Ng sees an opportunity to develop learning technologies beyond chatbots that enable one-to-one personalized, customized experiences instead of relying only on one-to-many online courses; he described this as work still ahead.
- Justin Reich says tutoring’s effectiveness is well established, but why it works remains poorly understood; dosage and implementation are clearer levers, and outcomes tend to weaken as programs scale. Existing digital tutors have their strongest record in secondary math, with Mathia showing about a 0.2-standard-deviation improvement; Reich sees the added value of generative AI in math as unclear and expects educational technology to produce incremental rather than transformative gains.
- In a Tennessee study adding Khanmigo to Khan Academy math practice, the reported gains were about the same as with the platform before AI; median students used the AI tutor after only about 17% of their mistakes. A separate one-hour study with 7,000 students found that an AI-guided review slowed students down but improved their next answer and slightly increased the chance of a correct answer a week later—a proof of concept, not evidence of large-scale impact.
- A Turkey high-school math experiment with about 1,000 students found that open ChatGPT access raised practice scores by 48% but lowered unaided exam scores by 17% versus no AI; panelists said a version designed to give hints rather than answers avoided that harm. In a separate, small 165-student trial, ED’s human-supervised, misconception-targeted AI tutor produced 66% correct answers on the next question, versus 60% after human tutoring and 56% after a static hint; 76% of AI-drafted messages were accepted unchanged, and no harmful messages reached students.
- In two randomized trials with about 350 elementary students using an AI literacy platform, students working alone averaged only 2–5 minutes of use per week against a 30-minute target. Human tutors increased story use by 70–80%, but added less than two hours of practice per student and produced no detectable reading-score gains; results also varied substantially by site.
- Andrew Ng warned that students who use AI to complete homework may earn higher homework grades but learn less and perform worse on closed-book exams; he says cognitive offloading can undermine learning, including for working adults, whose outcomes are less studied.
- To make AI support learning rather than just task completion, Ng said he leads a Coursera-funded Learn Vector team focused on AI-enabled learning and scalable learning experiences.
- Ng says many universities still graduate computer science students without experience using AI coding agents or making API calls, which he considers basic skills; he says he would not hire a candidate unfamiliar with coding agents and calls upskilling a major challenge.
- For India, he prioritizes training enough technical and nontechnical people to understand and guide AI tools for productive, responsible, and safe use.
- K–12 procurement is making evidence a condition of market access: 24 states vet early-reading curricula against approved lists, and some use ESSA evidence tiers in approvals, vendor scoring, or funding eligibility; the strictest rules exclude programs that cannot meet required tiers.
- Buyers increasingly want evidence that results will transfer to their own students and conditions, with outcomes-based contracts specifying targets, implementation, measurement, and payment consequences. South Bend’s middle-school literacy contract with Really Great Reading sets expectations for dosage, attendance, and staffing, and ties 40% of its value to students reaching proficiency on Indiana’s 2027 state test.
- AI evaluation remains unsettled: conference discussions raised whether policy will allow student use and whether AI tools work for education, while rapidly changing products can outpace traditional research cycles and complicate evidence of transferability. Evidence costs currently fall mostly on vendors, favoring larger, established companies able to fund credible studies.
- A breach at Educational Testing Service, which supports California’s statewide assessments, may have exposed student demographic, program-participation, and assessment records from the previous three school years; California is investigating, and ETS is offering affected students two years of identity protection.
- K–12 edtech consolidation continued: IXL Learning acquired NoRedInk, used by 90% of U.S. school districts, and Activate Learning acquired Open Up Resources.
Ethan Mollick contrasts AI’s reported human-work capability: 4 minutes of work at a 50% chance of success in 2024, 17 hours by the time he submitted Co-Existence, and 17 weeks by its publication. He clarifies that the 17-week figure is not AI runtime: the study involved replicating 17 weeks of human engineering work, with success rates above 50% and a token cost of about $241.
- Andrew Ng says students who use AI to do homework can earn better homework grades but learn less and perform worse on closed-book finals; he warns that this loss could compound over a long learning journey. He says emerging studies also raise concerns for adult and workplace learning, while noting that adults are less studied. He distinguishes AI’s value for completing work from helping people learn, and says his team at Learn Vector is focused on AI that supports learning and scalable learning experiences.
- Ng says many universities still graduate computer science students without experience using AI coding agents or calling current AI models through APIs, skills he considers basic now. He also shares concerns that AI-written code could leave students without fundamentals; he says his teams would not hire software engineers who cannot use coding agents, and identifies upskilling as a major challenge that Learn Vector, DeepLearning.AI, and Coursera are working to address.
- Andrew Ng warns that current AI chatbots may hinder learning: he says students using them can score better on homework but worse on closed-book exams, and cites a University of Pennsylvania/Wharton study reporting 17% lower grades among students who used AI.
- Ng says computer-science curricula are lagging: students can still earn degrees without using AI coding assistance or calling a leading frontier model’s API, while he says he would not hire a software engineer unfamiliar with an AI coding agent.
- For workforce learning, Ng says people need new skills for an AI-enabled economy and training should prepare them for jobs in 2028 and beyond, not only near-term roles.
- Andrew Ng warned that AI chatbots can help students complete homework without improving learning: he said emerging evidence shows homework scores rise while closed-book exam scores fall, and cited a University of Pennsylvania study as reporting “17% lowered grades” among students using AI.
- Ng said university curricula can lag workplace needs: students may earn computer science degrees without using AI coding assistants or calling frontier AI model APIs, while he would not hire a software engineer who cannot use an AI coding agent.
- He sees an opportunity to complement one-to-many online courses with non-chatbot tools for more one-to-one, personalized learning, while noting that this work remains ahead.
- Andrew Ng says evidence increasingly suggests AI chatbots can raise homework scores while lowering final closed-book exam scores; he cited a University of Pennsylvania study reporting 17% lower grades among students who used AI. He warns that cognitive offloading can hurt learning in K–12 and university settings, and may also undermine adult learning even when AI helps professionals complete work.
- Ng says university curricula worldwide lag current workplace practice: in many places, a computer-science degree can be completed without using AI coding assistance or calling a leading frontier AI model’s API. He says he would not hire a software engineer who cannot use an AI coding agent, and argues education systems need to change.
- For learning, Ng says conventional one-to-many online courses work and should continue, but sees an opportunity for a different generation of technology offering more personalized, one-to-one learning rather than chatbot-based experiences. He says Coursera and Udemy are working to help people gain skills, and that he is working on next-generation learning experiences.
- A recorded course’s third session, “Agent Memory: building memory-aware agents,” is taught by Richmond Alake and Nacho Martinez. It covers common memory patterns and has learners design a system for an LLM to store, retrieve, and use data to adapt and complete long-horizon tasks.
- Andrew Ng frames the course as moving beyond prompt and context engineering for single LLM calls: agents working over days or weeks need effective memory systems, an approach the course calls “memory engineering.”
- An AI Camp developer workshop used a hands-on format: developers worked through technical resources and built agents that could remember, while encountering friction from differences in their systems.
- Andrew Ng says AI can raise students’ homework scores while lowering closed-book exam performance; he cited a University of Pennsylvania/Wharton study reporting 17% lower grades among students who used AI and warned that sustained reliance could leave graduates knowing less.
- Ng warns that cognitive offloading can impair adult and professional learning too: AI helping someone finish work does not mean they learned the material. He says Coursera and Udemy are working on skills development, while DeepLearning.AI and Learn Vector are exploring personalized one-to-one learning experiences alongside conventional online courses.
- Ng says education systems are slow to update computer-science curricula: students may graduate without using AI coding assistants or making API calls to frontier AI models, despite his view that software engineers need AI coding-agent skills.
Will AI destroy jobs? Andrew Ng on the AI jobpocalypse debate | NDTV World Summit 2026
- Andrew Ng says evidence increasingly suggests AI chatbots can raise homework scores while lowering final closed-book exam scores; he cited a University of Pennsylvania study reporting 17% lower grades among students who used AI. He warns that cognitive offloading can hurt learning in K–12 and university settings, and may also undermine adult learning even when AI helps professionals complete work.
- Ng says university curricula worldwide lag current workplace practice: in many places, a computer-science degree can be completed without using AI coding assistance or calling a leading frontier AI model’s API. He says he would not hire a software engineer who cannot use an AI coding agent, and argues education systems need to change.
- For learning, Ng says conventional one-to-many online courses work and should continue, but sees an opportunity for a different generation of technology offering more personalized, one-to-one learning rather than chatbot-based experiences. He says Coursera and Udemy are working to help people gain skills, and that he is working on next-generation learning experiences.