What happens in year one when a district gives students an AI tutor? The reports that districts and researchers have published so far — a body of pilots, vendor case studies, and independent evaluations accumulated through 2025 — converge on an uncomfortable pattern: engagement rises quickly, measured learning effects stay modest and uneven, and the staffing required to keep the tool pointed at the curriculum is larger than procurement slides suggest. The company line and the classroom line remain different documents. A technology committee that reads both, and asks which one the pilot plan was written from, gets a more honest rollout.
What do the first-year reports actually measure?
Most first-year district reports are not efficacy studies, and reading them as if they were is the most common mistake in board presentations. A typical pilot reports usage: logins, sessions, questions asked, students active. Usage is easy to collect and flattering to everyone. Learning outcomes are harder — they require control groups, pre-post assessment, and a willingness to publish null results, which is why districts with well-designed evaluations often publish mixed findings while vendor case studies publish unqualified ones. The U.S. Department of Education's 2023 report on AI in teaching and learning cautioned explicitly against expecting large learning gains from early tools, and warned that engagement metrics are not outcomes. Two years of pilots since then have mostly confirmed that caution rather than overturned it.
What pattern shows up across rollouts?
Three findings repeat. First, usage concentrates: a minority of students — often those already motivated, or already behind and hungry for patient help — drive most sessions, while the middle of the distribution opens the tool when a teacher assigns it. Second, tutoring quality depends on curriculum alignment: off-the-shelf chat tutors answer grade-appropriate questions with varying accuracy, and districts that fed the tools their own course materials report fewer hallucinated lessons than districts using default configurations. Third, teacher framing decides usage: classrooms where teachers assigned specific practice inside the tool saw sustained use, while classrooms where the tool was simply available saw usage decay over weeks. None of these findings is hidden; all of them are quieter than the launch announcement.
What costs surprise districts in year one?
The license is the visible cost and rarely the decisive one. The surprises are integration labor — rostering the tool through the district's class-information system, and maintaining it when schedules change — plus the review workload of checking what the tutor tells students, plus helpdesk load that spikes whenever the vendor ships a model update mid-semester. Districts using interoperability standards such as LTI and One Roster report meaningfully lower integration effort than those handling CSV exports by hand, a difference that compounds at every term break. Budget hearings that discuss only per-seat prices are therefore discussing the smallest line.
How should a district structure its own first-year report?
Decide the evaluation before the purchase order, not after. A defensible first-year report has four elements: a usage picture broken out by school and student group, so concentration is visible; a learning measure tied to existing district assessments rather than the vendor's internal dashboards; an equity check on which students benefit; and a cost accounting that includes staff time, not only licenses. Districts that publish this internally — even a short memo to the board — enter renewal negotiations with evidence. Districts that do not enter renewal with the vendor's slide deck, which is the only document in the room with numbers in it.
What do the rollouts say about equity?
The equity question is where vendor narratives and district data diverge most sharply. The appealing story is that an always-available, infinitely patient tutor helps students who cannot afford private tutoring most — and some first-year reports partially support it, with struggling students asking questions at night and on weekends that they would never raise in class. But the same reports show the opposite risk: students with the least reading stamina can accept a plausible wrong answer without checking, and students with strong support at home get more out of the tool because adults help them use it well. Device and connectivity gaps reappear at home in districts without one-to-one take-home programs, meaning the tool is least available exactly where the need is argued to be greatest. Districts that disaggregated usage by student group found both patterns in the same dataset. The practical response is not to abandon the tool but to design for the gap — structured in-class time with the tutor for the students least likely to use it voluntarily, and spot-check habits that teach verification rather than answer-copying.
What happens at renewal?
Year one ends with a procurement decision dressed as an evaluation. Districts that pre-committed to measures walk into renewal with a bargaining position: real usage distributions, real cost figures, and at least one outcome measure the vendor cannot contradict. Districts that did not face the standard renewal conversation, in which the only new information since signing is the vendor's own usage dashboard. Procurement staff report that multi-year AI licenses are where leverage matters most, because model quality changes quarter to quarter and a three-year lock-in priced on this year's model is a bet on someone else's roadmap. A one-year renewal with a measured decision each spring costs more per seat and buys the only thing that matters in a fast-moving market: the right to be wrong for twelve months instead of thirty-six.
What questions cut through vendor claims?
Ask what the tool does when it is wrong, and who sees that. Ask for the district-level usage data — not averages, distributions. Ask which studies the vendor cites, who funded them, and whether any found no effect. Ask what changes when the vendor swaps the underlying model, and whether the district is notified. And ask the teachers who used it least, not only the volunteers who present at the launch event. First-year reports from real rollouts suggest the honest summary of year one is usually: promising, uneven, dependent on adults, and worth continuing only where the district measured something it actually values.
For more context, read What actually changes when a free edtech tool goes paid.
For more context, read What FERPA actually lets edtech vendors do with student data.