Distillation copies behavior. It does not copy responsibility. An open student can look like a closed teacher on a demo and still fail the trace you care about.
Understanding the parity claim
Closed APIs stayed ahead for years because pretraining plus post-training was a capital project. That lead is narrower now. Two things moved at once. Labs released stronger open weights, including Xiaomi’s MiMo-V2.6 Pro in the current open wave. Research on on-policy, token-level alignment distillation showed you can transfer a teacher’s preferences without replaying the teacher’s entire pretraining budget.
“On-policy” means the student learns on states it produces, not only on a frozen dataset of the teacher’s greatest hits. “Token-level” means the supervision is finer than a single thumbs-up on a whole answer. Together they are why people can talk about frontier-tier reasoning on a smaller post-training bill. They are not why you should skip an eval.
I will not invent a score for MiMo or for any student model here. If a vendor or a paper claims parity, ask for the task list, the contamination story, and the tool-use setup. A chat arena win is not a retrieval win.
The engineering framework
Use distillation as a procurement option with three artifacts.
Teacher traces you are allowed to learn from. Terms of service on many APIs forbid using outputs to train a competing model. If the teacher is a commercial API, read the contract before you build a pipeline. Public research sets and open teachers are the clean path.
Student you can freeze. Open weights you host are a version you can pin, scan, and re-run. That is the operational win: no silent upgrade the week of a launch. It is also the operational cost: GPUs, patching, and an owner.
A harness that can say no. The same promotion rule as in the pacing debate: your failures must get better, and your refusals must not get worse. A distilled model that is “as smart” and more willing to leak is a regression.
Where this shows up in products I actually ship is the model behind a RAG agent or a router, not the marketing model. Retrieval, citations, and tool ACLs dominate answer quality once the base model is “good enough.” That stack is what I mean by production RAG for California and Florida teams: the model is swappable, the trace is not.
How to score parity without a leaderboard
Pick twenty traces from production, not from a demo set. Include the boring ones: a tool call that must match a schema, a bilingual Florida caller, a long California policy excerpt that must be quoted rather than paraphrased, and a case that must refuse. Run the closed teacher and the open student on the same frozen traces. Score three things only: task success, schema validity, and refusal correctness.
Then score cost per success, including retries and your GPU hour. A student that is cheaper per token and fails twice as often is a more expensive system. Publish the table internally with model IDs and dates. “Feels as smart” is how distillation projects skip the only measurement that justifies leaving the API.
If the student wins on your twenty traces for two weeks, pin it and keep the teacher as a shadow on a sample. If it loses on refusals, do not average that away against a win on chit-chat. Refusals are the parity that matters when the user is not you.
What on-policy, token-level distillation is asking for
Distillation here is not “ask a big model to write a textbook and train on the textbook.” On-policy means the student generates (or the teacher and student share) traces on the tasks you care about, and the training signal is applied at the token, not only at the final answer. The hope is that the student picks up the teacher’s next-step habits — when to call a tool, when to stop, when to refuse — without repeating the teacher’s pretraining budget.
That hope is why releases like Xiaomi’s open MiMo-V2.6 Pro are being read as more than a weights drop. The interesting claim is not a mystery score. It is that frontier-style reasoning is showing up in open weights with a smaller pretraining and post-training bill than the closed labs spent to get there first. I will not invent a number for how close “showing up” is. Closeness is a table you compute.
There is a legal edge the engineering posts skip. Many commercial APIs forbid training on their outputs. If your teacher is a closed API, read the contract before you treat its traces as a dataset. A research paper that distilled from outputs you are not allowed to train on is not a plan you should copy into a company repo. Use traces you have the right to use. Prefer a teacher whose license matches the student you want to ship.
A scorecard that fits on one page
Freeze twenty production traces. Include the boring ones on purpose: a tool call that must match a schema, a bilingual Florida caller, a long California policy excerpt that must be quoted rather than smoothed, and a case that must refuse. Run the closed teacher and the open student on that frozen set. Score task success, schema validity, and refusal correctness. Then score cost per success, including retries and the GPU hour, not the sticker price per token.
A student that is cheaper per token and fails twice as often is a more expensive system. Publish the table internally with model ids and dates. “Feels as smart” is how these projects skip the only measurement that justifies leaving the API. If the student wins on your twenty traces for two weeks, pin it and keep the teacher as a shadow on a sample. If it loses on refusals, do not average that away against a win on chit-chat. Refusals are the parity that matters when the user is not you.
I would rather run that table against a retrieval stack you control than against a public leaderboard. Your documents, your tools, your refusal policy. A model that wins a general exam and drops a citation is not at parity for the product.
What you still have to pay for after the weights are yours
Owning the weights moves the bill. It does not delete it. You pay for serving, for the eval set, for the person who notices the student drifted, and for the fallback when it refuses too often or not often enough. You also pay attention. An open model can be fine-tuned by someone else into a behavior you did not buy. Pin a hash. Record the hash in the same place you record the prompt version.
Pacing, covered in the frontier safety piece, is the closed lab’s version of the same discipline: do not let a new capability become the default just because it exists. Distillation is the open lab’s version: do not let a new weight file become production just because it is downloadable. Both reduce to a promotion rule. The rule is the product. The model is the thing the rule allows through.
Key attributes and checklist
- Write the tasks that define parity for you: tool calls, bilingual Florida workflows, long-context California policy docs, refusal cases.
- Freeze teacher and student IDs. Re-run when either changes.
- Check the license and the API terms before any training run.
- Budget hosting, not just “the weights were free.”
- Compare cost per successful task, not cost per token. A cheaper model that retries three times is not cheaper.
- Test abstention. Distillation can copy confidence without copying calibration.
- Keep the closed API as a fallback until the harness has a month of green runs.
A distillation project that stays small on purpose
Do not start by distilling “everything the teacher knows.” Start with one workflow: the tool-calling router, the refusal policy, or the citation answer. Collect the twenty traces. Confirm you are allowed to train on them. Train or adapt the student. Put the student behind the same schema check the teacher had to pass. Shadow it.
The failure mode I see is a team that downloads a new open weight file, runs three flattering prompts, and announces parity. The file can be excellent and the announcement can still be false for your workflow. MiMo-V2.6 Pro and the papers on token-level alignment are reasons to run the table, not reasons to skip it. If your table is not built, you do not have a result. You have a download.
Keep the teacher available for the slice the student loses. That is not a defeat. It is how you get the cost down on the eighty percent of turns the student handles and keep the closed model on the turns where a miss is expensive. Report the split. “We moved to open weights” is less honest than “we moved these routes, and these routes still call the teacher.”
Serving is part of parity. A weight file that matches the teacher and cannot meet your latency on the hardware you have is not a replacement. Measure the wall clock of a successful task, including retries. Then decide. The decision belongs in the same promotion log you would use for a closed-model jump, because it is the same kind of change: a new system, wearing a familiar API.
Frequently asked questions
What is on-policy token-level distillation?
The student generates its own trajectories, and the teacher’s token-level preferences on those trajectories update the student. The student learns the teacher’s choices on states it will actually visit.
Does an open model matching a demo mean frontier parity?
No. Parity is task-specific. A model can match a leaderboard slice and miss your tools, your languages, or your refusal behavior.
Why do teams still pay for closed APIs?
Support, data-handling terms, tool reliability, and a vendor who patches breakages. Owning weights removes one dependency and adds MLOps you must staff.
If you want the model choice made against your traces instead of a leaderboard, describe the tasks and the data boundary. Remote delivery, California and Florida overlap, no invented US office.