Model Distillation: Teacher to Student for Production LLMs

Use model distillation when a small specialist can handle your repeated task, and keep the frontier model for cases it cannot. Training availability now matters as much as model choice: OpenAI's eligible existing fine-tuning customers lose new-job access on 6 January 2027, unless their base model shuts down earlier. OpenAI's availability timeline

The strongest candidate is a large language model (LLM) feature with a stable contract: classify this request, extract these fields, choose this route, or return this exact format. Start by testing a smaller model without training. Distillation earns its place when the larger model handles important cases that the smaller baseline misses, and you can turn that difference into reliable training examples.

Provider documentation and output-training terms verified on 26 October 2026. The distinctions below concern production APIs and named model licenses; a consumer chatbot subscription or a different hosting contract can have different rules.

How does model distillation work?

A large teacher model produces labels or demonstrations, and a smaller student model learns to perform the task from those examples. Once trained, the student can answer routine requests without asking the teacher on every call. In supervised fine-tuning, an existing model trains on example inputs and desired outputs. A managed distillation workflow can supply those outputs from the teacher. Documented distillation workflow

Think of a climbing instructor teaching a short route. The apprentice practices the moves needed for that route; they do not inherit the instructor's ability to climb every mountain. A production student has the same boundary: learning your refund classifier does not make it a general substitute for the teacher.

For example, imagine a support feature that assigns messages to billing, technical, cancellation, or needs_review. The teacher reads representative messages and proposes labels. A reviewer resolves ambiguous cases. The student trains on the approved message-label pairs, then faces unseen messages whose answers were checked independently.

That last step matters because agreement with the teacher is not the same as correctness. If the teacher routinely misreads a cancellation request as a billing question, copying it faithfully preserves the mistake.

LLM knowledge distillation through output examples

For an application builder, the practical starting point is usually output-based distillation: save the inputs and acceptable teacher responses, then train the student on those pairs. You do not need access to the teacher's weights, the learned numerical parameters inside the model, for this documented workflow. You do need permission to reuse the outputs and a student training route that supports the task. OpenAI's supervised fine-tuning guide

Distillation is broader than copying text. The foundational research also describes transferring knowledge from an ensemble, a group of models, into a single model that is easier to deploy. For a narrow production feature, however, the useful question is whether the student learns the decisions you need, not whether you reproduce every part of the teacher. Distilling the Knowledge in a Neural Network

These approaches can work together. LLM quantization can follow training, with another evaluation afterward. RAG vs fine-tuning explains why learning a response pattern does not replace retrieving changing company facts.

When AI model distillation pays, and when it does not

Distillation is most attractive when the task stays narrow, the traffic repeats, and serving the student improves a constraint you can measure. High volume helps spread the training and review effort across more requests. A latency requirement can also justify the work even when spending is not the main concern.

A document-intake service is a plausible candidate if it repeatedly extracts the same fields from familiar document families. An assistant expected to answer arbitrary questions, navigate unfamiliar tools and reason across new domains has a much wider job. Shrinking that whole assistant is a harder project than specializing its intent classifier.

For builders: freeze the input, output and failure behavior before collecting data. If an invoice has no purchase-order number, the correct extraction may be null, not a plausible-looking guess. Training examples must teach that distinction.

For operators: compare the complete serving path. Include retries, validation failures, human review, requests escalated to the teacher, ongoing evaluation and retraining. Lower inference resource use is useful only if those obligations do not erase the benefit. There is no universal traffic threshold that makes distillation worthwhile.

For buyers: verify that the required teacher, student, training method and deployment location are available together. A provider offering both models for ordinary inference does not prove it supports distilling one into the other.

Privacy can be a reason to pursue a smaller, appropriately hosted student, but small does not mean private. Trace where raw inputs, teacher calls, training files, logs and the deployed model live. A locally served student does not undo sensitive data already sent to a remote teacher during dataset creation.

Wait when the task definition changes frequently, the available examples are unrepresentative, the teacher's labels are unreliable, or the training rights are unresolved. Skip training when a smaller untuned model already passes. Google explicitly notes smaller distillation gains when the student is already close to the teacher on the target task. Google's distillation use-case guidance

Managed routes available today

Choose a managed route by its current training support and lifecycle, then check the terms for that route. The following comparison reflects the documentation verified on 26 October 2026.

OpenAI model distillation: an existing-user route with an end date

OpenAI's documented workflow is to improve the teacher prompt, capture responses that meet your criteria, build a dataset and fine-tune a smaller model. That remains a useful description of the mechanism. It is no longer a general onboarding recommendation: the current guide says the fine-tuning platform is unavailable to new users. OpenAI supervised fine-tuning

Eligible active customers can create jobs only until 6 January 2027, subject to earlier model shutdowns. GPT-4.1 and GPT-4.1 mini, including their fine-tuned versions, shut down on 14 January 2027. The documented GPT-4.1 nano and o4-mini shutdown occurred on 23 October 2026. These deadlines make a new long-lived dependency difficult to justify even if your account still has access. OpenAI deprecations

The practical call: existing users should plan migration and preserve their evaluation data. A new project should select an available training route rather than building around the historical distillation launch.

Google Vertex AI: distinguish Gemini tuning from open-model distillation

Google Vertex AI's generative-model documentation now resolves under Gemini Enterprise Agent Platform. Its Gemini tuning guide describes supervised fine-tuning, training from labeled examples, and lists models including Gemini 3.5 Flash and Gemini 3.5 Flash-Lite. It recommends this method for classification, sentiment analysis and entity extraction. Google's current tuning overview

Google separately documents managed open-model distillation through the GenAI SDK. Examples on its teacher list include DeepSeek R1 0528 and DeepSeek V3.2; students include Qwen 3 8B and Gemma 3 4B IT. Open-model supervised tuning also offers full fine-tuning and LoRA, which updates fewer parameters. Its model list differs from distillation eligibility. Open-model tuning documentation

The wall is the method, model and location combination. Teacher inference needs quota alongside training. The global endpoint chooses a supported region, so residency requirements need deliberate regional configuration. For simple classification, test the smaller model before committing to Google's reasoning-focused distillation route.

Amazon Bedrock model distillation: a managed workflow with fixed pairings

Amazon Bedrock Model Distillation combines teacher response generation, data synthesis and student fine-tuning in one workflow. You can supply prompts or use existing model invocation logs, records of previous requests and responses, stored in Amazon Simple Storage Service (Amazon S3). Bedrock also offers supervised and reinforcement fine-tuning as separate customization methods. Distillation workflow, customization methods

The current pair table includes Amazon Nova Pro → Nova Lite or Nova Micro in US East (N. Virginia), and Meta Llama 3.3 70B → Llama 3.1 8B among the Llama options in US West (Oregon). It explicitly says distillation is not currently available for Anthropic models, with no confirmed restoration timeline. Supported pairs and regions

Bedrock's data-preparation guide caps supplied input at 15K prompts or prompt-response pairs. That is a job limit, not evidence that your task needs that many examples. Select data for coverage of real decisions and failure cases. Dataset preparation limits

This is a practical route if your desired pair is supported and your deployment requirements fit. It is not an arbitrary teacher-to-arbitrary-student service.

The terms trap: owning outputs is not training permission

Check the contract for the endpoint that produced the data before using those outputs to train another model. Output ownership, the provider's promise not to train on your data, and your right to train a student are different provisions.

The excerpts below are deliberately short. The accompanying scope matters as much as the quoted words; your signed agreement, order form and applicable product terms control the actual use.

OpenAI: a competition restriction with named exceptions

OpenAI's Services Agreement restricts using outputs to develop models that “compete with OpenAI’s products and services”, except for a Permitted Exception. That definition includes internal models primarily intended to categorize, classify or organize data, provided those models are not distributed or made commercially available to third parties. It also includes fine-tuning or customizing models supplied through the specified OpenAI services. Services Agreement, restriction and definitions

That makes an internal classifier materially different from a general generative model offered to customers. Do not assume every extraction feature, hosted classifier or private general-purpose model fits the exception; have the person responsible for your contract confirm the actual use.

There is also a timing distinction. OpenAI has posted a policy update effective 20 November 2026, after this article's verification date. It addresses using models or outputs to supply training signals for alternatives or substitutes, including labels, feedback and filtering, while preserving uses expressly permitted by the applicable agreement. Treat it as an upcoming change, not a rule already effective on the verification date. Posted Usage Policies update

Anthropic: competing-model training requires express approval

Anthropic's Commercial Terms prohibit accessing its services to build a competing product or service, including to “train competing AI models”, except as expressly approved. The separate assignment of output rights does not remove that restriction. Anthropic Commercial Terms, D.4

Claude outputs therefore need a specific contract analysis before becoming another model's training set. An older cloud distillation announcement is not sufficient evidence of current availability or permission.

Google: Cloud and the Gemini developer API are separate contracts

Google Cloud's model restrictions prohibit using service outputs to “substitute, replace, or circumvent the use of a Google Model” or “create or improve models similar to a Google Model”. An explicit exception applies where the service offers fine-tuning or distillation: its outputs may be used to modify and use Modified Google Models instead of Google Pre-Trained Models. Separate offerings carry separate terms. Google Cloud Service Specific Terms

The Gemini developer API has different wording: you may not use its services to “develop models that compete with the Services”. Those additional terms explicitly distinguish direct Google Cloud Platform use. Permission in a managed Cloud workflow should not be copied over to an unrelated Gemini API data-generation plan. Gemini API Additional Terms

Amazon: service permission is the exception

AWS Service Terms start the relevant restriction with “Except where permitted by the AI Service” and otherwise bar generating content for “the express purpose of training an AI model or service”, or developing a substantially similar one. Bedrock also incorporates applicable third-party model terms. AWS Service Terms, AI Services and Bedrock

Use the documented managed workflow and its eligible pairings as the concrete route to investigate. Do not read its existence as blanket permission to export every Bedrock model's responses into any external training job.

Meta Llama: model naming and redistribution conditions matter

Meta's Llama 3.3 license explicitly addresses training another model with Llama materials or outputs. When that resulting model is distributed or made available, it requires Llama “at the beginning of any such AI model name”. License, attribution and other applicable conditions also matter. Llama 3.3 Community License

This is a named model license, not a universal statement about every Meta model or every hosted endpoint. Record the exact release and hosting terms alongside the dataset.

Mistral AI: distinguish text outputs from image-specific restrictions

Mistral AI's current commercial output restriction names “image Outputs” used to “develop or train any image generation product that competes”. That clause is image-specific; presenting it as a blanket ban on training with text would overstate it. The same terms separately prohibit using outputs to reverse engineer Mistral AI products, and additional product or deployment terms can apply. Mistral AI Commercial Terms

For an LLM dataset, review the text use under the applicable contract instead of importing either an image restriction or an assumption of unlimited permission.

Grok: enterprise output training needs an order-form exception

Grok's enterprise terms at x.ai prohibit using “any Output to train any foundation models, large language models, or other artificial intelligence systems”, unless expressly permitted in an Order Form. The page identifies its contracting provider as SpaceXAI LLC, using the name SpaceXSI. Grok Enterprise Terms, rights in input and output

This wording is broader than a competing-model-only restriction. Do not assume an internal student is exempt simply because it stays inside your company.

DeepSeek-R1: an explicit distillation allowance for a named release

DeepSeek-R1's official repository states that its code and weights use the MIT License and expressly allows “distillation for training other LLMs”. It also identifies the originating Qwen and Llama licenses for the already distilled releases. DeepSeek-R1 license section

That is useful permission tied to a specific release. It does not establish the terms of every future DeepSeek model, hosting service or student base model. Keep both the teacher permission and the student's license in the release record.

Evaluate before switching production traffic

A student is ready when it meets the feature's acceptance criteria on unseen production-like inputs and survives a controlled rollout. A falling training loss, the error measured during training, does not prove either condition.

The following is a proposed release procedure, not a report of benchmark results.

1. 1

   Freeze the task and the baseline

   Write the input contract, permitted output values, required fields and abstention behavior. Save the current teacher's results and a prompted small-model baseline on a representative sample. Define acceptable quality and latency before examining the trained student's score.

2. 2

   Separate training, validation and final testing

   Use training data to update weights and validation data to choose prompts or checkpoints. Keep a final test set untouched until those choices are complete. Group related customer conversations, document templates and near-duplicates so copies cannot leak across splits. Include recent real traffic, rare classes and difficult examples.

3. 3

   Curate teacher demonstrations

   Have the authorized teacher label the training inputs. Validate the format, review ambiguity and correct important mistakes. Preserve examples where the right response is an abstention or a missing value. Record the teacher model, prompt, source permissions and dataset version. The synthetic-data-generation guide covers this data layer in more depth.

4. 4

   Score the business task, not resemblance

   For classification, measure errors for each class, especially recall for costly misses: how many true cancellation requests were recognized? For extraction, check the exact required field values and missing-field handling. For fixed-format generation, check schema validity, whether the output follows the required structure, separately from factual correctness. For routing, measure whether the chosen downstream path resolves the request. Audit teacher-student disagreements against independently checked answers.

5. 5

   Run shadow traffic, then a canary

   In shadow mode, copy live inputs to the student while the existing system still supplies the user-facing result. Measure response quality, slow requests, timeouts and escalation needs under actual load. If it passes, a canary exposes only a controlled portion of traffic to the student, with explicit stop conditions and a working rollback.

6. 6

   Keep an escape route and monitor drift

   Route unsupported inputs, validation failures and defined high-risk cases back to the teacher or human review. Monitor changes in input mix, per-class errors and the share of requests that need fallback. Re-evaluate after changing prompts, schemas, student checkpoints, quantization or upstream data. A fallback that is frequently used is evidence to revisit the design.

A practical extraction release gate might require that the student preserve required-field accuracy on each important document family, stay within the existing service's latency objective and abstain correctly when information is absent. Your owner sets the thresholds from the consequences of errors; a vendor's general benchmark cannot set them for you.

Use the model-router guide to design the boundary between student traffic and frontier-model fallback. Do not rely solely on a student's self-reported confidence. Input eligibility rules, output validation and measured failure patterns give the router something concrete to act on.

What is overhyped about distillation

“The small model becomes the large model.” It learns from the supervision and coverage you provide. Success on familiar extraction patterns says little about an unfamiliar language, a new document family or a broader reasoning task.

“Generated data removes the labeling problem.” It moves part of the work from writing labels to checking them. A teacher can generate a large collection of consistently wrong examples. Human adjudication and independent test answers remain valuable.

“A smaller model guarantees savings.” The deployed feature includes fallback, review, hosting, monitoring and retraining. Judge the full outcome, and treat latency improvements as measurements to establish rather than promises attached to model size.

“Managed distillation settles the rights question.” It supplies a documented workflow with specific boundaries. It does not turn all model outputs into unrestricted training material.

“Valid JSON means the feature works.” A perfectly formatted wrong account ID is still wrong. Format checks are one release gate, not the whole evaluation.

Model distillation FAQ

Does model distillation actually work?

It can transfer useful task behavior from a teacher to a student. The evidence that matters for your feature is performance on unseen, independently checked inputs, followed by a controlled production rollout. It is not a guarantee of general teacher-level ability.

What is the difference between model distillation and fine-tuning?

Distillation describes learning from another model's supervision. Fine-tuning describes updating an already trained model with task data. When teacher responses become supervised student examples, the same job is both distillation and fine-tuning.

How do I distill an LLM model?

Define a narrow task, confirm the teacher-output and student-model rights, curate representative teacher demonstrations, train through a supported route, and compare the student with the teacher and an untuned small-model baseline. Use held-out evaluation, shadow traffic and a canary before a broader switch.

What is a distilled model?

It is a model trained using supervision from another model, often to make a smaller model effective on a particular task. Its competence still depends on the student, the training data and what you evaluate.

What is model distillation?

Model distillation transfers learned behavior from a teacher model to a student through training. In a common LLM workflow, the teacher produces labels or answers and the student learns from approved input-output pairs.

What are the four types of distillation?

There is no universal four-type taxonomy for LLM model distillation. Lists with that wording often concern separating liquids, a different meaning of distillation. For an LLM feature, identify the actual supervision and training method rather than choosing from a chemistry classification.

What are 5 examples of distillation?

For LLM applications, illustrative candidate tasks are support-ticket routing, intent tagging, invoice-field extraction, record normalization and fixed-format response generation. These are possible uses to evaluate, not claims that a distilled model automatically succeeds at all five.

What are the disadvantages of distillation?

You can inherit teacher errors, miss rare inputs, spend heavily on data review, and need retraining as the task changes. Provider lifecycle, teacher-output restrictions and the student license can also constrain deployment.

What is a simple distillation technique?

For LLMs, use an authorized teacher to produce responses for representative inputs, review those responses, and supervised-fine-tune a student on the approved pairs. Test it on separate real cases. This is distinct from the laboratory meaning of simple distillation.

Your next build step

Pick the owner of one repeated feature next week. Have them write its output contract, assemble a held-out sample from real requests and compare the current teacher with a prompted small model. If a meaningful quality gap remains, confirm an available, authorized teacher-to-student route and build the training set around the failures.

If the smaller baseline already passes, take that simpler win. If neither the teacher nor your reviewers can label the difficult cases consistently, resolve the task definition before training.

Subscribe to the newsletter for practical decisions about building and operating AI features.