Synthetic Data Generation: LLM Training and Test Data

Provider documentation and output-use terms verified on 22 October 2026. The support-ticket examples and pilot sizes below are an authored illustration, not a reported model experiment.

What Is Synthetic Data Generation?

Synthetic data generation creates artificial records or examples for a defined purpose. For a builder using a large language model, or LLM, that usually means generating text such as support requests, question-answer pairs, extraction examples or conversations from a specification and a small set of approved examples.

Think of a driving simulator. You can rehearse a difficult junction repeatedly without waiting for the same situation on the road. Passing the simulator still does not establish how often that junction appears in traffic, or prove that you can handle every road. Synthetic datasets offer the same useful separation between practice coverage and evidence about the world.

The method matters. A rule-based generator follows instructions such as keeping an order total equal to its line items. A tabular synthesizer learns patterns across columns. An LLM generates language conditioned on a prompt and supplied context. You can combine them: code creates a valid order, and an LLM writes the customer's complaint about that order.

Synthetic data is also different from masking a record. Replacing a customer's name leaves the rest of their story intact. Augmentation changes an existing example, such as paraphrasing a question; its descendants remain related to the source. Neither operation automatically makes a record private or creates independent evidence.

Evals: Create Challenges With Independently Checked Answers

For an application builder, synthetic evals are most useful for known failure modes: missing information, conflicting instructions, unsupported requests and unusual wording. An eval is a repeatable test of your system's behavior.

If a support assistant must ask for clarification when an account issue is ambiguous, generate ambiguous requests deliberately. Have someone verify the expected action against the policy. Do not let the generator's answer become the answer key merely because it sounds confident.

Keep a separate evaluation set drawn from permitted real traffic where available. If you cannot use real traffic, commission independently authored cases from someone who knows the task. That can check the specification; it cannot establish accuracy across your production population.

Fine-Tuning: Teach a Specific Behavior

For the builder adapting a model, synthetic examples can demonstrate a format, classification rule or response style that is underrepresented in the available data. Fine-tuning means updating a model through training on examples.

Start from an observable failure: the model keeps routing ambiguous tickets to billing. Add reviewed ambiguity cases and compare the adapted model with the existing baseline on untouched examples. A larger training file is not itself a success metric.

Use the fine-tuning workflow once you have a stable task and acceptance set. If the problem is missing or changing factual context, fix the context supplied to the application before trying to teach facts through generated answers.

Synthetic Test Data Generation

For a software builder, synthetic test data generation should make the application exercise its rules: missing fields, duplicate submissions, unusual text lengths, invalid state transitions and related database records.

A checkout test needs valid relationships between orders, customers and payments. It does not need a language model to invent arithmetic. Build those constraints in code, then use an LLM where realistic language adds value, such as a delivery note that contradicts the shipping address.

An operator buying a dataset should ask which of these jobs it serves. A dataset suited to a demo is not necessarily useful for training, and a training corpus is not automatically an evaluation set.

A Worked Recipe: From Seed Examples to Reviewed Candidates

Start with a narrow task whose correct answers you can explain without consulting the generator. This recipe builds candidate examples for routing fictional support tickets to billing, access or human_review.

The fictional policy is simple: payment and invoice issues go to billing; sign-in and password issues go to access; unclear requests or requests combining both go to human review. Do not infer facts the customer did not supply.

Synthetic Data Examples: Six Seeds With Explicit Labels

Use these authored seeds to show the boundary:

• “My card was charged twice.” → billing
• “This invoice lists the wrong plan.” → billing
• “The reset link has expired.” → access
• “I cannot sign in after resetting my password.” → access
• “I paid, but I cannot sign in.” → human_review
• “It still does not work.” → human_review

The last examples matter as much as the obvious ones. They teach the generator that uncertainty is a valid outcome. The few-shot prompting guide explains how examples communicate rules that a general instruction can leave unclear.

A cleaned candidate might look like this. It is an illustrative record, not captured API output:

{
  "case_id": "billing-frustrated-indirect-a",
  "text": "Another invoice with the wrong plan, seriously?",
  "proposed_label": "billing",
  "policy_reason": "The customer identifies an invoice problem."
}

The important result is a traceable, reviewed case. Whether the initial generator returned all requested candidates is secondary. Never refill a weak cell simply to hit the target count.

Synthetic Data Generation Tools: What Their Docs Promise

Choose tools by the part of the process they control. A model API is the interface your code uses to request model outputs; its batch service schedules those requests. A dataset library organizes generation and feedback. A tabular synthesizer learns relationships in structured data. None of those responsibilities replaces your acceptance criteria.

Generative AI for Synthetic Data Generation: The Batch APIs

OpenAI API provides Batch for asynchronous requests submitted in a JSONL file, meaning one JSON object per line. Its docs require a unique custom_id and warn that output order may differ from input order. Match results by identifier, then inspect failures and incomplete generations before parsing candidates. OpenAI Batch documentation.

Anthropic's Claude API provides Message Batches for independent Messages requests. Results distinguish successful, errored, canceled and expired requests, and custom_id connects each result to its input. Its wall is operational: an accepted batch is not proof that every example completed. Retry missing work by identifier rather than regenerating the whole dataset. Claude batch documentation.

Google's Gemini API provides Batch with either inline request objects or an uploaded JSONL file. File-based requests carry a key that identifies the corresponding result. Use the file route for larger offline datasets; the documented turnaround is a target, so avoid making an interactive workflow depend on immediate completion. Gemini Batch documentation.

These are the batch limits that affect job splitting, verified in the linked docs. They are not limits on the total dataset you can build across multiple jobs.

A batch request and a candidate example are different units. If each successful request returns several candidates, track both counts. Record parse failures, refusals and truncated output alongside transport errors. A successful HTTP response alone does not make a usable training row.

Distilabel: Organize Generation and Feedback

Distilabel is a framework for synthetic data and AI feedback that composes generation, evaluation and data-handling steps into pipelines. Its documentation describes classification, extraction, dialogue and instruction-generation use cases. That is useful when a one-off script has grown into several generation and filtering stages. Distilabel documentation.

The project uses the Apache-2.0 license. Its repository also says the original authors moved on and community collaborators are maintaining it. Pin and validate the version you adopt. The practical wall remains the evaluator: orchestrating a model's judgment does not make that judgment independent or correct. Distilabel repository.

Faker: Make Fixtures Without a Language Model

Faker is a Python package that generates fake data through providers, with locale support and random seeding. It is useful for creating application fixtures when you need controlled fields rather than natural-language reasoning. Its repository identifies the license as MIT. Faker documentation, Faker repository.

You still need to encode business relationships yourself. Independently generated addresses, dates and transactions do not establish a realistic customer history. Use reserved test identifiers and domains where possible, and prevent generated fixtures from triggering messages or other production actions.

Synthetic Data Vault: Model Tables and Their Relationships

Synthetic Data Vault, or SDV, provides tabular synthesis and statistical evaluation, with documented approaches for single tables, sequential data and related tables. Its trained synthesizers learn from data you supply; they do not remove the need for a permitted source dataset. SDV documentation.

Within SDV, GaussianCopulaSynthesizer uses classical statistical methods to learn column distributions and relationships. CTGANSynthesizer uses a generative adversarial network, a model trained through a generator and a competing discriminator. Both take data and metadata describing its structure. These are tabular synthesizers, not general-purpose chat models. Gaussian Copula docs, CTGAN docs.

The wall is fidelity to the task. Similar-looking column distributions do not prove that rare combinations, cross-table rules or a downstream model's behavior are preserved. Validate the relationships your application relies on, including foreign keys linking child records to parents.

Choose a Synthetic Data Generator by the Missing Evidence

Use a model API when the missing ingredient is language variation. Add Distilabel when repeatable generation and feedback orchestration earns the extra dependency. Choose Faker when fixtures need controlled values. Consider SDV when statistical relationships across tables matter and you have source data you are allowed to model.

Do not buy a larger generator to solve an unclear labeling policy. That problem sits upstream of every tool here.

What Provider Terms Say About Training on Outputs

Check output-use restrictions before selecting a teacher model, meaning the model generating examples for another model to learn from. The following lines summarize the standard terms for the three hosted LLM providers above; they do not declare that a particular training project qualifies.

• OpenAI: Its Services Agreement restricts using outputs to develop competing AI models, with stated exceptions for models primarily categorizing, classifying or organizing data that are not distributed or made commercially available to third parties, and for customizing models through OpenAI's specified services. Services Agreement.
• Anthropic: Its Commercial Terms prohibit accessing the services to build competing products or services, including training competing AI models, unless Anthropic expressly approves. Commercial Terms.
• Google: Gemini API's Additional Terms prohibit using the services to develop competing models; their Google Search Grounding provisions separately prohibit training on Grounded Results or Search Suggestions. Gemini API terms.

A library's software license is a separate question from the terms of the model it calls, the license of any model weights, and your rights to the source examples. Output ownership alone does not settle all of those questions.

Privacy-Safe Test Data Needs Its Own Check

Synthetic origin is not a privacy guarantee. If you copy confidential records into an unapproved generation service, generating fictional-looking replacements afterward does not undo that disclosure.

Stadler, Oprisanu and Troncoso's research on synthetic-data publishing found that privacy gains and usefulness varied across the models they evaluated. The practical lesson is to assess the released data and the generation process, rather than treating the word “synthetic” as an anonymity certificate. USENIX research paper.

Prefer fictional seeds written from an approved schema and policy when that is enough for software testing. When preserving real statistical patterns is necessary, keep source handling in an approved environment and evaluate what information the generated dataset can reveal.

Check direct identifiers, copied passages, rare combinations and linkage to information someone might already know. Replacing a name while preserving a distinctive incident, location and date can leave the underlying person recognizable. For sensitive releases, a few manual spot checks and a duplicate scan are insufficient; the privacy assessment needs to match the threat and intended recipients.

Provider training policies answer a different question from your right to train on outputs. For example, Gemini's unpaid-service terms warn against submitting sensitive information, while its paid-service data-use terms say prompts and responses are not used to improve Google's products. The terms apply the paid-service data-use rules to unpaid use in the EEA, Switzerland and UK too. Gemini data-use terms.

Neither a provider's no-training commitment nor local execution proves that your resulting dataset is safe to share. Keep provenance and access controls attached to it until the release review is complete.

What Is Overhyped About Synthetic Data?

The biggest overstatement is that more generated rows solve a shortage of evidence. They can broaden practice and expose failures, but they can also reproduce the same assumptions at scale.

Sameness Hides Behind Different Wording

A dataset can contain distinct sentences and still repeat a tiny set of situations. Track scenario coverage after filtering: what business rules, ambiguities and failure modes remain represented?

For the ticket example, repeated invoice complaints do not compensate for missing combined billing-and-access requests. Review nearest neighbors and inspect rejected examples as well as accepted ones. An overly strict quality filter can quietly remove the difficult cases you were trying to add.

Bias Can Survive the Rewrite

Do not assume that paraphrasing a biased seed removes its assumptions. Review label choices across language style, dialect, spelling quality and any relevant user groups. A frustrated or poorly spelled request should not be routed differently unless the policy justifies that distinction.

Ask reviewers to compare matched cases where only an irrelevant attribute changes. If labels change, investigate the seed, prompt and filter together. A generator and a judge can reinforce the same mistake.

A Model Grading Its Own Homework Can Look Convincing

Zheng and colleagues documented position, verbosity and self-enhancement biases in LLM judges. A model may reward wording or presentation instead of the property you intended to measure. LLM-as-a-judge research.

Use deterministic checks wherever the answer is mechanically verifiable. For subjective checks, calibrate the judge against human decisions, hide generator identity where practical and review disagreements. Changing the judge's provider can reduce one shared dependency; it does not guarantee independent errors.

The LLM-as-a-judge guide explains how to check the evaluator before trusting its score. In synthetic generation, that check belongs before the judge decides which examples become training truth.

Model Collapse Is a Specific Risk, Not a Universal Verdict

Shumailov and colleagues studied recursive training on generated data and showed how less common parts of the original distribution can disappear. That supports caution about repeatedly replacing source evidence with a model's own descendants. It does not establish that every reviewed, task-specific synthetic fine-tuning run must fail. Nature research paper.

Keep permitted original evidence, preserve rare cases deliberately and compare downstream behavior against an untouched baseline. There is no universal synthetic-to-real mixing ratio that can replace that evaluation.

What to Do Differently on Monday

Act now if you can name a missing case, write its correct behavior and review generated candidates against a stable rule. Start with the small coverage grid, then expand only the cells that add useful cases.

Wait on fine-tuning if reviewers disagree about the target labels or you lack an independent acceptance set. Resolve the policy and evaluation problem first.

Use ordinary fixtures if the job is database population, boundary values or transaction integrity. Add generated language only where it exercises behavior that simpler data misses.

Before the next release, compare the baseline and changed system on the same untouched cases. For the support router, inspect incorrect routes and missed escalations by category. Keep the synthetic data only if it improves the behavior you intended without unacceptable regressions elsewhere.

Questions About Synthetic Data

Get the newsletter for practical guides to building and evaluating AI systems.