Assembly AI

Choose AssemblyAI when you are building a product that needs transcripts, speaker attribution and structured notes from the same audio. The key Assembly AI decision is which processing path fits: the current recorded flagship covers 18 languages, while the current realtime flagship covers 32. Start with recorded audio for meeting notes; choose streaming when the application must respond before the conversation ends.

What Is Assembly AI?

AssemblyAI is a developer platform that converts audio into text and exposes additional APIs for analyzing that text, generating speech and calling language models. You supply a recording or live audio stream and build the application around the returned results: a searchable interview, a meeting summary, a call-analysis screen or a voice agent. It supplies the speech components; your product still needs its own interface, storage, permissions and business actions.

Models, capabilities and prices verified against AssemblyAI's live pages on 5 November 2026. This evaluation uses documentation, public product pages and the cost calculations below; it does not claim an audio accuracy or latency test.

Who Should Use It, and Who Should Skip It?

AssemblyAI belongs on your shortlist when the deliverable includes both a transcript and something derived from it. A recruiting product needs speaker-attributed interview notes. A support product needs searchable calls and extracted issues. A voice assistant needs incoming words and a usable signal that the caller has finished speaking.

The criteria that decide this review are processing mode, required languages, speaker attribution, downstream output and the bill for that configuration. An overall accuracy slogan cannot answer those questions.

Assembly AI vs Deepgram

Deepgram fits better when controlling conversational turns is the central problem: Flux exposes model-integrated end-of-turn detection and configurable turn behavior. Its Flux quickstart also identifies an integration boundary: Flux uses /v2/listen, so an older /v1/listen transcription connection is not a drop-in implementation. Shortlist it for a phone assistant that keeps interrupting callers or waiting after they finish; compare those behaviors on your own calls.

When ElevenLabs Fits Better

ElevenLabs fits better when you need a broader live-language shortlist or media-oriented transcription: its speech-to-text documentation lists Scribe v2 and Scribe v2 Realtime across 90+ languages, with non-speech audio-event tags on batch output. Check each mode separately before promising identical features. The same documentation says there is no verbatim mode, a specific reason to qualify another option if retaining every hesitation is essential.

When OpenAI Fits Better

OpenAI fits better when your application already uses its API and needs transcription alongside that existing integration. Its transcription migration guide distinguishes GPT-Transcribe (gpt-transcribe) for completed audio from GPT-Live-Transcribe (gpt-live-transcribe) for continuous live input. Speaker-labeled files use a separate gpt-4o-transcribe-diarize workflow; do not assume switching to a newer model preserves subtitle formats or timestamps.

For the wider shortlist, use our speech-to-text API comparison. If your only requirement is a finished meeting-notes application, evaluate that product category before committing to API integration work.

Assembly AI Transcription: Recorded Audio and Model Choice

AssemblyAI's pre-recorded API is the natural starting point for completed meetings, interviews and calls. It processes a file asynchronously: your application submits the job, then retrieves the result when processing finishes. The person reading the notes does not need the transcript to arrive while someone is still speaking.

The model names matter because the stable recorded model, the preview and the live flagship are different products. The current model catalog and the mode-specific selection pages establish this lineup:

The identical universal-3-5-pro identifier in the recorded and realtime rows is not permission to interchange their APIs. Recorded requests use the plural speech_models array; streaming connections use the singular speech_model parameter.

Universal-3.5 Pro's recorded languages are English, Spanish, German, French, Portuguese, Italian, Turkish, Dutch, Swedish, Norwegian, Danish, Finnish, Hindi, Vietnamese, Arabic, Hebrew, Japanese and Mandarin. Code switching means recognizing a conversation that changes languages, rather than requiring each recording to remain in one language.

AssemblyAI's recorded model-selection documentation specifies the default combination as ["universal-3-5-pro", "universal-2"]. It selects the first model that supports the detected or specified language. Inspect speech_model_used in the result instead of assuming every job used the flagship.

That matters for a product serving English and Korean interviews. English can use Universal-3.5 Pro; Korean is outside that model's listed languages and needs the wider recorded route. Calling the whole application “99-language transcription” conceals a model change that your evaluation needs to include.

A Recorded Meeting Workflow

AssemblyAI's file-transcription quickstart supports a hosted audio URL or an uploaded local file. For a meeting-notes feature, the useful sequence is:

1. 1

   Choose the model and language route

   Set the intended speech_models array explicitly. Enable language_detection when the application does not already know the language. Keep the selected model in the job's metadata so your evaluation can separate different routes.

2. 2

   Request the transcript your product needs

   Submit the recording to POST /v2/transcript with speaker_labels: true if the interface needs attributed dialogue. Add relevant keyterms, such as customer and product names, when your qualification recordings show those terms need help.

3. 3

   Wait for a completed job

   Poll GET /v2/transcript/{id} or use a completion webhook. Handle failed jobs explicitly. Store the transcript ID with your own meeting ID so later summaries and corrections remain attached to the right meeting.

4. 4

   Build notes from attributable evidence

   Preserve speaker utterances and timestamps before generating a summary. A manager should be able to open the source passage behind a disputed decision instead of receiving an unsupported paragraph.

Keyterms and contextual prompts solve different problems. Keyterms name likely vocabulary; a contextual prompt describes the situation. Universal-3.5 Pro accepts up to 1,000 keyterms, while Universal-2 accepts up to 200. Neither is a reason to judge success only on a clean sample containing familiar words.

When Sync or Dictation Is the Better Interface

AssemblyAI's Sync Speech-to-Text API returns a finished transcript in one HTTP response. Choose it for a short voice message or another interaction where managing an asynchronous job adds unnecessary application state. Its Sync quickstart distinguishes raw HTTP uploads of up to 20 minutes from SDK or live-upload requests capped at 120 seconds. That interface-specific limit matters if you prototype with an SDK and later accept longer recordings.

The Dictation API adds cleaned or formatted text to the transcript in the same request. It fits a spoken message that should become ready-to-send text. Keep ordinary transcription when the original wording is the product; choose dictation when editing the wording is part of the requested outcome.

Streaming for Voice Agents and Live Captions

AssemblyAI streaming is for applications that must react during the conversation. A persistent WebSocket connection receives audio and emits transcript events as speech arrives. A captioning interface displays those updates; a voice-agent orchestrator uses them to decide what to do next.

For a new integration, start qualification with Universal-3.6 Pro Realtime. The streaming model-selection page names it as the default when speech_model is omitted. Pin the model explicitly so a configuration review can explain what you intended to deploy.

The budget alternatives are narrower. Universal-Streaming English, called Universal-Streaming on the pricing page, serves English. Universal-Streaming Multilingual supports English, Spanish, German, French, Portuguese and Italian, with language handling per turn. The realtime flagship supports native code switching across its wider language set. A bilingual caller switching within a sentence therefore deserves a separate qualification recording from a caller who changes language between turns.

For a booking assistant, the meaningful result is the correct appointment request and an appropriate moment to answer. A transcript fragment can contain the right date while the customer is still correcting the time. Treating every fragment as a completed instruction can create the wrong booking even when the recognized words are correct.

The streaming quickstart documents Begin, Turn and Termination messages. A Turn with end_of_turn: true marks a finalized turn. An endpointing signal marks a speech boundary, not authorization for a transaction. Handle partial text separately from the text your business logic commits, match the audio encoding and sample rate to your source, and terminate the session when the call ends.

A sensible evaluation records failures separately: wrong words, wrong entities, premature answers and long waits after a completed request. Measure the complete delay from a finished request to the agent's reply as well as transcription timing; the application, language model and speech output all contribute.

Assembly AI Speaker Diarization: Who Said What

AssemblyAI speaker diarization groups speech by speaker. It answers “which voice said this?” Speaker Identification is a separate feature that attempts to attach names or roles. A transcript containing “A” and “B” has not yet established which person is the interviewer or customer.

For recorded audio, speaker_labels: true produces speaker-attributed utterances, as shown in the recorded diarization documentation. Those segments are useful for an interview player that highlights the speaker and jumps to the corresponding audio.

For live audio, AssemblyAI returns turn-level speaker_label values and speaker fields on final words. However, those assignments are provisional enough that your application must support correction. The streaming speaker documentation describes SpeakerRevision messages that amend earlier turns, with the final revisions delivered before Termination.

Consider a project meeting where a brief “I'll do it” becomes an assigned action. If the initial speaker label is wrong and the application never applies the revision, the final notes can give the task to the wrong person. Store turns by turn_order, apply the corrections, and generate the final notes from the corrected transcript. An unchangeable log of the first text event is the wrong storage design for this feature.

Two less obvious constraints matter:

• max_speakers is a hard cap. Additional speakers are merged into existing labels when the cap is exceeded. Setting it to the expected attendance without allowing for a late participant can damage attribution.
• Brief responses can be PENDING. Treat them as unattributed until sufficient evidence or a correction arrives. Overlapping speech also remains a difficult case; the documentation says simultaneous speakers cannot simply be split into independent voices.

If the recording system already captures each participant on a separate channel, retain that structure and evaluate multichannel transcription. Channel separation and diarization are different tools: one uses your recording's structure, while the other infers speaker changes from mixed audio. Multichannel processing also changes the bill.

Speech Understanding and LLM Features

AssemblyAI's Speech Understanding features make the strongest case for choosing it over a transcript-only integration. The Speech Understanding documentation lists Action Items, Auto Chapters, Custom Formatting, Entity Detection, Key Phrases, Sentiment Analysis, Speaker Identification, Summarization, Topic Detection and Translation.

For meeting notes, the useful workflow is specific: transcribe the meeting, resolve the speaker attribution, generate a summary, then extract actions and their owners. For support analytics, it might be entities and topics across completed calls. Buying every analysis feature by default adds expense without establishing that the product uses its output.

Use LLM Gateway when the output needs your own schema or reasoning instructions. A large language model, or LLM, can turn transcript text into fields such as decision, owner, due_date and evidence. The Gateway quickstart documents transcript injection by transcript_id, structured outputs, tool calling, retries and model fallbacks through a common API.

The workflow still needs a definition of a valid answer. If a speaker says “sometime next week,” your extraction should preserve that uncertainty instead of inventing a calendar date. If no owner accepts a task, an empty owner field is more useful than a plausible name. Link extracted items back to the relevant utterance so someone can check them.

A valid JSON response proves that the output has the requested structure. It does not prove the proposed action happened or that a person agreed to it. Use an application approval step before extracted notes change customer records or assign consequential work.

Medical Mode is an optional domain add-on for medical vocabulary, available for supported recorded and streaming models in English, Spanish, German and French. It is a narrower feature than the platform's overall language coverage. A multilingual clinical product must qualify the domain mode itself rather than assuming every supported transcription language receives the same specialization.

AssemblyAI-TTS-1 handles the other direction: it turns text into spoken audio over streaming WebSocket or HTTP. It supports English, Spanish, German, French, Italian and Portuguese, plus voice cloning and word timings. For a voice agent, that means the input model, reasoning model and output voice need separate checks. The realtime transcription model's language list does not automatically become the speech-output language list.

Assembly AI Pricing

AssemblyAI's pay-as-you-go option is the sensible starting point for this reader: it has no minimum commitment, and the bill depends on the models and features you enable. These are the public rates on AssemblyAI's pricing page, verified on 5 November 2026. Recorded rates apply to processed audio; streaming rates apply to open-session time.

• Universal-3.5 Pro: $0.21 per hour of pre-recorded audio.
• Universal-2: $0.15 per hour of pre-recorded audio.
• Universal-3.6 Pro Realtime: $0.45 per streaming session-hour.
• Universal-Streaming: $0.15 per streaming session-hour; this is Universal-Streaming English in the docs.
• Universal-Streaming Multilingual: $0.15 per streaming session-hour.
• Universal-3.6 Pro Preview: no separate rate listed on the current pricing page; confirm before budgeting a preview deployment.
• Universal-3.5 Pro Realtime: still documented as supported, but no separate rate row on the current pricing page; confirm your pinned integration's rate.
• Sync API: $0.30 per hour of audio.
• Dictation API: $0.62 per hour, including its listed formatting and prompting features.
• AssemblyAI-TTS-1: $15 per million input characters.

For the meeting-notes and voice-agent workflows above, these additional lines affect the decision:

• Speaker Diarization: add $0.02 per recorded hour or $0.12 per streaming session-hour.
• Summarization: add $0.02 per hour at low effort, the default, or $0.07 at medium effort.
• Speaker Identification: add $0.02 per hour at low effort, the default, or $0.10 at medium effort.
• Medical Mode: add $0.15 per hour on eligible models.
• Universal-3.5 Pro prompting: keyterms and contextual prompting are separately listed at $0.05 per hour each; Universal-2 includes keyterms.
• Streaming keyterms: included with Universal-3.6 Pro Realtime; add $0.04 per hour with Universal-Streaming.

LLM Gateway usage has a separate input-token and output-token bill for the model you select. A token is a unit of text processed by that model. The advertised Gateway rates use global routing; the pricing page states that US/EU regional routing costs 10% more. Include that choice when estimating a custom extraction workflow.

That comparison makes the selection practical. If both recorded models meet your acceptance criteria, Universal-2 saves money. If the more expensive model prevents enough corrections to justify the difference, choose it. The calculation does not assume either model passes your audio evaluation.

For live sessions, the billing documentation makes the unit explicit: the clock runs while the WebSocket remains open, including idle time. An illustrative 1,000 session-hours of Universal-3.6 Pro Realtime with speaker diarization costs $570, before LLM calls or generated speech. A connection open for an hour still incurs an hour of usage when it contains only 20 minutes of speech. Close it promptly when the interaction finishes.

Multichannel transcription is billed per channel. A one-hour recording processed as two independent channels creates two hours of transcription usage. Budget the recording layout as well as the model; a mono meeting and a separated-channel call are not interchangeable cost inputs.

The pricing page advertises free signup without a card and up to 185 hours of pre-recorded transcription and up to 333 hours of streaming transcription. Treat these as published maximum allowances, not an entitlement to those hours on every configuration. The free streaming plan permits 5 new connections per minute; pay-as-you-go starts at 100 new sessions per minute with automatic scaling. Those are connection-start rates, not a count of simultaneous calls. A launch that opens many sessions at once can hit that free-plan wall before it exhausts the allowance.

Limitations That Can Change the Decision

AssemblyAI has enough model and feature variation that a successful sample transcript is only the start of qualification. These are the constraints most likely to change the implementation or the vendor choice.

Language coverage is not uniform across the workflow. A recorded fallback can extend transcription coverage, but a specialist add-on, live model or generated voice can have a narrower list. Write a requirement for the whole path, such as “Korean audio to attributed Korean meeting notes,” and verify each stage.

Live attribution requires revisable state. The streaming speaker documentation describes provisional labels, PENDING responses and later corrections. Its experimental speaker-confidence field is also not a calibrated probability. Do not convert that score into a promise that an action owner is correct. Use uncertain segments for review, and apply revisions before producing the final record.

The recorded preview has explicit weaknesses. AssemblyAI's model-selection documentation warns that Universal-3.6 Pro Preview can produce less punctuation in some European languages and lose accuracy on complex code-switching audio. Those are vendor-stated limitations, not results from this review. Qualify it separately from stable Universal-3.5 Pro; the higher version number does not settle the decision.

Data handling needs an account-level decision. AssemblyAI's data-retention and model-training documentation distinguishes production retention from model-training use. Training eligibility depends on the applicable contract and controls; the documented streaming zero-retention option is conditional on opting out of training, and operational metadata remains. Check the chosen account and processing region before sending customer recordings.

Speech analysis does not finish the application. You still need to handle failed jobs, revise notes, connect records to customers and decide which extracted actions require review. Gateway tool calling can support an integration, but your application supplies the permissions and action logic. That engineering work is part of the buying decision even when the transcription invoice is small.

Verdict: Choose by the Output Your Product Needs

AssemblyAI earns a shortlist place for products that turn conversations into attributed, structured information. For recorded meeting notes, start with Universal-3.5 Pro plus only the analysis features the interface uses, and compare Universal-2 if cost or broader language coverage matters. For a new live assistant, qualify Universal-3.6 Pro Realtime against the appropriate budget streaming model and Deepgram Flux.

The decision rule: choose AssemblyAI when its exact model, language route and speaker-output contract satisfy your product, and its transcript-to-insight features reduce the integration work you would otherwise build. Choose an alternative when its required language, turn behavior or output format fits better. Do not switch providers solely because a clean demonstration sounded convincing.

Next week, use a small set of your permitted recordings containing ordinary conversation, unfamiliar names, interruptions, short acknowledgments and the language switches your users make. Write down the fields your application needs before comparing outputs. Check the final attributed notes as well as the transcript, then calculate usage with the selected add-ons and actual connection duration. The public playground is a useful starting interface; your production configuration is the acceptance test.

FAQ

Assembly AI API key

Create an account and retrieve your key from the dashboard. Keep the durable key on your server. The recorded transcription API uses it in the authorization header without a Bearer prefix; follow the authentication instructions for the particular API rather than assuming every endpoint is identical.

Assembly AI Playground

The public playground lets you choose sample audio, upload a file or supply a URL, then configure model context and supported analysis features. It helps inspect the product before integration. Use representative audio and evaluate the same configuration you intend to deploy.

AssemblyAI API

Use the pre-recorded API for completed jobs that can finish asynchronously, streaming for ongoing audio and live responses, or Sync for a finished transcript in one request-response interaction. Dictation adds edited output when the application needs ready-to-use written text.

AssemblyAI docs

Begin with the current model-selection page for your processing mode, then its matching quickstart. Copy the current API identifier, check language and feature support, and pin the model in your configuration. A product name from an older tutorial can describe a different model generation.

Use the AI Business Workflow Audit Checklist to map your audio source, required output and application handoffs. Get the checklist.