Gemini agentic video understanding, explained: when 88% fewer tokens is real

On September 1, 2026, Google gave the Gemini API a different way to read long video: instead of loading the whole timeline at a fixed rate, the model can now inspect only the transcript, frames, or audio it needs for your question. Google reports up to 88% fewer tokens on long-form video, but that number is a ceiling, not a blanket discount on every bill.

What Gemini agentic video understanding actually is

The old default is static processing. Gemini extracts one frame per second, processes the audio, puts that material into the context window, and answers in one pass. It is predictable because the model sees the timeline at a fixed sampling rate whether your question needs all of it or not.

The new option is agentic processing. The model first works out what evidence it needs. It can request a transcript section, load frames from a relevant moment, inspect audio, then repeat that loop before it writes the answer. In plain words, Gemini now searches inside the video before it answers about the video.

The useful analogy is a researcher in a video archive. Static mode rolls every reel past the desk. Agentic mode reads the catalog, pulls the likely reel, checks the exact scene, and goes back for another piece only if the first one was not enough.

This shipped for gemini-3.7-flash, gemini-3.6-flash, and gemini-3.5-flash-lite. It works through both the Interactions API and the GenerateContent API. The Interactions API is the recommended path, and that is the one I will use below because its response exposes the model's processing steps.

There are two new response-step types behind that loop. A processing_call means the model asked for another video segment or transcript. A matching processing_result contains what came back. If those step types appear in interaction.steps, agentic navigation ran. Your application can show them as progress, but it does not have to answer them like a custom function call.

One boundary matters: this is video understanding, not video generation. These three models take video in and return text. If you want to create or edit video, that is the separate Gemini Omni Flash workflow.

Why the 88% number matters

Static video gets expensive in a very mechanical way. At low media resolution, the guide estimates about 100 tokens for each second of video. One hour therefore lands around 360,000 input tokens before the answer. At the current Gemini 3.7 Flash or 3.6 Flash Standard rate of $0.75 per 1 million input tokens through December 31, 2026, that input costs about $0.27.

Apply the full 88% reduction to that input budget and you get 43,200 tokens, or roughly $0.0324 at the same rate. That is the attractive case: a targeted question about a long recording no longer needs every sampled frame in context.

It is not the whole bill. Agentic exploration creates thought tokens, billed at the output rate, and tool-use tokens for the transcript, audio, and frames it loads. The model may also inspect more material when the question is broad or the video is visually dense. Google reports approximately 7% higher quality on long-form content, but the public guide does not show the benchmark behind that figure.

The model choice changes the unit price too. Gemini 3.5 Flash-Lite costs $0.30 per 1 million input tokens and $2.50 per 1 million output tokens on Standard. Gemini 3.7 Flash and 3.6 Flash cost $0.75 for input and $3.75 for output through the end of 2026, then both scheduled rates double on January 1, 2027.

That makes agentic video useful on two axes at once. It can fit a longer source into the context budget, and it can reduce the amount of paid material the model has to inspect. The win is strongest when the source is long and the question is narrow.

Who is unaffected? A team processing thirty-second product clips is unlikely to see the same benefit. A workflow that already knows the exact five-minute section it needs may be better off clipping it and using static mode. A consumer using the Gemini app also gets no new switch here. Google shipped an API processing parameter, not a consumer control.

Who can use it tomorrow

An L&D lead with hour-long training recordings

A learning and development lead at a large company can upload a recorded workshop and ask for the main procedures, the examples attached to each one, and a quiz. Agentic mode can start with the transcript, pull frames when a slide or physical demonstration matters, and leave unrelated sections alone.

The payoff is not merely a cheaper summary. The same source can support targeted questions such as, "What did the presenter say about escalation?" without filling the context window with every frame first.

A SaaS research lead reviewing customer interviews

A product research lead can ask an hour-long interview for every moment where a customer mentions onboarding friction, pricing confusion, or a missing workflow. The answer can include timestamps so a researcher returns to the original recording before turning a model output into a product decision.

The payoff is review speed with an audit path. Gemini can narrow the footage. The researcher still checks the clip that supports the claim.

A media-operations team searching an archive

A media team can point Gemini at a long webinar, town hall, or interview and ask for the moment a named topic appears. This is the shape agentic mode was built for: a large timeline and a specific retrieval job.

The payoff is faster handoff. An editor gets the likely moments and timestamps instead of a loose summary of the whole program.

A manufacturing QA engineer triaging inspection footage

A QA engineer can use agentic mode to search a long camera recording for a named event, such as a guard opening or a warning light changing. Once Gemini finds a suspicious interval, the engineer can process that short section in static mode.

That second step matters. Static mode is still the right tool when every sampled frame matters, and Google warns that one-frame-per-second processing can miss fast action. This workflow uses agentic mode to find the neighborhood, then static or a dedicated vision system to verify the event.

An education product keeping a lecture conversation alive

An education-product builder can create one interaction around a lecture, then let a student ask follow-up questions with previous_interaction_id. The server retains the video context, so the product does not need to rebuild the whole conversation on every turn.

The payoff is a real study session rather than a one-shot summary. The catch is state: if the product runs stateless, it must replay every returned processing step or the next answer loses the video context.

How to use agentic mode with the Interactions API

The shortest production-shaped path uses the official Google Gen AI Python SDK and the Files API. Use the Files API for a meaningful-duration video, anything you plan to ask about more than once, and any total request over 20 MB.

1. 1

   Set the API key and install the SDK

   Create a Gemini API key in Google AI Studio, put it in GEMINI_API_KEY, then install the current SDK:

   export GEMINI_API_KEY="YOUR_API_KEY"
   pip install -U google-genai

2. 2

   Upload the video and request agentic processing

   Replace the path with a local video. This example is the Python flow from Google's agentic video guide:

   import time
   from google import genai
   
   client = genai.Client()
   
   # Upload a long video
   video_file = client.files.upload(file="path/to/lecture.mp4")
   
   while video_file.state.name == "PROCESSING":
       time.sleep(2)
       video_file = client.files.get(name=video_file.name)
   
   # Use agentic processing
   interaction = client.interactions.create(
       model="gemini-3.7-flash",
       input=[
           {
               "type": "video",
               "uri": video_file.uri,
               "mime_type": video_file.mime_type,
               "processing": "agentic"
           },
           {"type": "text", "text": "What are the three main arguments presented?"}
       ]
   )
   print(interaction.output_text)

3. 3

   Verify the mode before trusting your test

   Inspect interaction.steps. You should see processing_call and processing_result entries before the final model_output. If you do not, you have not proved that the request used dynamic navigation.

4. 4

   Measure the right comparison

   Run the same representative videos and questions in static and agentic mode. Compare total input tokens, thought tokens, tool-use tokens, time to first token, answer quality, and final cost. The 88% figure is a vendor maximum. Your prompt mix decides whether it survives contact with production.

The easiest mistake is putting processing on the request instead of the video object. It belongs inside that video input. When one request contains multiple videos, each video can choose its own mode, so a long lecture can be agentic while a short experiment clip stays static.

Prompt order matters too. For one video plus text, Google recommends putting the video first and the text prompt after it, as the example does.

The honest part

Agentic mode trades a fixed scan for a search loop. On clips under five minutes, that loop can slightly increase time to first token because the model reasons, asks for material, and waits for internal tool results before it starts the final answer. If the clip is short and your application is latency-sensitive, static mode is the cleaner default.

Custom clipping and frame-rate controls are another hard boundary. start_offset, end_offset, and a custom fps work only with static processing. If you already know the exact interval, clipping a static request can be simpler and more predictable than asking the model to find it again.

The input limits need care because Google's current guide contradicts itself. Its comparison table labels inline data as under 100 MB, but the detailed inline section says the total request must stay under 20 MB and explicitly tells you to use the Files API above 20 MB. Follow the conservative 20 MB threshold until Google reconciles the page.

The File API itself allows files up to 20 GB on paid accounts and 2 GB on free accounts. Models with a 1,048,576-token context window can process up to three hours of low-resolution video or one hour at high resolution. Public YouTube URLs work, but private and unlisted videos do not. The free tier also caps YouTube input at eight hours per day.

Multi-turn state has a production trap. Stateful requests using previous_interaction_id keep the video context on the server. Stateless requests must send back every processing_call and processing_result step. Leaving those steps out does not currently produce an error, but Google says the video context disappears and follow-up quality drops sharply. Replaying the steps also adds input tokens.

Finally, retrieval is not verification. Google's own safety guidance says model outputs can be inaccurate and calls post-processing and human evaluation essential. For medical, legal, safety, compliance, or frame-precise decisions, use Gemini to locate evidence and keep a human or deterministic system in the approval path.

What to do now

Use agentic mode this week if your product sends long recordings to Gemini and asks narrow questions about moments, topics, or arguments. Start with Gemini 3.7 Flash when answer quality is the priority, and test Gemini 3.5 Flash-Lite when volume and cost dominate.

Keep static mode if your clips are short, startup latency matters, you need clipping or a custom frame rate, or the job requires a consistent scan across the whole timeline. The new option does not make the old one obsolete. It gives you a second reading strategy.

Wait if your product cannot yet capture the separate usage fields or compare answers against source timestamps. Without those receipts, you cannot tell whether agentic navigation saved money or merely moved tokens between categories.

You are unaffected if your need is video generation, a consumer Gemini feature, or deterministic frame-by-frame analysis. Those are different products and different technical problems.

If you want the next platform change translated into a working decision, join the newsletter.