Gemini API Tool Retrieval: How an Agent Loads a Schema on Demand
Follow a refund from schema search to function call, choose server-side or client-side retrieval, and see when deferring Gemini tools costs more.
Published

A support agent looking up an order can carry the full refund schema through every turn, even if nobody asks for a refund. Gemini API Tool Retrieval lets the model see that a refund tool exists and load its parameters when a customer actually needs it. For a developer with a large catalog, the decision turns on how much schema the agent can leave unread and how often it has to search.
What Gemini sees before the search
Google launched Tool Retrieval in preview on November 19, 2026. As of November 2026, it requires v1beta on the Interactions API. Interactions itself is generally available. An agent using generateContent needs to move its tool loop to Interactions to use retrieval. API changelog · Interactions API overview
Google’s announcement names Gemini 3.6 and later, including all Gemini 4 models. The retrieval guide explicitly lists gemini-3.6-flash, gemini-3.8-flash, gemini-4-argon and gemini-4-hydrogen. Hydrogen is the model in Google’s examples below. Launch announcement · Supported models
With default, server-side retrieval, your application still sends the complete declarations. What changes is the input Gemini sees. Each deferred function initially occupies a line with its name and short_description, or the first sentence of its description. The API holds the parameter types, properties and required fields until the model asks for the function by name. This distinction matters if you log HTTP payload sizes to estimate model costs. Tool retrieval guide
As of Nov 20, 2026 · Source: Google, Gemini API Tool Retrieval
A refund in one server-side request
Here is Google’s Python example. The plain tool_search entry enables server-side lookup. The flag belongs to the function. Keep its complete parameters object here so that Gemini has something to retrieve. Google’s refund example
from google import genai
client = genai.Client()
tools = [
{"type": "tool_search"},
{
"type": "function",
"name": "issue_refund",
"description": "Refund an order to the original payment method.",
"defer_loading": True,
"parameters": {
"type": "object",
"properties": {
"order_id": {"type": "string"},
"amount": {"type": "number"},
},
"required": ["order_id", "amount"],
},
},
# ... 50+ other deferred functions (leave frequent tools undeferred)
]
interaction = client.interactions.create(
model="gemini-4-hydrogen",
input="Order A-1042 arrived damaged. Please refund 40 dollars.",
tools=tools,
)Order A-1042 arrived damaged. Please refund 40 dollars.
function_names = ["issue_refund"]
issue_refund requires order_id (string) and amount (number).
issue_refund · {"order_id": "A-1042", "amount": 40}
The first two steps record work the API has already done. The last gives your application work to do. At that point the customer has requested a refund and Gemini has supplied arguments. Your existing handler validates the request, runs the refund and sends back a function_result with the call’s ID. Inspect interaction.steps on every response. A function_call can be present even when the status differs from requires_action. Google’s response format · Function calling explained
Which schemas should stay loaded?
Google’s rule is “keep your most-used tools loaded up front and defer the long tail.” For a support agent that looks up an order on most turns and issues refunds occasionally, keep the lookup loaded and defer the refund. If refunds are the agent’s primary job, the same refund schema belongs in its initial context. Google’s launch guidance
There is an exception worth checking in your prompts. If a system instruction or another tool’s output mentions a function by name, leave that function loaded. The model can otherwise emit a call directly, skip the search that supplies its schema, and leave the API unable to continue when you return the result. For tools you do defer, make the first line distinguish similar operations. Google’s examples billing_get_customer and crm_get_customer give the model a service to choose before it sees either parameter list. Google’s best practices
When your app owns the registry
Consider an agent whose user connects a billing integration halfway through a conversation. Your registry knows which billing tools that user may access. With client-side retrieval, you initially declare a search function with a name, description and query schema. You can make a newly available function discoverable without having declared it in the original request. Each search costs one extra application-to-API round trip. Server-side and client-side search are mutually exclusive within a request. Retrieval modes
Google’s client-side excerpt starts with the search declaration and resumes inside the handler for a search call. Here interaction is the preceding interaction, step is its ordinary function_call for search_tools, and my_registry is the lookup your application supplies. Google’s client-side example
tools = [{
"type": "tool_search",
"execution": "client",
"name": "search_tools",
"description": "Search the internal tool registry for functions.",
"parameters": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
}]
# When the model emits a function_call for "search_tools",
# return matching {"type": "function", ...} schemas in function_result:
followup = client.interactions.create(
model="gemini-4-hydrogen",
previous_interaction_id=interaction.id,
tools=tools,
input=[{
"type": "function_result",
"name": "search_tools",
"call_id": step.id,
"result": my_registry.search(**step.arguments),
}],
)Return complete function declarations as objects in function_result.result. The call_id ties those matches to the search that requested them. The next function_call can name a discovered billing function, which your normal handler executes. A registry that finds nothing can return a short text result naming the categories it can search so the model can revise its query. Client-side retrieval
What changes for remote MCP servers?
A remote Model Context Protocol (MCP) server already supplies tool definitions and runs its tools. Add defer_loading: true to the mcp_server declaration alongside the server-side tool_search entry. The flag covers every tool the server exposes. You can declare your own functions alongside it. Google’s MCP configuration · What is MCP?
Google’s weather and currency example shows why the execution boundary matters. A single search requests both weather_service:get_weather and convert_currency. For London weather the API contacts the server and returns an mcp_server_tool_call followed by an mcp_server_tool_result. For the currency conversion, a local function, the response ends with a function_call for your app. Dispatching every step as local work would repeat work the remote server already completed. Google’s mixed-tool response
What it costs, and when it costs more
Tool Retrieval has no separate fee. The model’s standard input, cached-input and output rates apply. Enabling server-side search adds a small fixed instruction block. Each deferred tool still contributes its catalog line. Once retrieved, its schema stays in context and is billed on subsequent calls for the rest of that conversation. Google’s pricing explanation · Gemini API rates by model
A server-side search saves a network round trip for your app, but the API makes another model call with the accumulated context and newly loaded schemas. In usage, the initial prompt is counted in total_input_tokens and the subsequent pass in total_tool_use_tokens, billed at input rates. Check total_cached_tokens for the portion served at the cached rate. Retrieved schemas are appended where they were loaded, preserving the earlier prefix for reuse. A smaller initial prompt can also fall below the model’s cache minimum. Google’s billing guide
Google illustrates the trade-off over a call and its follow-up, with two schemas retrieved and 400 tokens per schema. These are worked examples that exclude output and caching. Google’s token arithmetic
In the smaller catalog, a search makes the model read the long conversation again to recover a small amount of schema. For a short, stable list, Google suggests static loading for fewer than roughly 20 tools. For a large, fixed list where most tasks touch only a few tools, try server-side retrieval. Choose client-side search when a live registry, namespace ranking or per-user filtering justifies the extra request. Google’s mode comparison
How the search differs across labs
If you port a tool catalog, review the text the model sees before search. A service description that makes a namespace easy to find in Responses may still leave similarly named Gemini functions hard to distinguish. Google’s retrieval guide · Anthropic’s tool search guide · OpenAI’s tool search guide
Migration checklist for an existing agent
- Record a baseline. Pick representative conversations, including one long conversation and one that needs a rare function. Record billed usage, latency and whether the agent called the right function with usable arguments.
- Choose the retrieval boundary. Use Interactions with a supported model and the preview API version. Start with server-side lookup for a known catalog. Choose client-side lookup when the app must decide which tools a user can discover.
- Audit names before setting flags. Keep frequent tools and functions named in instructions or tool outputs loaded. Give each deferred function a distinct first-line description and its full schema.
- Check the follow-up request. Resend the complete declared
toolsarray even withprevious_interaction_id, including when returning a function result. That ID preserves history. Tool declarations apply to each interaction separately. - Preserve stateless history. With
store=false, replay the response steps in order, including the original retrieval calls, results and opaque signatures. A dropped search result can lose the loaded schema. A missing call or signature can surface as a 400 on the follow-up. - Compare the whole conversation. Count search passes and cached input alongside initial input. If the agent searches for the same frequently used capability across most conversations, try keeping that schema loaded and compare the totals again.
Google’s multi-turn and stateless examples
Which MCP servers belong in your agent?
Compare servers by the work they expose before deciding which tool schemas to defer.
- Published
- Category
- Build







