Sending an agent the full instruction manual for every tool it might use is wasteful. Loading only the relevant schemas sounds like an obvious improvement. But a smaller first prompt is not the same thing as a smaller bill: the model may need to read the conversation again to decide what to do with the tools it just discovered.
Google’s November 19 launch of Gemini Tool Retrieval puts that trade-off into a concrete API. The preview lets agents load parameter schemas on demand through the Interactions API. The more important detail is in the technical billing reference: every server-side search introduces another model call with the full context, plus the loaded schemas.
That makes this an engineering decision, not a flag to enable everywhere. A large catalog with sparse usage is a promising fit. A small catalog behind a long conversation can become more expensive.
What Actually Gets Deferred#
With server-side retrieval, you still declare the complete library. Add a tool_search entry and set defer_loading: true on the functions whose schemas should stay out of the initial context. Gemini sees their names and compact descriptions, then requests full schemas when needed. You can also defer an entire remote MCP server’s tools, according to Google’s server-side guide.
“Deferred” therefore does not mean “absent from the request.” It means the full parameter schema is withheld from the model’s initial context. Google explicitly says to send complete schemas even for deferred functions; sending only a name and short description leaves retrieval with an empty schema.
The distinction extends the problem discussed in my earlier MCP adoption analysis. Connecting more capabilities is useful, but deciding how much of that catalog the model should actively read is a separate design problem. Tool Retrieval addresses schema exposure within Gemini, not interoperability in general.
A minimal server-side request body looks like this:
{
"model": "gemini-4-hydrogen",
"input": "Find the delivery status of order A-1042.",
"tools": [
{ "type": "tool_search" },
{
"type": "function",
"name": "orders_get_delivery_status",
"description": "Read the current delivery status for an order.",
"short_description": "Orders: look up delivery status.",
"defer_loading": true,
"parameters": {
"type": "object",
"properties": {
"order_id": { "type": "string" }
},
"required": ["order_id"]
}
}
]
}This is an illustrative configuration for POST /v1beta/interactions, adapted from the documentation—not a live-tested integration. A one-function example shows the syntax, not a workload likely to benefit from retrieval. The function still needs an application implementation, argument validation and execution controls.
There are two compatibility checks before trying it. The feature is a v1beta preview on the Interactions API only; generateContent does not support it. Also, check the explicit supported-model list, not just whether a model supports ordinary function calling. On November 20, that list names gemini-4-hydrogen, gemini-4-argon, gemini-3.8-flash and gemini-3.6-flash.
Choose Where Search Runs#
The retrieval-mode comparison describes three useful starting points:
| Approach | Who supplies schemas to the model? | Main trade-off |
|---|---|---|
| Static loading | Every schema is supplied upfront | Simple baseline; potentially large recurring context |
| Server-side retrieval | Gemini loads deferred schemas from your declared library | No application round trip for search, but additional internal model passes |
| Client-side retrieval | Your application searches and returns declarations | Control over catalog filtering and ranking, with an extra application round trip per search |
For client-side retrieval, declare a tool_search with execution: "client", a name, a required description and a parameter schema. Gemini emits an ordinary function_call for that search. Your application returns complete function declarations as objects in function_result.result, not as a JSON string. Those functions need not have appeared in the original catalog.
That is attractive for proprietary registries, changing integrations and per-user capability sets. My recommendation would be to filter that registry by the current user’s permissions before returning matches, then enforce authorization again when executing the selected function. Discoverability is not permission to act.
Do not mix the modes in one request. Google permits exactly one tool_search, and defer_loading belongs to server-side retrieval—not a client-side search declaration. Client retrieval also does not remove inference costs: each search continuation is another normally billed request.
One HTTP Call Can Contain Several Input Passes#
The phrase “all inside the same request” is easy to misread. Server-side search avoids an application-level round trip, but it does not imply one model invocation.
Google documents the usage fields this way:
total_input_tokenscovers the first model call’s prompt, including history, instructions, non-deferred schemas, retrieval instructions and the deferred-tool list.total_tool_use_tokenscovers prompts for additional internal model calls after server-side tools run, including searches. These calls resend the full context plus newly loaded schemas and are billed at the input rate.total_cached_tokensidentifies prompt tokens served from the implicit cache, charged at the cached rate instead.- Output and thought-token usage must also be accounted for when evaluating the complete workflow.
Tool Retrieval has no separate feature charge, but that is not the same as free retrieval. Monitoring only total_input_tokens can make a successful optimization appear larger than it is—or hide a regression. Conversely, do not attribute every tool-use token to search if a request also runs other server-side tools. Inspect the steps to understand the work behind the usage.
Google’s illustrative billing examples make the reversal clear. Both assume two requests: one ending in a function call, then one returning its result. They load two 400-token tools and omit output tokens and caching:
| Hypothetical workload | Static input tokens | Retrieval input + tool-use tokens |
|---|---|---|
| 200 tools at 400 tokens each; 2,000-token conversation | 164,000 | About 23,650 |
| 10 tools at 400 tokens each; 30,000-token conversation | 68,000 | About 92,830 |
These are Google’s worked arithmetic, not benchmark measurements. The examples assume roughly 26 tokens per deferred-tool entry and 150 tokens of retrieval instructions. They demonstrate a mechanism, not a reliable estimate for your schema format or traffic.
In the second case, there simply is not enough schema overhead to remove before another pass through the long conversation overwhelms the saving. Several searches within one turn can make that trade-off worse. Retrieved schemas also remain in context for the rest of the conversation; the preview has no unload mechanism.
Caching changes the economics again. The caching guidance says retrieved schemas are appended where they are loaded, preserving the earlier prefix for implicit caching. But deferring schemas can shrink a prompt below the model’s minimum cacheable size. Measure cold and warm sessions separately rather than assuming every repeated pass is either fully cached or entirely fresh.
Conversation State Is Part of Correctness#
The first retrieval call working is not enough. The multi-turn contract requires the full tools array on every request, including requests using previous_interaction_id and requests returning a function_result. Stored history does not mean stored declarations.
With server-side retrieval, keep the search entry, all deferred declarations and all non-deferred tools in that array. For client-side retrieval, resend its search declaration and other explicitly declared tools; discovered schemas arrive through the search-result history.
Stateless operation adds another obligation. If you use store: false and replay history yourself, Google requires the retrieval steps to circulate unchanged:
- Preserve both
tool_search_callandtool_search_result, including their opaquesignaturefields. - Keep their original order before the function call that uses the loaded schema.
- Preserve the other response steps and signatures too, including thought steps.
- Use an SDK version that recognizes these step types rather than dropping unknown fields.
A missing search-call step can cause the request returning the function result to fail, even though the first request succeeded. A missing search-result step can lose loaded schemas without the same validation check. Test the continuation, not just initial discovery.
Google also warns against deferring tools mentioned by name in system instructions or other tool outputs: the model can call one without retrieving its schema first, breaking the subsequent continuation. Keep those tools—and the frequently used core—loaded. For dispatch, inspect function_call steps on every response rather than relying solely on a requires_action status.
This is a practical complement to StateBench’s distinction between context and reliable state. A conversation can look semantically complete to a human while the execution protocol is missing information the next call needs.
Evaluate Catalog Size and Conversation Length Together#
I would compare static loading, server retrieval and client retrieval on the same accepted tasks, model and execution policy. This is a proposed evaluation plan, not a test I have run:
- Vary catalog size and schema weight. Use small, medium and large subsets of your real catalog; measure the actual token footprint rather than assigning every tool the same size.
- Vary history independently. Include short sessions and long production-shaped histories. A large catalog alone does not predict a win.
- Include discovery failures. Test similar names, no matching capability, multiple searches and newly connected integrations. Check outcomes, not merely whether a search step appeared.
- Exercise continuations. Return function results, restart from stored history and replay stateless histories through the real serializer. Include tools named in instructions.
- Measure complete tasks. Record usage across all requests, cache behavior, search count, latency distributions, correct completion and retries. Compare cold and warm runs under the same conditions.
The decision metric is the same one I argued for in my agent-cost comparison: cost per correctly completed task. A cheaper initial prompt is irrelevant if discovery adds retries or silently selects the wrong operation.
My Take#
I like this feature most as a way to keep an agent’s working interface deliberate. Leave the core capabilities visible, make the specialized tail discoverable, and require evidence that the smaller interface improves completed work. I would trial server-side retrieval first for a large, stable catalog; I would take on client-side search when permission-aware discovery or a dynamic registry justifies owning that extra component.
The detail worth carrying into a design review is that an API request is not a unit of inference cost. Internal search passes, cache eligibility and accumulated schemas can dominate the invoice without adding another visible HTTP call. Google’s detailed documentation makes that boundary unusually explicit. Teams should put it in their telemetry before celebrating lower input-token counts.
For a small catalog, I would keep static loading as the default until a controlled comparison beats it. For a large one, I would start with the long tail, test the second turn as carefully as the first, and approve a rollout only when task accuracy, end-to-end latency and the complete bill improve together.




