Overview of model routing

Model routing for API Gateway is a managed traffic management layer that accepts OpenAI-compatible prompt requests, transcodes them in-flight, and routes them to specific Gemini Enterprise Agent Platform models. Model routing acts as a managed alternative to client-side proxies such as LiteLLM, providing centralized infrastructure to manage the lifecycle of AI agents.

Model routing moves routing logic to the network edge and integrates with Agent Platform Model Garden for same-host optimizations. This architecture removes the requirement to host, scale, and maintain unmanaged proxy servers, reducing operational overhead and infrastructure costs.

Scope and user journeys

Model routing supports the following core user journeys:

• Model selection: An AI developer uses open models from Model as a Service (MaaS) in Agent Platform Model Garden. These are Gemini, Anthropic Claude, or OpenAI GPT family models.
• Specification authoring: An AI developer creates or updates a model router configuration within an OpenAPI 3.x specification to reference the deployed models.
• Gateway deployment: An AI developer deploys an API config and an API Gateway instance using the authored OpenAPI specification.
• Prompt routing: Client applications send OpenAI-compatible prompt requests to the gateway, which routes requests and translates payloads based on the model name specified in the JSON payload.

Future versions of API Gateway are planned to support additional user journeys.

Benefits of model routing

Implementing model routing in API Gateway provides the following advantages:

• Centralized management: Consolidate AI traffic management within a single managed gateway, replacing fragmented client-side routing configurations.