What an LLM gateway is, and why you need one

Picture a company six months into using LLMs. The recommendation service calls OpenAI directly. The support bot calls Anthropic. A data team is on a self-hosted model. Every one of them has its own API key pasted into its own config, its own retry logic, its own idea of what to do when a provider returns a 429. Nobody can answer "what are we spending on models this month?" and nobody can swap providers without a code change in five repos. A gateway fixes this the same way it was fixed for web services a decade ago: put one thing in front of everything.

One endpoint, five jobs

An LLM gateway is a service your apps talk to instead of talking to model providers directly. It exposes one stable, usually OpenAI-compatible, API and does the plumbing behind it:

• Routing — pick a model/provider per request. Send cheap traffic to a small model, sensitive traffic to a self-hosted one, and switch providers by editing config, not code.
• Caching — if two requests ask the same thing, serve the second from cache. With token-metered pricing, a cache hit is money you didn't spend.
• Quotas & budgets — per-team keys with spend limits and rate limits, so one runaway loop can't burn the monthly budget or trip a provider's rate cap for everyone.
• Failover — when the primary provider errors or throttles, retry against a fallback automatically, so a single vendor outage isn't your outage.
• Observability — one place that logs every call: latency, tokens, cost, which app, which model. This is the answer to "what are we spending, and where?"

# One config the gateway reads; apps never see provider keys.
model_list:
  - model_name: chat-default          # what apps ask for
    litellm_params: {model: openai/gpt-4o, api_key: os.environ/OPENAI_KEY}
  - model_name: chat-default          # same alias → load-balanced / failover
    litellm_params: {model: anthropic/claude-3-5-sonnet, api_key: os.environ/ANTHROPIC_KEY}

# App code stays boring and provider-agnostic:
#   POST https://gateway.internal/v1/chat/completions
#   { "model": "chat-default", "messages": [...] }

It's an old pattern in new clothes

None of this is novel. The API gateway has been standard architecture for years: a single front door that handles auth, throttling, monitoring, and caching for a fleet of backend services. Amazon's API Gateway, for instance, bills itself as a "front door" that manages traffic, authorization, monitoring, and access control at scale. An LLM gateway is that pattern retargeted at model providers — with two twists that matter. First, the "backends" are external, metered, and flaky, so cost routing and failover stop being nice-to-haves. Second, the unit of billing is tokens, not requests, which makes caching and per-team budgets financially load-bearing rather than cosmetic. Open-source gateways such as LiteLLM package exactly these features — routing, fallbacks, caching, budgets, and logging — behind one OpenAI-compatible endpoint.

Why it matters: as soon as LLM calls become infrastructure rather than a demo, you need the same governance you'd demand of any other dependency — cost control, failover, and an audit trail. A gateway is where that lives. Skip it and you get shadow spend, five copies of retry logic, and a vendor outage that takes your product down with it. It's the least glamorous box in the AI stack and one of the first you'll wish you had.