LLM Gateways & Cost Optimization

What is an LLM gateway?

An LLM gateway is a proxy that sits between your application and one or more model providers (OpenAI, Anthropic, Google, AWS Bedrock, your own vLLM replicas). It gives you a single OpenAI-compatible endpoint, then handles the messy production concerns: routing each request to the cheapest capable model, fallbacks when a provider is down, caching to skip redundant calls, cost tracking per team or user, and rate limiting to prevent budget overruns. Without a gateway, every one of these is bespoke code you maintain in your app.

The three options that matter in 2026

LiteLLM is fully open-source and self-hostable — one proxy that speaks to 100+ providers with a unified API. It is the default for teams that want control and zero per-request markup. Portkey is a managed SaaS (with a self-host option) focused on enterprise-grade observability, RBAC, and governance. OpenRouter is managed-only — the fastest to start with (one API key, every model) but it charges a per-request markup on top of provider pricing.

Why cost optimization lives at the gateway

According to Pluralsight's 2026 analysis, 31% of enterprise LLM queries are redundant — identical or near-identical to a previous call. A gateway with semantic caching eliminates those before they reach the provider. Combined with model routing (sending easy queries to GPT-4o-mini instead of GPT-4o), published benchmarks show total API spend dropping 50-85%. These are not theoretical savings — they are the baseline expectation for production LLM billing in 2026.

Related clusters

• Inference Engines — vLLM, SGLang, TGI for self-hosted models (the gateway's downstream)
• KV Cache Quantization — compress the cache that makes self-hosted serving affordable
• Speculative Decoding — 2-3x throughput on self-hosted inference