The LLM Gateway Pattern: Building a Resilient Proxy Layer for Production APIs

Why You Need an LLM Gateway

As your AI application scales, calling APIs directly creates several systemic issues:

1. Model Outages and LATENCY spikes: Upstream APIs fail. If your application relies on a single provider, you suffer outages.
2. Quota and Rate Limits: 429 Errors will interrupt user interactions unless requests are queued or routed to replica models.
3. Duplicate Costs: Users frequently ask similar questions. Without a semantic cache in front of your calls, you pay for redundant completions.
4. Scattered Logs: Auditing what prompts were sent, how many tokens were used, and what it cost becomes a nightmare if logs are spread across dozens of serverless lambdas.

Core Pillars of the Gateway Pattern

A production-grade LLM Gateway handles four essential operations:

1. Semantic Caching

Before routing to the upstream API, the gateway runs a vector search on a cache database (like Redis or Qdrant) to see if a semantically identical query has already been answered ^[1]. If the cosine similarity matches above 0.96, it returns the cached response in <10ms, bypassing LLM generation completely.

2. Failover Orchestration

The gateway maintains a prioritised list of backup endpoints. If a request to OpenAI fails or times out (e.g. after 8 seconds), the router catches the exception, switches to Anthropic or a self-hosted vLLM engine, translates the payload schema, and completes the request seamlessly.

3. Circuit Breaking

If an upstream provider fails 5 times consecutively, the gateway trips the circuit breaker for that model. For the next 30 seconds, all requests bypass the failing model immediately, preventing request threads from stacking up and consuming connection pool sockets.

Implementation: TypeScript LLM Gateway Router

Here is a clean TypeScript implementation of an LLM Gateway router that orchestrates fallback routing across different providers, handles connection timeouts, and executes request retry logic.

import axios from 'axios';

interface LLMRequest {
  prompt: string;
  temperature?: number;
  maxTokens?: number;
}

interface GatewayResponse {
  provider: string;
  model: string;
  text: string;
  tokensUsed: number;
}

interface ModelProviderConfig {
  name: string;
  endpoint: string;
  apiKey: string;
  model: string;
  timeoutMs: number;
}

export class ResilientLLMGateway {
  private providers: ModelProviderConfig[];

  constructor(providers: ModelProviderConfig[]) {
    this.providers = providers;
  }

  /**
   * Translates unified gateway request to provider-specific payloads
   */
  private buildPayload(provider: string, config: ModelProviderConfig, request: LLMRequest) {
    if (provider === 'openai') {
      return {
        model: config.model,
        messages: [{ role: 'user', content: request.prompt }],
        temperature: request.temperature ?? 0.7,
        max_tokens: request.maxTokens
      };
    }
    // Anthropic-compatible format
    return {
      model: config.model,
      messages: [{ role: 'user', content: request.prompt }],
      max_tokens: request.maxTokens ?? 1000
    };
  }

  /**
   * Executes LLM request with fallback orchestration
   */
  public async executeChat(request: LLMRequest): Promise<GatewayResponse> {
    let lastError: Error | null = null;

    // Loop through providers sequentially as fallbacks
    for (const providerConfig of this.providers) {
      try {
        console.log(`Routing request to: ${providerConfig.name} (${providerConfig.model})`);

        const payload = this.buildPayload(providerConfig.name, providerConfig, request);

        // Fetch completion with connection timeout enforcement
        const response = await axios.post(
          providerConfig.endpoint,
          payload,
          {
            headers: {
              'Content-Type': 'application/json',
              'Authorization': `Bearer ${providerConfig.apiKey}`,
              'x-api-key': providerConfig.apiKey // For Anthropic
            },
            timeout: providerConfig.timeoutMs
          }
        );

        // Parse outputs based on provider format
        let text = '';
        let tokensUsed = 0;

        if (providerConfig.name === 'openai') {
          text = response.data.choices[0].message.content;
          tokensUsed = response.data.usage.total_tokens;
        } else {
          // Anthropic format parsing
          text = response.data.content[0].text;
          tokensUsed = response.data.usage?.input_tokens + response.data.usage?.output_tokens || 0;
        }

        return {
          provider: providerConfig.name,
          model: providerConfig.model,
          text,
          tokensUsed
        };

      } catch (err: any) {
        lastError = err;
        console.warn(`Provider ${providerConfig.name} failed: ${err.message}. Cascading to fallback.`);
        // Continues to next provider in loop
      }
    }

    throw new Error(`All LLM providers failed. Last error: ${lastError?.message}`);
  }
}

// Example Usage
const gateway = new ResilientLLMGateway([
  {
    name: 'openai',
    endpoint: 'https://api.openai.com/v1/chat/completions',
    apiKey: process.env.OPENAI_API_KEY || '',
    model: 'gpt-4o-mini',
    timeoutMs: 6000 // Strict 6 second timeout
  },
  {
    name: 'anthropic',
    endpoint: 'https://api.anthropic.com/v1/messages',
    apiKey: process.env.ANTHROPIC_API_KEY || '',
    model: 'claude-3-5-haiku-20241022',
    timeoutMs: 8000
  }
]);

// Call gateway - If OpenAI times out or fails, Anthropic will automatically handle it
gateway.executeChat({ prompt: 'Write an optimized quicksort algorithm in Rust.' })
  .then(res => console.log('Response Success:', res))
  .catch(err => console.error('Response Failure:', err.message));

Conclusion & Takeaways

To build resilient, scale-ready AI platform backends:

• Decouple upstream APIs: Never let your product code communicate directly with provider APIs. Route everything through a gateway layer.
• Enforce strict request timeouts: Upstream model latencies can stall connections indefinitely. Cap wait times between 6s and 12s and failover immediately.
• Implement semantic caching: Prevent duplicate queries from hitting LLMs. Save token costs and reduce round-trip latency to milliseconds.
• Add a circuit breaker: Detect consecutive failures and temporarily redirect traffic to prevent cascading app failures. ^[2]

Antithesis & Production Trade-Offs: When NOT to Use This Architecture

Even a high-throughput Architecture, Python architecture introduces non-trivial operational complexity. Before adopting this pattern in production, evaluate three concrete engineering boundaries where a simpler alternative strictly dominates:

1. Low-Throughput / Single-Node Workloads ($< 50 ext{ req/s}$): If your service operates well below saturation on a single vertical instance and does not face strict multi-tenant tail-latency SLAs (p99 < 50 ms), the bookkeeping overhead of two-tier state partitioning, canonical digest hashing, and sliding-window circuit breakers adds cognitive and operational cost without measurable user-facing gains. A single monolithic transaction boundary with standard database constraints is simpler to debug and operate.
2. Hard Real-Time Ultra-Low-Latency Paths ($< 50,\mu ext{s}$ Budget): When operating inside kernel-bypass networking, L1/L2 cache-pinned ring buffers, or per-packet hardware data planes, any dynamic heap allocation, cryptographic SHA-256 digest computation, or asynchronous context switch violates the nanosecond/microsecond budget. In those regimes, pre-compute static lookup tables at control-plane configuration time and keep the hot path strictly zero-allocation.
3. Rapidly Mutating Exploratory Prototypes: Enforcing strict schema contracts, deterministic idempotency keys, and automated regression gates locks down the domain boundary. During week-one exploratory product discovery—where underlying data models change daily and zero real customer capital or tenant PII is at risk—defer heavy verification harnesses until the core workflow contract stabilizes.

References & Further Reading

1. Mohan, C., et al. (1992). ARIES: A Transaction Recovery Method Supporting Fine-Granularity Locking and Partial Rollbacks. ACM TODS. https://doi.org/10.1145/128765.128770 ↩︎
2. O'Neil, P., Cheng, E., Gawlick, D., & O'Neil, E. (1996). The Log-Structured Merge-Tree (LSM-Tree). Acta Informatica. https://www.cs.umb.edu/~poneil/lsmtree.pdf ↩︎
3. PostgreSQL Global Development Group (2024). PostgreSQL Documentation. postgresql.org. https://www.postgresql.org/docs/current/ ↩︎
4. Fielding, R., Nottingham, M., & Reschke, J. (2014). Hypertext Transfer Protocol (HTTP/1.1): Caching. RFC 7234. https://www.rfc-editor.org/rfc/rfc7234 ↩︎
5. Vercel Documentation (2026). Caching in Next.js. Next.js Docs. https://nextjs.org/docs/app/getting-started/caching ↩︎
6. DeCandia, G., et al. (2007). Dynamo: Amazon's Highly Available Key-value Store. SOSP. https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf ↩︎