SSE vs. WebSockets: Architectural Choices for Real-Time Token Streaming

Streaming Real-Time Agent Outputs

Traditional HTTP request-response cycles are built for static content:

• The Latency Bottleneck: Waiting for a complete LLM generation before sending a response frustrates users ^[1].
• Bi-directional vs. Uni-directional Needs: Agents stream data to the client, but clients occasionally need to interrupt execution threads mid-stream.
• The Options: SSE (Server-Sent Events) and WebSockets. While SSE is simple and built on standard HTTP, WebSockets supports full-duplex, bi-directional communication channels.

1. Comparing Topologies: SSE vs. WebSockets

Choose the correct protocol based on your application's communication requirements:

• Server-Sent Events (SSE): Uses standard HTTP connections. It is uni-directional (Server to Client) and supports automatic reconnection out of the box. Ideal for streaming LLM text outputs.
• WebSockets: Establishes a persistent, bi-directional TCP socket. It is full-duplex, permitting clients to send interrupt controls or intermediate instructions to the running agent.

2. Decoupling Streaming Gateways

To scale real-time streaming:

1. Configure Keep-Alives: Ensure intermediate gateway proxies (like Nginx) do not close idle connections during planning phases.
2. Buffer Chunks: Bundle small token outputs into multi-token segments before sending to optimize network framing overhead.

Code Demo: FastAPI SSE & WebSocket Streaming Server

Below is a Python script using FastAPI simulation libraries. It routes SSE events and WebSocket connection requests to handle real-time streaming token channels.

import time
import asyncio
from typing import AsyncGenerator

# Mock FastAPI Streaming Server Simulation
class StreamingGatewayServer:
    async def simulate_sse_stream(self) -> AsyncGenerator[str, None]:
        # Server-Sent Events follow a strict "data: <content>\n\n" format
        tokens = ["Executing", " planning", " step...", " Querying", " database", " table."]
        
        for token in tokens:
            await asyncio.sleep(0.2) # Simulate model generation delay
            yield f"data: {json.dumps({'token': token})}\n\n"

        # Signal stream end
        yield "data: [DONE]\n\n"

    async def handle_websocket_session(self, ws_client_id: str):
        print(f"\n🔌 [Websocket] Established bi-directional socket for client: {ws_client_id}")
        
        # Simulating bi-directional event loop (receive interrupt and send updates)
        steps = ["Analyzing logs", "Linting files", "Executing tests"]
        
        for step in steps:
            await asyncio.sleep(0.3)
            # Send frame to client
            print(f"   📤 Sending WS Frame: {step}")
            
            # Simulate receiving client interrupt instruction
            if step == "Linting files":
                print("   📥 Received Client Interrupt: 'Abort execution loop'")
                print("   🚫 Halting WebSocket Session.")
                break

if __name__ == "__main__":
    import json
    server = StreamingGatewayServer()

    # 1. Run SSE Stream Simulation
    async def run_sse():
        print("⚡ Simulating Server-Sent Events (SSE) Stream...")
        print("-------------------------------------------------")
        async for chunk in server.simulate_sse_stream():
            print(chunk.strip())

    # 2. Run WebSocket Simulation
    async def run_ws():
        await server.handle_websocket_session("client_user_303")

    asyncio.run(run_sse())
    asyncio.run(run_ws())

Architectural Guidelines

• Default to SSE for Text: Use Server-Sent Events (SSE) as your default protocol for simple streaming text outputs to minimize connection setup overhead.
• Use WebSockets for Interactive Swarms: Implement WebSockets when you need bi-directional communication to support real-time client interrupts.
• Adjust Gateway Timeouts: Configure proxy server timeouts (e.g. proxy_read_timeout in Nginx) to match maximum planning latencies. ^[2]

Antithesis & Production Trade-Offs: When NOT to Use This Architecture

Even a high-throughput System Design, Gateways architecture introduces non-trivial operational complexity. Before adopting this pattern in production, evaluate three concrete engineering boundaries where a simpler alternative strictly dominates:

1. Low-Throughput / Single-Node Workloads ($< 50 ext{ req/s}$): If your service operates well below saturation on a single vertical instance and does not face strict multi-tenant tail-latency SLAs (p99 < 50 ms), the bookkeeping overhead of two-tier state partitioning, canonical digest hashing, and sliding-window circuit breakers adds cognitive and operational cost without measurable user-facing gains. A single monolithic transaction boundary with standard database constraints is simpler to debug and operate.
2. Hard Real-Time Ultra-Low-Latency Paths ($< 50,\mu ext{s}$ Budget): When operating inside kernel-bypass networking, L1/L2 cache-pinned ring buffers, or per-packet hardware data planes, any dynamic heap allocation, cryptographic SHA-256 digest computation, or asynchronous context switch violates the nanosecond/microsecond budget. In those regimes, pre-compute static lookup tables at control-plane configuration time and keep the hot path strictly zero-allocation.
3. Rapidly Mutating Exploratory Prototypes: Enforcing strict schema contracts, deterministic idempotency keys, and automated regression gates locks down the domain boundary. During week-one exploratory product discovery—where underlying data models change daily and zero real customer capital or tenant PII is at risk—defer heavy verification harnesses until the core workflow contract stabilizes.

References & Further Reading

1. Malkov, Y. A., & Yashunin, D. A. (2018). Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE TPAMI. https://arxiv.org/abs/1603.09320 ↩︎
2. Jégou, H., Douze, M., & Schmid, C. (2011). Product Quantization for Nearest Neighbor Search. IEEE TPAMI. https://hal.inria.fr/inria-00514462v2/document ↩︎
3. Johnson, J., Douze, M., & Jégou, H. (2019). Billion-scale Similarity Search with GPUs. IEEE Transactions on Big Data. https://arxiv.org/abs/1702.08734 ↩︎