Streaming AI Chat Responses with Server-Sent Events

Waiting eight seconds for a full LLM answer feels broken. Seeing the first words after 400 ms feels fast — even if the total time is identical. On the AI tutoring platform I work on, chat answers are streamed for exactly this reason.

This post walks through the setup: a FastAPI endpoint that streams model output as Server-Sent Events (SSE), and a Next.js client that renders it incrementally.

Why SSE and not WebSockets?

Chat streaming is one-directional: the user sends a message, the server streams an answer back. SSE fits that exactly:

• It's plain HTTP, so auth cookies, CORS and load balancers work as usual.
• The wire format is trivial text: data: ... lines separated by blank lines.
• No connection state to manage on the server between messages.

WebSockets earn their complexity when both sides push messages continuously — collaborative editing, live presence, games. A chat answer isn't that.

The FastAPI side

The endpoint returns a StreamingResponse with the text/event-stream media type. Each chunk from the model becomes one SSE event:

import json
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from google import genai

app = FastAPI()
client = genai.Client()

def sse(event: str, data: dict) -> str:
    return f"event: {event}\ndata: {json.dumps(data)}\n\n"

@app.post("/chat")
async def chat(request: Request, body: ChatRequest):
    async def stream():
        try:
            response = await client.aio.models.generate_content_stream(
                model="gemini-2.5-flash",
                contents=body.message,
            )
            async for chunk in response:
                if await request.is_disconnected():
                    break
                if chunk.text:
                    yield sse("token", {"text": chunk.text})
            yield sse("done", {})
        except Exception:
            yield sse("error", {"message": "Generation failed"})

    return StreamingResponse(
        stream(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",
        },
    )

Three details matter here:

1. JSON-encode every payload. Model output contains newlines, and a raw newline inside data: breaks the SSE framing. json.dumps escapes them.
2. Check for disconnects. If the user navigates away, the generator would otherwise keep pulling tokens you pay for.
3. Send explicit done and error events. The client needs to tell "finished" apart from "the connection dropped".

The buffering trap

A common surprise: streaming works locally, then arrives all at once in production. The usual culprit is something between the server and the browser buffering the response.

• Nginx buffers upstream responses by default. The X-Accel-Buffering: no header turns that off per response.
• Compression middleware (gzip) can also hold bytes back until it has enough to compress. Exclude text/event-stream from it.
• If you proxy through a Next.js Route Handler, return the upstream body stream directly instead of awaiting .text().

If streaming works on localhost but not in production, it's almost always a buffer somewhere in between.

The Next.js client

The browser's built-in EventSource only supports GET and can't send a JSON body, so I read the stream with fetch and parse the events manually:

export async function streamChat(
  message: string,
  onToken: (text: string) => void,
  signal?: AbortSignal,
) {
  const res = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ message }),
    credentials: "include",
    signal,
  });
  if (!res.ok || !res.body) throw new Error(`Chat failed: ${res.status}`);

  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  let buffer = "";

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += value;

    const events = buffer.split("\n\n");
    buffer = events.pop() ?? "";

    for (const raw of events) {
      const event = raw.match(/^event: (.*)$/m)?.[1];
      const data = raw.match(/^data: (.*)$/m)?.[1];
      if (!data) continue;
      const payload = JSON.parse(data);
      if (event === "token") onToken(payload.text);
      if (event === "error") throw new Error(payload.message);
    }
  }
}

The important part is the buffer: a network chunk can end in the middle of an event, so only complete events (terminated by \n\n) are parsed and the rest waits for the next read.

In the component, append tokens to state and pass an AbortController signal so a "Stop generating" button — or unmounting — cancels the request. That abort is what triggers request.is_disconnected() on the server.

Rendering without jank

Appending a token to state on every chunk re-renders the message dozens of times per second. That's fine for plain text, but if you render Markdown, re-parsing the whole message each time gets expensive on long answers. Two things helped:

• Batch updates with requestAnimationFrame, so you render at most once per frame.
• Memoize finished messages, so only the one currently streaming re-renders.

Takeaways

• SSE is the right default for LLM streaming: plain HTTP, simple format, works with your existing auth.
• JSON-encode payloads and send explicit done/error events.
• Stop generating when the client disconnects.
• If it streams locally but not in production, look for a buffering proxy.

Streaming doesn't make the model faster. It makes the wait visible and short, and users notice that more than any backend optimization.