Cutting AI Chat Latency — SSE Streaming and TTFT Optimization

AI LLM SSE web-development
The short version

The single highest-impact latency fix for an AI chat is moving from a batch-JSON API to Server-Sent Events (SSE) streaming — cutting the time to the first character from seconds down to a few hundred milliseconds.

The factor that most decides user satisfaction in an AI chat is not how fast the whole reply finishes, but how fast the first character appears. A batch design that makes users wait seconds to tens of seconds for the full generation destroys perceived speed.

Perceived-speed comparison with and without streaming

How batch (JSON) and streaming (SSE) differ in performance and behavior.

DimensionBatch response (JSON)SSE streaming
TTFT (Time To First Token)3.0s – 15.0s (no visual change until done)0.2s – 0.8s (characters paint immediately)
HTTP headerContent-Type: application/jsonContent-Type: text/event-stream
TransportNormal HTTP POST / GETHTTP/1.1 (Keep-Alive) or HTTP/2
Server resourcesHolds full text in memory until doneSends and discards each chunk immediately
Mid-stream disconnectThe whole result is lostReceived text up to the cut remains

Implementing SSE streaming

The steps to add SSE streaming.

  1. Build a response with the Content-Type: text/event-stream header on the server.
  2. Receive the response chunk-by-chunk on the client with the Fetch API and ReadableStreamReader.
  3. Append decoded text to UI state in real time and keep the scroll position in sync.

1. Streaming response on the server

Return the Content-Type: text/event-stream header from a Cloudflare Pages Function or a Node.js backend.

// Server side (Cloudflare Worker / Pages Function example)
export async function onRequestPost(context) {
  const { readable, writable } = new TransformStream();
  const writer = writable.getWriter();
  const encoder = new TextEncoder();

  // Issue a streaming request to the LLM API (background task)
  (async () => {
    try {
      const llmStream = await callLlmStreamApi(context.env);
      for await (const chunk of llmStream) {
        const text = chunk.text || "";
        // Format as SSE and write
        await writer.write(encoder.encode(`data: ${JSON.stringify({ text })}\n\n`));
      }
      await writer.write(encoder.encode("data: [DONE]\n\n"));
    } catch (err) {
      await writer.write(encoder.encode(`data: ${JSON.stringify({ error: "Stream error" })}\n\n`));
    } finally {
      await writer.close();
    }
  })();

  return new Response(readable, {
    headers: {
      "Content-Type": "text/event-stream; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      "Connection": "keep-alive",
    },
  });
}

2. ReadableStream receive on the frontend

The EventSource API can’t send POST requests or custom headers, so receive with the fetch API and ReadableStreamReader instead.

async function streamChatResponse(message: string, onChunk: (text: string) => void) {
  const response = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ message }),
  });

  if (!response.ok || !response.body) {
    throw new Error(`HTTP error: ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder("utf-8");
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split("\n\n");
    buffer = lines.pop() || ""; // Keep the incomplete trailing line in the buffer

    for (const line of lines) {
      if (line.startsWith("data: ")) {
        const dataStr = line.slice(6).trim();
        if (dataStr === "[DONE]") return;
        
        try {
          const parsed = JSON.parse(dataStr);
          if (parsed.text) {
            onChunk(parsed.text);
          }
        } catch (e) {
          // Handle JSON parse errors
        }
      }
    }
  }
}

3. Painting to the UI and scroll control

Append the received text to component state (React / Astro) and auto-scroll to the bottom.

// Client-side call example
let currentResponse = "";

await streamChatResponse("Hello", (chunkText) => {
  currentResponse += chunkText;
  updateChatUi(currentResponse); // Update the message on screen
  scrollToBottom();              // Follow the scroll
});

Failure boundary — real incidents and fixes

The typical failure patterns when running streaming, and their fixes.

[Sample error log]: TypeError: Failed to execute 'read' on 'ReadableStreamDefaultReader': ReadableStream is locked
[Root cause]: A re-request was issued without releasing the reader, or the same stream was read in multiple loops.

Buffering that breaks streaming

If buffering is enabled on Cloudflare / Nginx or a reverse proxy, chunks sit at the proxy until a certain size accumulates — reproducing the same delay as a batch response.

  • Fix: Always add X-Accel-Buffering: no (Nginx) and Cache-Control: no-cache to the response headers.

Browser mojibake (multi-byte characters cut mid-way)

UTF-8 Japanese characters take 3–4 bytes each. When the LLM’s byte stream boundary lands mid-character, decoding produces mojibake.

  • Fix: Initialize TextDecoder("utf-8") and pass the decoder.decode(value, { stream: true }) option so partial byte sequences are buffered correctly across boundaries.