Streaming LLM Responses in Next.js: SSE and Cancellation

The short answer: To stream LLM responses in Next.js, return a ReadableStream from an App Router route handler with Content-Type: text/event-stream, forward each chunk from your model provider as a Server-Sent Event, and read it in the browser with fetch and a stream reader (not EventSource, which can't send a POST body). Then handle the parts that break in production: pass the request's abort signal to the provider so cancelled requests stop costing money, disable proxy buffering, send keep-alive comments, report errors as events once the 200 has been sent, and raise your function's maximum duration.

Streaming is the single cheapest improvement you can make to how an AI feature feels. A response that takes eight seconds to generate feels broken if the user stares at a spinner for eight seconds. The same response feels fast if the first words appear after half a second and the rest flows in as it's written.

The basic version is easy, and every tutorial shows it. The production version is where teams lose days: responses that arrive all at once despite "streaming," tokens that keep generating (and billing) after the user closes the tab, errors that leave the interface hanging forever, and functions that get killed halfway through a long answer. This guide covers the basic version and then all of those problems, using the Next.js App Router and plain web standards so the ideas carry over to any provider and most frameworks.

The case for streaming

Language models generate text one token at a time. Without streaming, your server waits for the whole response, then sends it in one piece. The user waits for the slowest part of the process before seeing anything.

With streaming, each piece is forwarded as soon as it's generated. Total generation time doesn't change, but time to first token, the delay before the user sees anything, drops dramatically. That's the number people perceive as speed. It also enables things that are impossible otherwise: a Stop button that actually stops generation, showing a long answer's first paragraph while the rest is still being written, and displaying progress during multi-step agent work.

The trade-off is complexity. A streaming response is a long-lived connection with partial state, and every layer between the model and the browser (your function, your proxy, your CDN, your client code) has to cooperate.

Choosing a transport

You have three realistic options for getting tokens to the browser:

Transport	Good for	Drawbacks
Server-Sent Events over `fetch`	Most chat and generation features: server-to-client streaming over plain HTTP	One direction only; needs manual parsing on the client
Raw text stream	The simplest possible case, plain text only	No room for metadata, tool events or errors without inventing a format
WebSockets	Genuinely two-way, real-time interaction such as voice or collaborative editing	More infrastructure; harder on serverless platforms

For most AI features, Server-Sent Events (SSE) is the right default. It's plain HTTP, works through most proxies once configured correctly, and its simple format lets you send different kinds of events (text, tool activity, usage, errors) on one connection. Most LLM provider APIs stream to you using SSE as well, so you're speaking the same language end to end.

One detail trips people up: the browser's built-in EventSource API only makes GET requests and can't send a request body or custom headers. Chat requests need to send the conversation, so you'll use fetch with a POST body and parse the SSE stream yourself. It's about thirty lines of code, shown below.

The SSE format in two minutes

An SSE stream is text. Each event is a few lines followed by a blank line. The HTML specification defines the format, and MDN has a readable guide. The parts you need:

event: delta
data: {"text":"Hello"}

event: delta
data: {"text":" world"}

: keep-alive comment, ignored by clients

event: done
data: {"inputTokens":812,"outputTokens":164}

Lines starting with data: carry the payload, event: names the event type, and a line starting with a colon is a comment, which is useful as a keep-alive. Sending JSON in data lets you include structured information with each event.

The server: a streaming route handler

In the App Router, a route handler can return any web-standard Response, including one whose body is a ReadableStream. The pattern below is provider-agnostic: streamModel stands for your provider SDK's streaming call, which typically yields text deltas as an async iterable. The Anthropic and OpenAI SDKs both provide helpers for this; see, for example, Anthropic's streaming documentation.

// app/api/chat/route.ts
import { streamModel } from "@/lib/llm"; // wraps your provider SDK

export const maxDuration = 60; // seconds; raise if long answers get cut off

const encoder = new TextEncoder();
const sse = (event: string, data: unknown) =>
  encoder.encode(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = new ReadableStream({
    async start(controller) {
      // Comments keep idle connections from being closed by proxies.
      const keepAlive = setInterval(() => controller.enqueue(encoder.encode(": ping\n\n")), 15_000);
      try {
        // Pass the request's abort signal so generation stops when the user leaves.
        const response = streamModel({ messages, signal: request.signal });
        for await (const chunk of response) {
          if (chunk.type === "text") controller.enqueue(sse("delta", { text: chunk.text }));
          if (chunk.type === "usage") controller.enqueue(sse("usage", chunk.usage));
        }
        controller.enqueue(sse("done", {}));
      } catch (err) {
        if (!request.signal.aborted) {
          // Headers (and the 200 status) are already sent: report errors as an event.
          controller.enqueue(sse("error", { message: "The response could not be completed. Please try again." }));
          console.error("stream failed", err);
        }
      } finally {
        clearInterval(keepAlive);
        try { controller.close(); } catch { /* already closed by a disconnect */ }
      }
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      Connection: "keep-alive",
      "X-Accel-Buffering": "no", // tells nginx-style proxies not to buffer
    },
  });
}

Several details in there are doing real work, and each one corresponds to a production problem covered in the sections that follow: the abort signal, the keep-alive comments, error events, the headers and maxDuration.

The client: reading the stream

On the client, use fetch, read the body with a stream reader, and split the text into events. The parser has to handle events that arrive split across chunks, because network boundaries don't line up with event boundaries.

export async function streamChat(
  messages: { role: string; content: string }[],
  onEvent: (event: string, data: any) => void,
  signal: AbortSignal,
) {
  const res = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
    signal,
  });
  if (!res.ok || !res.body) throw new Error(`Request failed: ${res.status}`);

  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  let buffer = "";
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += value;
    // Events end with a blank line; keep any partial event in the buffer.
    const events = buffer.split("\n\n");
    buffer = events.pop() ?? "";
    for (const raw of events) {
      if (raw.startsWith(":")) continue; // keep-alive comment
      const event = raw.match(/^event: (.*)$/m)?.[1] ?? "message";
      const data = raw.match(/^data: (.*)$/m)?.[1];
      if (data) onEvent(event, JSON.parse(data));
    }
  }
}

This simple parser assumes one data: line per event, which holds because the server JSON-encodes each payload onto a single line. If you consume streams from elsewhere, use a full SSE parser that handles multi-line data and other fields.

In a React component, keep an AbortController per request. The Stop button calls controller.abort(), and so does cleanup when the component unmounts or the user starts a new message.

Production problem 1: tokens keep generating after the user leaves

This is the expensive one. A user asks a question, gets bored, and closes the tab. Without cancellation, your server keeps reading from the provider until the response is complete, and you pay for every token nobody will ever read. On features with long outputs and impatient users, that waste adds up quickly.

The fix is to propagate cancellation all the way through:

The browser aborts the fetch (Stop button, navigation, new message).
The server sees the client disconnect. In recent Next.js versions, the route handler's request.signal fires when the client goes away. Verify this on your hosting platform, because proxies and runtimes differ.
The server passes that signal to the provider SDK call, which closes the upstream connection so generation stops.

Test it explicitly: start a long generation, close the tab after a second, and check your provider usage or your logs to confirm that output tokens stopped. If they didn't, a layer in between isn't passing the signal along.

Production problem 2: the "stream" arrives all at once

You've written streaming code, it works locally, and in production the whole answer appears in one go after eight seconds. Something between your function and the browser is buffering: collecting the response before forwarding it.

Common culprits:

Reverse proxies. Nginx buffers proxied responses by default (see proxy_buffering in the nginx proxy module docs). The X-Accel-Buffering: no response header disables it per response; you can also turn buffering off for the route in the proxy config.
Compression. Some compression middleware waits for enough data to compress efficiently. Exclude text/event-stream responses from compression, or make sure your compressor flushes.
CDNs and caches. Make sure the route isn't cached. Cache-Control: no-cache, no-transform helps, and the no-transform part discourages intermediaries from altering the response.
Your own code. Accidentally awaiting the whole provider response before writing to the stream, or reading response.text() somewhere in a wrapper, silently turns streaming back into a single response.

A quick diagnostic is curl -N against your endpoint from outside your network. If events arrive gradually in curl but not in the browser, look at the client; if they arrive all at once in curl too, look at the server and anything in front of it.

Production problem 3: errors after the response has started

With a normal response, an error becomes a 500 status and the client knows something went wrong. With streaming, the 200 status and headers go out with the first byte. If the provider fails halfway through (a rate limit, an overload, a network blip), you can't change the status any more.

So errors have to travel inside the stream, as an event the client understands. That's why the route handler above sends an error event. The client should then show a clear message, keep whatever partial text arrived (marked as incomplete), and offer a retry.

The worst failure mode is silence: the server crashes, the stream ends without a done or error event, and the interface sits with a blinking cursor forever. Guard against it on the client too. If the stream ends without a done event, treat it as an error.

Production problem 4: long responses get cut off

Serverless functions have a maximum duration. If an answer takes longer than the limit, the platform stops the function mid-stream and the user gets half an answer. On Vercel you can raise the limit per route with the maxDuration export shown above, up to what your plan allows; see Vercel's function duration docs. Other platforms have equivalents.

Raising the limit isn't always enough for agents that may run for minutes. For genuinely long tasks, a better design is to start the work as a background job, return a job ID immediately, and stream progress from a separate endpoint that the client can reconnect to. That also survives the user refreshing the page, which a single long-lived stream does not.

Idle connections are a related problem. Some proxies close connections that send nothing for a while, which can happen while a model is "thinking" before its first token. The keep-alive comments in the route handler prevent that.

Retries and interrupted streams

Networks drop. Phones switch from Wi-Fi to mobile data, laptops go to sleep, and users walk into lifts. A non-streaming request that fails can simply be retried. A streaming request that fails halfway is more awkward, because part of the answer has already been shown.

For most chat features, the simplest correct behaviour is also the best one: keep the partial answer on screen, mark it clearly as incomplete, and offer a "Retry" button that regenerates the response from scratch. Don't try to stitch a new generation onto the end of the old partial one. The model will produce a different continuation, and the seam is usually obvious.

Automatic retries are worth doing only before the first token arrives. If the connection fails while you're still waiting for the response to start, perhaps because of a provider rate limit or a cold start that timed out, retrying once or twice with a short backoff is invisible to the user and usually succeeds. Once text is on screen, let the user decide.

If you need answers to survive a page refresh or a dropped connection, for example for long agent tasks or reports, stream from a stored result instead of directly from the model. The server writes generated text to durable storage as it arrives, and the client reads from that store. SSE has a built-in mechanism that helps here: each event can carry an id, and a reconnecting client can tell the server the last ID it received so the server resumes from that point. With fetch-based streaming you implement the same idea yourself, by sending the last received position when reconnecting. This is more machinery than a typical chat feature needs, so reserve it for features where losing a long response would genuinely hurt.

Whatever you choose, make sure the server knows when a client gave up. Log cancellations and broken connections alongside completed responses. If many users abandon responses at a similar point, your answers may be too long or too slow, and that's a product problem rather than a networking one.

Testing streaming code

Streaming code is easy to break in ways that ordinary tests miss, because most tests wait for the whole response and never notice that it arrived all at once, or never arrived in pieces at all. A few targeted tests catch the important failures.

Parser tests with awkward chunk boundaries. Feed your client-side parser the same stream split at every possible position: in the middle of an event, in the middle of a JSON payload, between the two newlines that end an event. The output should be identical every time. This one test catches the most common client bug.
A fake model stream. Replace the provider call with a generator that yields known chunks with small delays, then a usage event. Assert that the route handler emits the right sequence of SSE events, ending in done.
Failure mid-stream. Make the fake generator throw after a few chunks. Assert that the client receives an error event, that the partial text is preserved, and that the interface leaves the loading state.
Cancellation. Abort the client request partway through and assert that the fake generator saw the abort signal and stopped. This guards the most expensive production bug.
A smoke test against the deployed environment. After each deploy, request a deliberately slow response through the real infrastructure and measure the time to the first event. If it's close to the total time, something in front of your function has started buffering again.

None of these require calling a real model, so they're fast and cheap enough to run on every pull request. The deployed smoke test is the one that catches configuration changes in proxies and CDNs, which no unit test can see.

Rendering streamed text well

Getting tokens to the browser is half the job; displaying them nicely is the other half.

Batch updates. Updating React state on every token can mean dozens of re-renders a second. Accumulate text in a ref and flush to state on an animation frame or every few dozen milliseconds. The text still looks continuous, and the page stays responsive.
Render partial Markdown carefully. Mid-stream, the Markdown is incomplete: an unclosed code block or a half-written table can make the whole message jump around. Markdown renderers that tolerate incomplete input help, as does treating an unclosed code fence as still open.
Don't fight the user's scrolling. Auto-scroll to the bottom only if the user is already near the bottom. If they've scrolled up to read, leave them there.
Make Stop obvious. A visible Stop button that actually cancels generation (see problem 1) is a feature users notice.
Be accessible. Screen readers can be overwhelmed by text changing many times a second. Consider announcing the response once it's complete rather than on every token, using an appropriately configured live region.

Streaming structured output and tool calls

Not everything worth streaming is prose. When a model returns structured data or calls tools, the stream contains partial JSON. You can display progress (for example, "searching orders…" when a tool call starts) but don't act on partial structured data. Parse and validate it only when the corresponding block is complete. The guide to reliable JSON from LLMs covers validation, and the same rules apply to streamed output.

For agents, it often makes sense to stream higher-level events rather than raw tokens: "planning," "calling search_orders," "reading 3 results," "writing answer." The SSE event names make that easy, and users find it far more reassuring than a long silence followed by a wall of text.

Measuring streaming quality

Track two latency numbers separately: time to first token and total time. Time to first token is what users feel, and it's affected by things total time hides: long prompts, slow retrieval before the model call, cold starts, and queueing at the provider. Also track how often streams end with an error event, end without any final event, or are cancelled by the user. A rising cancellation rate is a useful signal that responses are too slow or too long. The guide on LLM observability covers how to capture these per request.

Production checklist

☐ Route handler returns a ReadableStream with text/event-stream and no-cache, no-transform
☐ Client uses fetch plus a stream reader, with a parser that handles split events
☐ An AbortController cancels requests on Stop, new message and unmount
☐ request.signal is passed to the provider call, and cancellation is verified in production
☐ Proxy buffering and compression are disabled for streaming routes
☐ Keep-alive comments prevent idle connections from being closed
☐ Errors are sent as events; a stream that ends without done is treated as a failure
☐ maxDuration fits your longest realistic response; very long tasks run as background jobs
☐ UI batches updates, handles partial Markdown and respects the user's scroll position
☐ Time to first token, total time, errors and cancellations are tracked separately

Frequently asked questions

How do I stream an LLM response in Next.js?

Create an App Router route handler that returns a Response whose body is a ReadableStream, with the Content-Type set to text/event-stream. Inside the stream, iterate over your provider SDK's streaming response and enqueue each text chunk as a Server-Sent Event. On the client, call the route with fetch and read the body with a stream reader.

Should I use Server-Sent Events or WebSockets for AI chat?

Server-Sent Events are the better default for most AI chat because responses flow in one direction, from server to client, over plain HTTP that works well with serverless platforms. WebSockets are worth the extra infrastructure when you need genuinely two-way, real-time interaction, such as live voice or collaborative editing.

Why is my streaming response arriving all at once?

Usually because something buffers it: a reverse proxy such as nginx, compression middleware, a CDN, or code that waits for the full provider response before writing. Disable proxy buffering for the route (for example with an X-Accel-Buffering: no header), exclude event streams from compression, prevent caching, and check with curl -N where the buffering happens.

How do I stop LLM generation when the user cancels?

Abort the fetch on the client with an AbortController, and on the server pass the route handler's request.signal to your provider SDK call, so the upstream request is closed when the client disconnects. Then confirm in production that output tokens stop after a cancellation, since proxies and runtimes can interfere with disconnect signals.

Why can't I use EventSource for my chat endpoint?

EventSource only makes GET requests and can't send a request body or custom headers, but a chat request needs to send the conversation. Use fetch with a POST body instead and parse the Server-Sent Events stream from the response body yourself, which takes only a small amount of code.

The takeaway

Streaming turns a slow-feeling AI feature into a fast-feeling one at almost no extra model cost. The happy path takes an afternoon. The production path is a checklist: cancel what nobody is reading, stop intermediaries from buffering, report errors inside the stream, plan for long responses, and render partial text gracefully. Work through it once, wrap it in a small shared module, and every AI feature you build afterwards gets it for free.

Streaming LLM Responses in Next.js: SSE, Cancellation and Production Pitfalls