Why streaming matters

A language model produces its answer piece by piece, yet a plain request and response hides that progress until everything is finished. The reader meanwhile stares at a spinner and wonders whether anything is happening. The OpenAI streaming guide makes this point.

When generating long outputs, waiting for a response can take time.

The cure is to stream the answer. Instead of holding the finished answer back, the server forwards each fragment as soon as it exists, and the page can show it straight away, so the reader begins reading while the rest is still being produced. That is how streaming lowers the perceived latency of a reply. This lesson covers the transport behind the streaming modes in the vendor guides cited in this lesson: server-sent events, usually shortened to SSE. By the end you should be able to read an SSE payload by eye, write a small server that produces one, and explain what the browser does when the connection breaks.

The shape of an event stream

An SSE response is an ordinary HTTP response whose body never quite ends. What makes it special is the content type and the layout of the text inside. The sentences in the rest of this section come from MDN's guide to using server-sent events.

A server-side script that sends events needs to answer with the MIME type text/event-stream.

The event stream is a simple stream of text data that must be encoded as UTF-8.

Each notification is sent as a block of plain text terminated by a pair of newlines.

Inside one block, each line has a field name, a colon and a value. The data field carries your payload.

When the EventSource receives multiple consecutive lines that begin with the prefix data:, it concatenates them, inserting a newline character between each one.

Two more fields matter for resilience. The id field labels an event, and retry suggests a delay before reconnecting.

The id is the event ID to set as the EventSource object's last event ID value.

A colon as the first character of a line of text is in essence a comment, and is ignored.

A server can send a comment line periodically to keep the connection alive.

Comments are useful because they cost the client nothing and still put bytes on the wire. If a model pauses for a long time while it thinks, a periodic comment tells every hop between server and browser that the connection is still in use.

Here is a short capture you can compare with your own output:

: stream opened

id: 0
data: {"token":"Streaming"}

id: 1
data: {"token":"keeps"}

event: done
data: {}

Read it from the top. The first block is only a comment, so no message reaches your handler. The next two blocks each carry an id and one JSON payload, and the last block names an event of its own. Because every block is separated from the next by an empty line, you can split a captured stream on blank lines and inspect the blocks one at a time.

A minimal server

The server below needs no framework. It sets the content type, then writes one event per fake token with a short delay, so you can watch the stream arrive. Notice that every event ends with a blank line, and that each one carries an id so a reconnecting client has something to report.

import { createServer } from "node:http";

const words = ["Streaming", "keeps", "the", "reader", "engaged", "while", "text", "arrives."];

createServer((req, res) => {
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
  });
  res.write(": stream opened\n\n");

  let i = 0;
  const timer = setInterval(() => {
    if (i >= words.length) {
      res.write("event: done\ndata: {}\n\n");
      clearInterval(timer);
      res.end();
      return;
    }
    res.write(`id: ${i}\ndata: ${JSON.stringify({ token: words[i] })}\n\n`);
    i += 1;
  }, 200);

  req.on("close", () => clearInterval(timer));
}).listen(3000);

Read the req.on("close") line twice. A reader who closes the tab leaves the loop running unless you stop it, and a forgotten timer is a quiet way to burn server time.

Receiving events in the browser

The browser ships a purpose-built class for this format, and the next sentences come from MDN's guide to it.

To begin receiving events, create a new EventSource object with the URL of a server script that generates the events.

This is a one-way connection, so you simply can't send events from a client to a server.

That restriction shapes your design. When the user submits a prompt, you normally send it with a separate request, then open the stream to read the answer. Wire the pieces together like this:

const source = new EventSource("/stream");

source.onmessage = (event) => {
  const { token } = JSON.parse(event.data);
  document.querySelector("#out")!.append(token + " ");
};

source.addEventListener("done", () => source.close());

You can take action on an error programmatically by implementing the onerror callback on the EventSource object.

Errors and closures are different events, and the next two rules decide what happens after each.

By default, if the connection between the client and the server closes, the connection is restarted.

That default is convenient for dropped networks, but it needs an explicit exit.

The connection is terminated with the .close() method.

Which of these behaviours would surprise a user of your page? Decide deliberately whether your page should call close() on the final event, as the example does, instead of leaving the restart rule to decide for you.

Reconnection and its limits

Auto-reconnect is the feature that makes SSE pleasant for long answers. The next sentence comes from MDN's guide.

If the connection to the server is lost, the browser waits for the specified time and then attempts to reconnect.

The HTML Standard states the default plainly.

Clients will reconnect if the connection is closed.

The server has levers as well. It can tell a client to give up, and it can relocate the stream. The next two sentences come from the HTML Standard.

A client can be told to stop reconnecting by an HTTP 204 No Content response code.

Event stream requests can be redirected with HTTP 301 and 307 redirects, just like normal HTTP requests.

Browsers also cap how many connections of this kind you can open. The next two sentences come from MDN's guide.

The limit is per browser and domain, which means you can open six SSE connections across all of the tabs.

The count is shared by every tab of the same browser that talks to the same domain, so it is not tracked page by page. The rule changes when HTTP/2 is used.

When using HTTP/2, the maximum number of simultaneous HTTP streams gets negotiated between the server and the client.

If your product lets people open many tabs, count the streams each tab holds, and close the ones nobody is watching.

How model vendors use SSE

You rarely write the byte format by hand when you call a model, but you will read it in logs and debuggers, so it pays to recognise it. The two vendor guides cited in this lesson take slightly different approaches, so the sentences below say which guide each one comes from.

Anthropic

This part comes from the Anthropic streaming guide.

When creating a Message, you can set the stream: true option to incrementally stream the response using server-sent events.

Each server-sent event includes a named event type and its associated JSON data.

In practice that means a line naming the event, then a line carrying a JSON object. Your parser reads the name first and decides what to do with the data.

Each event uses an SSE event name, such as event: message_stop, and also includes the matching event type in its data.

Two further event kinds deserve a place in your parser, because both can arrive mid-stream.

Event streams might also include any number of ping events.

The API might occasionally send errors in the event stream.

Treat an unknown event type as something to skip, never as a crash, and treat an error event as a reason to show the reader what has arrived so far.

OpenAI

This part comes from the OpenAI streaming guide.

The guide focuses on HTTP streaming with stream=true over server-sent events.

The Responses API uses semantic events for streaming.

That wording matters because it points to a different parsing style from the one above: you register interest in particular event kinds instead of scanning raw text.

Each event is typed with a predefined schema, so that you can listen for the events you care about.

The two designs differ in detail, yet both reward the same habit: switch on the event type, and never assume that every event is a text fragment.

When you need traffic in both directions on one connection, SSE is the wrong tool.

For a persistent WebSocket transport that takes incremental inputs via previous_response_id, see the WebSocket mode of the Responses API.

Browser support

On browser support for server-sent events, the HTML Standard page says this.

There is support in all current engines.

That is broad support, but you should still test in the browsers your own readers use. Where you need to send a request body along with the stream, read it with fetch instead, which the next lesson builds out.

Practice

Run the server, open the page, and then stop the server in the middle of a stream. Watch what the browser does, then restart the server and confirm the stream resumes. Next, change the id values and check what your server sees in a reconnect request.