Two kinds of trouble

Streaming adds two problems that ordinary request and response code never meets, and both show up in the latency the reader feels. The first is pace: the producer and the consumer work at different speeds. The second is fragility: a stream lives for as long as the answer takes, so a failure can strike halfway through. Keep these separate in your head, because they call for different tools. The first calls for buffering policy, and the second for retry and reporting policy.

Backpressure on the client

Backpressure is the pressure a slow consumer puts back on a fast producer. In a chat page the producer is the network and the model, and the consumer is your rendering code. If the consumer cannot keep up, something has to give: the queue grows without limit, the page freezes, or the producer slows down.

You have three honest options, and a good client combines them. Pull data from the response only when your code is ready for the next chunk, so that you never read further ahead than you can handle. Hand the waiting fragments over together, as one string per frame, instead of rendering each separately. And put an upper bound on what you keep, so that a stalled tab cannot swallow memory. The class below does all three jobs.

export class BoundedQueue {
  private parts: string[] = [];
  private size = 0;
  private waiters: Array<() => void> = [];

  constructor(private readonly maxChars = 200_000) {}

  // Resolves once the fragment is stored. While the queue is full the caller waits.
  async push(fragment: string): Promise<void> {
    while (this.size > 0 && this.size + fragment.length > this.maxChars) {
      await new Promise<void>((resolve) => this.waiters.push(resolve));
    }
    this.parts.push(fragment);
    this.size += fragment.length;
  }

  // Returns everything waiting as one string, then wakes the callers that were blocked.
  drain(): string {
    const text = this.parts.join("");
    this.parts = [];
    this.size = 0;
    for (const wake of this.waiters.splice(0)) wake();
    return text;
  }
}

Use it like this: await queue.push(text) inside the loop that reads the response, and queue.drain() once per animation frame, as in the previous lesson. Each push is a constant-time append. The queue never holds more than maxChars characters, apart from a single fragment that is larger than the limit on its own, which is admitted when the queue is empty so that the loop cannot deadlock. When the queue is full, push waits, so the loop stops calling read on the response until the next frame has drained it. That wait is the backpressure: the slow consumer now controls the pace of the producer.

Reading lines correctly

A parser written for the happy path breaks on real servers, so check the rules that govern line splitting. The first of the next two sentences comes from the HTML Standard, and the second from MDN's guide.

Lines must be separated by either a CRLF character pair, a single U+000A LINE FEED character, or a single CR character.

Messages in an event stream are separated by a pair of newline characters.

Combine those two rules: split on any of the three terminators, and treat an empty line as the end of a message. A parser that looks only for two line-feed characters in a row will wait forever on a server that uses carriage returns. The test is easy to run, so feed your parser a recording rewritten with each terminator in turn.

Failures inside and outside the stream

Now consider what can go wrong. Start with the browser class from lesson 1. The next sentences come from MDN's guide to it.

You can take action on an error programmatically by implementing the onerror callback on the EventSource object.

By default, if the connection between the client and the server closes, the connection is restarted.

That default is helpful on a flaky network, and it is also the reason an error handler must not simply log and carry on. The browser may already be opening a new connection while your handler runs.

The connection is terminated with the .close() method.

Next, consider failures that the server reports through the stream itself. These come from the Anthropic guide.

The API might occasionally send errors in the event stream.

This is a different class of failure from a dropped connection: the connection is healthy and the server is telling you something went wrong. Do not treat the two the same way.

Event streams might also include any number of ping events.

So your event switch needs a branch for an error and a quiet branch for pings. Show whatever text has already arrived when an error occurs, label it clearly as incomplete, and offer a retry button that the reader controls.

Retry rules for a fetch-based reader

The built-in class retries by itself. A reader that you build on top of fetch does not, so you must decide the rules. The first sentence below comes from MDN's guide, and the next two come from the HTML Standard.

If the connection to the server is lost, the browser waits for the specified time and then attempts to reconnect.

Clients will reconnect if the connection is closed.

There is one exception to that rule, and it matters for retry code.

A client can be told to stop reconnecting by an HTTP 204 No Content response code.

Your own loop should mirror those rules. Wait before reconnecting, increase the wait after each failure, reset it after a success, give up after a sensible number of attempts, and never retry when the server has told you to stop. The sketch below shows the shape, and it deliberately leaves the actual stream reading to a callback.

type Attempt = { status: number; run: () => Promise<void> };

export async function runWithRetry(
  open: () => Promise<Attempt>,
  signal: AbortSignal,
  maxAttempts = 5,
): Promise<"finished" | "stopped" | "gave-up"> {
  let delayMs = 500;
  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
    if (signal.aborted) return "stopped";
    try {
      const { status, run } = await open();
      if (status === 204) return "stopped";
      await run();
      return "finished";
    } catch {
      if (signal.aborted) return "stopped";
      await new Promise((resolve) => setTimeout(resolve, delayMs));
      delayMs = Math.min(delayMs * 2, 8_000);
    }
  }
  return "gave-up";
}

Notice that the loop checks the abort signal before each attempt and again after each failure. Forgetting the second check is an easy bug to write: the reader presses stop, and the code obediently starts another request.

Keeping a quiet connection open

A model that is thinking, or waiting on a tool, can go silent for a while, so plan for silence. The first sentence below comes from MDN's guide, and the following ones come from the HTML Standard.

A server can send a comment line periodically to keep the connection alive.

Authors can include a comment line, one starting with a colon character, every 15 seconds or so.

The Standard also has a note for the people who build browsers. It speaks about streams like the ones in this lesson.

Connections for such resources are expected to be long-lived.

That expectation has a practical consequence for the software on the receiving end. In the Standard's wording, UAs are user agents, which here means browsers.

UAs should ensure that appropriate buffering is used.

That rule is aimed at browser makers, not at your code, but it tells you what to expect: long-lived connections are normal, and a well-behaved browser buffers sensibly. Your task is to make sure nothing on your own side of the connection adds buffering of its own.

Put those together into a heartbeat policy. Send a comment line at a regular interval whenever no real event is due, and make sure that no layer between your server and the browser holds the response back to fill a buffer. When you operate your own proxy, test with a slow stream, not with a quick one, because quick responses hide the problem.

Telling reconnecting clients apart

When a client reconnects, you may want to attach it to the stream it had before. It is tempting to use the network address as the key. The next sentence comes from the HTML Standard.

Individual clients can have multiple IP addresses, and individual IP addresses can have multiple clients.

Use an identifier that you issue yourself, such as a session token carried in the request, to tie a returning client to its earlier stream.

Cancelling cleanly

Cancellation is part of error handling, because a reader who leaves should not keep a connection and a timer alive. Create an AbortController for each request, pass its signal to fetch, and call abort when the reader presses stop or when the component is torn down. Cancel the retry timer in the same place. Treat the resulting abort error as expected rather than as a failure to report.

const controller = new AbortController();
const stop = document.querySelector<HTMLButtonElement>("#stop");

stop?.addEventListener("click", () => controller.abort());

const response = await fetch("/api/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ prompt: "Hello" }),
  signal: controller.signal,
});

Partial output needs a policy

Finally, errors are not the only reason to stop a stream early. Sometimes the content itself is the problem. This comes from the OpenAI guide.

Partial completions may be more difficult to evaluate.

There is a policy side to this as well.

This may have implications for approved usage.

A scoring service adds one more wrinkle.

The scores arrive after the full generated output is available, if you request moderation scores with a generation request.

Decide in advance what your product does if a check fails midway: remove the text, replace it with a notice, or cut the stream. Write that decision down, and test it with a deliberately bad input.

Practice

Build the fake server from the exercise before you touch a real model. A server that you control can drop the connection on demand, which is far easier than waiting for a real network to fail.