From bytes to words on screen

The previous lesson covered the wire format. This lesson covers what happens after the bytes reach the browser. A model produces its reply as a sequence of tokens, and streaming carries them to the page in network chunks. A network chunk has no respect for event boundaries: one chunk may hold half an event, or three events and the start of a fourth. Your code therefore needs a small buffer that collects text until a complete event is available, and a renderer that turns events into pixels without making the page stutter.

Two vendor guides drive the examples in this lesson, and each block below says which guide it comes from. The next two sentences come from the OpenAI guide.

The response is sent back incrementally in separate chunks with an event stream.

This returns an object that streams back its response as data-only server-sent events.

What do you think the phrase "data-only" implies for the fields your parser has to handle? Keep that question in mind while you read the code below, which is deliberately small.

Splitting the stream into events

Start with the reading loop. In this lesson you read the response with fetch, which lets you control the request yourself, rather than with the EventSource class from lesson 1. Read the body as a stream, decode each chunk to text, append it to a buffer, and then cut the buffer at every blank line. Whatever remains after the last blank line is an unfinished event, so keep it for the next round.

export async function* readEvents(response: Response): AsyncGenerator<string> {
  if (!response.body) throw new Error("response has no body");
  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });

    let cut = buffer.indexOf("\n\n");
    while (cut !== -1) {
      yield buffer.slice(0, cut);
      buffer = buffer.slice(cut + 2);
      cut = buffer.indexOf("\n\n");
    }
  }
  if (buffer.trim()) yield buffer;
}

This reader cuts events at a double line feed only, so lesson 3 shows how to accept the other line terminators as well. Use one decoder for the whole response and pass the streaming option, so a character that is split across two chunks is not damaged. Then write a second helper that pulls the data: lines out of one event and joins them, as lesson 1 described.

Deciding what each event means

Once you have whole events, look at what each one is. The two vendors label their events differently, so read the labels before you write the switch statement. The next three sentences come from the OpenAI guide.

You can identify individual events by the type property of the event.

That gives you one reliable field to branch on.

Some key lifecycle events are emitted only once, whereas others are emitted multiple times as the response is generated.

In other words, some events are one-off markers while others repeat as the reply grows.

Common events to listen for when you stream text are response.created, response.output_text.delta, response.completed, and error.

Only one of those four names carries new text, and the others mark the beginning, the end, or a failure. A renderer that appends something on every event will print junk. Pick out the text-bearing event, append its payload, and ignore the rest, including any type you have never seen.

The OpenAI guide also covers the older chat completions format, which works differently.

When you stream a chat completion, the response carries a delta field rather than a message field.

That detail is easy to miss. A parser that expects text in every chunk will fail on a chunk that carries something else, so guard every access to the text field, and treat a missing value as an empty string.

Rendering without jank

A fast model can deliver many small fragments in a second, and each fragment is tempting to write straight into the page. Resist that. Every write can trigger layout work, and a burst of writes inside one frame buys you nothing, because the user only ever sees the state at the end of the frame. Collect fragments in an array and flush the array once per animation frame.

export class FrameBatcher {
  private pending: string[] = [];
  private scheduled = false;

  constructor(private readonly target: HTMLElement) {}

  push(fragment: string): void {
    this.pending.push(fragment);
    if (this.scheduled) return;
    this.scheduled = true;
    requestAnimationFrame(() => {
      this.target.append(this.pending.join(""));
      this.pending = [];
      this.scheduled = false;
    });
  }
}

This version appends plain text. If you render Markdown, hold back an unfinished construct such as half a link until the closing characters arrive, otherwise the reader sees brackets flicker. You may also add a blinking caret at the end of the growing text so the reader knows more is coming; remove it on the final event.

Partial JSON from tool calls

When the model calls a tool, the arguments can be streamed as well. The next three sentences come from the Anthropic guide, which describes how they arrive.

The deltas for tool_use content blocks correspond to updates of the input field of the block.

The shape of those deltas explains most of the trouble people have with streamed tool calls.

To support maximum granularity, the deltas are partial JSON strings, while the final tool_use.input is always an object.

The practical rule follows from the second sentence: never parse a fragment.

You can accumulate the string deltas, then parse the JSON once you receive a content_block_stop event.

The next two sentences also come from the Anthropic guide, and they describe two details that affect how the interface should feel.

Current models only support emitting a single complete key and value property from input at a time.

That limit has a visible consequence for the person waiting.

As such, when you use tools, there might be delays between the streaming events while the model is working.

So a pause in the middle of a tool call is normal. Show a status line such as "calling a tool" instead of a frozen caret, otherwise the reader may assume that the connection has hung.

type ToolDelta = { type: string; partial_json?: string };

export function makeToolAccumulator(onDone: (input: unknown) => void) {
  let raw = "";
  return (event: { type: string; delta?: ToolDelta }) => {
    if (event.type === "content_block_delta" && event.delta?.partial_json !== undefined) {
      raw += event.delta.partial_json;
    }
    if (event.type === "content_block_stop" && raw) {
      onDone(JSON.parse(raw));
      raw = "";
    }
  };
}

Partial output and safety

Showing text early has a side effect for any product that checks what the model says. The next two sentences come from the OpenAI guide.

Partial completions may be more difficult to evaluate.

There is a second consequence as well.

This may have implications for approved usage.

Decide up front whether your product can live with the reader seeing a sentence before a check has run. If it cannot, hold the text back until the check passes, or stream only to the stage where a person reviews it.

When you do not need the stream

Sometimes you want the benefits of a streaming connection but not the incremental display. The next two sentences come from the Anthropic guide and concern its SDK.

If you don’t need to process the text as it arrives, the SDK also provides a way to use streaming internally while it returns the complete Message object.

This is especially useful for those requests with large max_tokens values, where the SDK requires streaming in order to avoid HTTP timeouts.

That is a useful middle path for background jobs: the transport stays alive, yet your code handles one finished object.

Practice

Record a real response body to a file, then replay it through readEvents in chunks of random sizes. If the rendered text changes with the chunk size, the buffer logic has a bug. Fix it before you connect the parser to a live model.