From a demo to a service

A streaming demo needs a browser and an API key. A streaming service needs a place to enforce limits, a way to see what is happening, and a plan for the moments when something between the user and the model goes quiet. This lesson is about those extra parts. It builds on the parsing and error handling from the previous lessons, and it assumes you now treat a stream as a long-lived connection rather than a single call.

Choosing the transport

Start with the choice that the OpenAI guide puts in front of you.

For a persistent WebSocket transport that takes incremental inputs via previous_response_id, see the WebSocket mode of the Responses API.

For most chat interfaces the one-way flow of server-sent events is enough, because the prompt can travel in a normal request. A persistent two-way connection earns its extra complexity only when the client needs to send more input while a response is still running. Ask which of those you really need before you commit.

The OpenAI guide describes how its HTTP form delivers a reply. The next sentence comes from it.

The response is sent back incrementally in separate chunks with an event stream.

The OpenAI guide names the endpoint alongside the same stream=True switch. The next two sentences come from that guide.

To start streaming, set stream=True in your request to the Responses endpoint.

To stream completions, set stream=True when you call the Chat Completions or legacy Completions endpoints.

You will meet both in existing code. The OpenAI guide describes the newer endpoint like this.

The Responses API uses semantic events for streaming, and it is type-safe.

Compare that with the older format, which the OpenAI guide also describes.

Streaming Chat Completions is fairly straightforward.

Easy does not mean complete, though. If you plan to listen for many kinds of event, the typed design gives you more to build on.

Why put your own server in the middle

You could call a vendor straight from the browser, but then you have no place of your own to enforce anything. A server of your own gives you one place to keep credentials out of reach, to apply rate limits per user, to log what you need, and to translate between vendor event formats and the one your front end understands. The price is that the server becomes a stream consumer and a stream producer at the same time, so it inherits every problem from the previous lessons twice.

Think of the relay as three small loops: one reads from the upstream source, one writes to the browser, and one keeps time for heartbeats and metrics. The sketch below uses a plain async iterable as the upstream, so you can replace it with whichever vendor SDK you choose.

import { createServer, type ServerResponse } from "node:http";

type Upstream = (signal: AbortSignal) => AsyncIterable<string>;

export function startRelay(upstream: Upstream, port = 3000, heartbeatMs = 15_000) {
  return createServer(async (req, res: ServerResponse) => {
    const abort = new AbortController();
    const startedAt = Date.now();
    let firstAt = 0;
    let fragments = 0;
    let outcome = "finished";

    res.writeHead(200, { "Content-Type": "text/event-stream", "Cache-Control": "no-cache" });
    const beat = setInterval(() => res.write(": keep-alive\n\n"), heartbeatMs);
    req.on("close", () => {
      outcome = "cancelled";
      abort.abort();
    });

    try {
      for await (const fragment of upstream(abort.signal)) {
        if (!firstAt) firstAt = Date.now();
        fragments += 1;
        res.write(`data: ${JSON.stringify({ text: fragment })}\n\n`);
      }
      res.write("event: done\ndata: {}\n\n");
    } catch (err) {
      if (!abort.signal.aborted) {
        outcome = "failed";
        res.write(`event: failure\ndata: ${JSON.stringify({ message: String(err) })}\n\n`);
      }
    } finally {
      clearInterval(beat);
      res.end();
      console.log(JSON.stringify({ outcome, fragments, ttfbMs: firstAt ? firstAt - startedAt : null, totalMs: Date.now() - startedAt }));
    }
  }).listen(port);
}

Walk through the finally block. It runs in every outcome, so each stream produces exactly one log line with an outcome, a count, the delay before the first output, and the total time. That single line is the raw material for every dashboard you will build later.

The event structure you build on

When you relay or log a vendor stream you need to understand the shape of the message being built. The next sentences come from the Anthropic guide.

Each stream uses the following event flow, and message_start contains a Message object with empty content.

A content block has a content_block_start and one or more content_block_delta events.

A content block also has a content_block_stop event.

Every block, whatever its content, also carries a position.

Each content block has an index that corresponds to its position in the final Message content array.

The index is what lets a relay assemble the finished message from parts without guessing. If you store blocks by their index as events arrive, you can rebuild the final array at the end, even when you have only seen the stream and never the finished object.

Three other facts from the same Anthropic guide shape your metrics and your error handling.

The token counts that are shown in the usage field of a message_delta event are cumulative.

So read the last value, and never add the values together. Then there are events that carry no content at all.

Event streams might also include any number of ping events.

The API might occasionally send errors in the event stream.

Pass pings through quietly, and handle errors as described in lesson 3. A relay should also forward an error as an event of its own, so that the browser can tell a failed answer from a finished one.

Long outputs and timeouts

A request that asks for a very long answer can run for longer than a plain request is willing to wait. The next two sentences come from the Anthropic guide and concern its SDK.

If you don’t need to process the text as it arrives, the SDK also provides a way to use streaming internally while it returns the complete Message object.

This is especially useful for those requests with large max_tokens values, where the SDK requires streaming in order to avoid HTTP timeouts.

That pair is a good fit for a background job that needs the finished text and not the live view. The SDK keeps the transport streaming, and your code receives one object. The Python example below shows the shape.

import anthropic

client = anthropic.Anthropic()  # reads the key from the environment

with client.messages.stream(
    max_tokens=4096,
    messages=[{"role": "user", "content": "Summarise the lesson in three bullet points."}],
    model="claude-opus-5-5",
) as stream:
    message = stream.get_final_message()

print(message.content[0])

The .stream() call keeps the HTTP connection alive by using server-sent events.

The example above shows that call in use. The sentence above and the next one also come from the Anthropic guide, and the next one is the part that matters for a background job.

The SDK’s message-accumulation helper, stream.get_final_message(), collects all of the events into the complete Message object.

The Anthropic guide also lists language-specific details. The next three sentences come from it.

The Python SDK allows both sync and async streams.

The PHP SDK provides streaming through createStream().

See the documentation in each SDK for details.

Whichever language you use, read that documentation before you copy a pattern, because a helper that exists in one SDK may have a different name in another.

Heartbeats and buffering

Your relay can be perfectly correct and still deliver nothing, because something between it and the browser is holding bytes back. Two rules from the earlier lessons matter here: the first comes from MDN's guide, and the second from the HTML Standard.

A server can send a comment line periodically to keep the connection alive.

UAs should ensure that appropriate buffering is used.

Treat those as a pair. Send a comment line on a timer, as the code above does, and test the full path with a slow stream. Include every layer you operate, such as a reverse proxy or a content delivery network, and confirm that each one passes the response through as it arrives. Find the maximum request duration of your hosting platform, because a platform that cuts a long request will cut a long stream too.

Watching the stream

The relay above logs one line per stream. Turn those lines into four questions you can answer at any time. How many streams are open right now? How long does the typical stream wait before its first output, which is the latency your readers notice? How long does a stream last? What fraction ends in failure or cancellation? Chart each over time and alert on sudden changes rather than on fixed numbers, because normal values depend on your model and your prompts.

Use the cumulative usage value when you want a throughput figure: divide the final count by the stream duration, and compare it with the same figure for yesterday. A sudden drop can point to a slowdown on the provider side, although other causes are possible. Label error counts by the phase in which they occurred, such as before the first output or midway, because the fixes differ.

Content policy at the edge

Finally, a relay is the natural place to apply content rules, and the OpenAI guide warns that streaming complicates them. The next two sentences come from it.

Partial completions may be more difficult to evaluate.

The OpenAI guide adds a second concern.

This may have implications for approved usage.

Decide whether the relay buffers a sentence at a time before forwarding it, or forwards immediately and retracts later. Either choice has a cost in delay or in exposure, so make it deliberately and write it down.

Practice

Build the relay from the exercise with a fake upstream that sleeps between fragments, then run it behind the proxy you plan to use in production and watch whether the fragments arrive one by one.