Streaming

Streaming lets a client receive incremental content before the model finishes generating the complete response. It works well for chat interfaces, long-form generation, and experiences that need to display the first output quickly, but the client must correctly handle event boundaries, content assembly, completion states, and connection interruptions.

Before you begin

1. Confirm in the Model marketplace that the target model supports the selected protocol and Streaming.
2. Use the endpoint and event structure for that protocol. Do not make one parser guess across every protocol.
3. Confirm that the client, reverse proxy, and gateway do not buffer the entire response before returning it at once.

Complete fields are maintained in the Chat Completions, Responses, Anthropic Messages, and Gemini Stream Generate Content references.

Send a minimal streaming request

The following example enables streaming output through Chat Completions:

curl --no-buffer 'https://api.agtcloud.cn/v1/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer your-api-key' \
  --data '{
    "model": "MODEL_ID",
    "messages": [
      {"role": "user", "content": "Explain exponential backoff in three points."}
    ],
    "stream": true
  }'

--no-buffer makes cURL display received data promptly. It does not change the server-side event structure.

Read by event, not by network packet

The number of bytes delivered to the client at one time is not the same as one complete event. A JSON object can be split across several chunks, and multiple events can arrive in one chunk.

For SSE, identify each complete event by its blank-line boundary before reading its data: content:

const response = await fetch('https://api.agtcloud.cn/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    Authorization: 'Bearer your-api-key',
  },
  body: JSON.stringify({
    model: 'MODEL_ID',
    messages: [{ role: 'user', content: 'Explain exponential backoff.' }],
    stream: true,
  }),
});

if (!response.ok || !response.body) {
  throw new Error(`Request failed: ${response.status}`);
}

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { value, done } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const events = buffer.split('\n\n');
  buffer = events.pop() ?? '';

  for (const event of events) {
    const data = event
      .split('\n')
      .filter((line) => line.startsWith('data:'))
      .map((line) => line.slice(5).trimStart())
      .join('\n');

    if (!data || data === '[DONE]') continue;
    const chunk = JSON.parse(data);
    const delta = chunk.choices?.[0]?.delta?.content;
    if (delta) process.stdout.write(delta);
  }
}

Production code should also handle request cancellation, read timeouts, heartbeats, non-JSON events, and any unconsumed buffer remaining when the stream ends.

Streaming structures across four protocols

The presence of a protocol entry point does not mean every model returns every content type described above. Implement only the branches your application uses and that the target model has actually returned.

Assemble text and tool arguments

• Assemble text deltas separately by candidate, output item, or Content Block index.
• Tool names, call IDs, and arguments can arrive in several deltas; maintain an independent buffer for each call.
• Parse tool arguments as JSON only after they are complete. Never execute them chunk by chunk.
• Keep reasoning summaries, Thinking Blocks, and final answers in separate buffers instead of mixing them into one string.
• If the protocol returns Usage at the end, continue reading until the completion event arrives.

Determine success, failure, and interruption

After a streaming request has already displayed partial content, do not replay it indefinitely. A retry can produce a different answer or repeat tools and other operations with side effects.

Pre-production checklist

• The page displays text incrementally instead of waiting for the connection to close.
• Chinese and other multibyte characters remain valid when split across chunks.
• Tool arguments are assembled by call ID and validated before execution.
• User cancellation, network disconnection, timeout, and server errors each have an explicit state.
• Logs do not contain API Keys, complete sensitive prompts, or unredacted tool arguments.

Continue with Tool Calling to assemble streaming tool arguments, or see Errors and troubleshooting when a request fails.