OpenAI Realtime

OpenAI Realtime uses a persistent connection and bidirectional events to process realtime text, audio, and function calls. The client sends session, audio-buffer, conversation-item, and response-control events, while the server continuously returns session status, transcripts, text, audio, function arguments, and error events.

This page primarily covers WebSocket connections. When connecting, replace MODEL_ID with a model ID that is visible to the current account and supports Realtime.

Typical use cases include realtime voice conversations, live meeting transcription and voice assistants, realtime translation, speech transcription, interactive coding, and server-side realtime integrations. A single connection can carry both text and audio while events coordinate VAD, function calls, and state updates.

Authentication and connection

WebSocket authentication

Server applications establish a WSS connection with a Bearer API key and include the Realtime Beta header in the handshake request.

Long-lived API keys should only be used on trusted servers. Do not include them in browsers, mobile application bundles, or public repositories. Always use HTTPS/WSS for connections.

Ephemeral tokens

Before a client connects, a trusted server can create an ephemeral token that is valid for one minute and limited to a single connection.

POST https://api.agtcloud.cn/v1/realtime/sessions
Content-Type: application/json
Authorization: Bearer your-api-key

{
  "model": "MODEL_ID",
  "voice": "verse"
}

Connection flow

In the primary WebSocket flow, the server connects directly with an API key. After the handshake succeeds, it sends client events and receives server events over the same connection.

In the WebRTC flow, the client first requests an ephemeral token from its application server, which calls the session endpoint to create the token. The client then creates an SDP offer, submits it to the Realtime endpoint, receives the SDP answer, and exchanges JSON events through the oai-events data channel.

WebSocket connection examples

The examples below show Node.js, Python, event sending, and event receiving. After establishing a connection, the client sends JSON events and the server continuously returns JSON events.

import WebSocket from 'ws';

const ws = new WebSocket('wss://api.agtcloud.cn/v1/realtime?model=MODEL_ID', {
  headers: {
    Authorization: 'Bearer your-api-key',
    'OpenAI-Beta': 'realtime=v1',
  },
});

ws.on('open', () => {
  console.log('Connected to server.');
});

ws.on('message', (message) => {
  console.log(JSON.parse(message.toString()));
});

WebRTC integration

WebSocket is currently recommended. Before using WebRTC, confirm that the target model supports it. The following examples illustrate the basic connection structure for ephemeral tokens, SDP, audio tracks, and data channels.

async function init() {
  const tokenResponse = await fetch('/session');
  const data = await tokenResponse.json();
  const ephemeralKey = data.client_secret.value;

  const pc = new RTCPeerConnection();
  const audioElement = document.createElement('audio');
  audioElement.autoplay = true;
  pc.ontrack = (event) => {
    audioElement.srcObject = event.streams[0];
  };

  const mediaStream = await navigator.mediaDevices.getUserMedia({ audio: true });
  pc.addTrack(mediaStream.getTracks()[0]);

  const dataChannel = pc.createDataChannel('oai-events');
  dataChannel.addEventListener('message', (event) => {
    console.log(JSON.parse(event.data));
  });

  const offer = await pc.createOffer();
  await pc.setLocalDescription(offer);

  const response = await fetch(
    'https://api.agtcloud.cn/v1/realtime?model=MODEL_ID',
    {
      method: 'POST',
      body: offer.sdp,
      headers: {
        Authorization: `Bearer ${ephemeralKey}`,
        'Content-Type': 'application/sdp',
      },
    },
  );

  await pc.setRemoteDescription({
    type: 'answer',
    sdp: await response.text(),
  });
}

init();

import express from 'express';

const app = express();

app.get('/session', async (_request, response) => {
  const result = await fetch('https://api.agtcloud.cn/v1/realtime/sessions', {
    method: 'POST',
    headers: {
      Authorization: 'Bearer your-api-key',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: 'MODEL_ID',
      voice: 'verse',
    }),
  });

  response.send(await result.json());
});

app.listen(3000);

const dataChannel = pc.createDataChannel('oai-events');

dataChannel.addEventListener('message', (event) => {
  const realtimeEvent = JSON.parse(event.data);
  console.log(realtimeEvent);
});

dataChannel.send(JSON.stringify({
  type: 'response.create',
  response: {
    modalities: ['text'],
    instructions: 'Write a haiku about code',
  },
}));

const ws = new WebSocket(
  'wss://api.agtcloud.cn/v1/realtime?model=MODEL_ID',
  [
    'realtime',
    'openai-insecure-api-key.your-api-key',
    'openai-organization.ORGANIZATION_ID',
    'openai-project.PROJECT_ID',
    'openai-beta.realtime-v1',
  ],
);

ws.addEventListener('open', () => {
  console.log('Connected to server.');
});

ws.addEventListener('message', (event) => {
  console.log(JSON.parse(event.data));
});

Sessions, audio, and conversations

Realtime supports text-only, voice-only, and mixed text-and-audio conversations. Session configuration can include input and output audio formats, voice, temperature, maximum output length, tools, and VAD settings.

• Input and output audio formats: pcm16, g711_ulaw, and g711_alaw.
• Example voice options: alloy, echo, and shimmer.
• VAD configuration includes the activation threshold, prefix padding, and silence duration.
• Sessions, conversation items, content parts, and response events jointly advance the conversation.

See the event browser below for individual event fields. Shared parameters and value ranges are collected under Common types.

Event reference

Select an event to view its field definitions and JSON example. You can switch event direction, filter by category, or search by event name.

{
  "event_id": "event_123",
  "type": "session.update",
  "modalities": [
    "text",
    "audio"
  ],
  "instructions": "Respond clearly and concisely.",
  "voice": "alloy, echo, shimmer",
  "input_audio_format": "pcm16, g711_ulaw, g711_alaw",
  "output_audio_format": "pcm16, g711_ulaw, g711_alaw",
  "input_audio_transcription": {
    "model": "whisper-1"
  },
  "turn_detection": {
    "type": "server_vad",
    "threshold": 0.8,
    "prefix_padding_ms": 500,
    "silence_duration_ms": 1000
  },
  "tools": [],
  "tool_choice": "auto",
  "temperature": 0.8,
  "max_output_tokens": "inf"
}

Common types

The sections below summarize shared parameter tables for rate limits, function calls, audio, VAD, tools, models, events, sessions, conversation items, content, responses, transcription, audio streams, and WebRTC.

Error handling and recovery

Realtime errors generally fall into four categories: connection, authentication, audio, or session errors.

Recovery strategies can include automatic reconnection, session recovery, limited retries, and feature fallback. When an error occurs, first inspect the error event and save its event_id, error.type, error.code, error.message, and related-event information before deciding whether to resend anything.