Realtime transcription

The Muse Voice Transcribe realtime API receives raw PCM audio over a persistent WebSocket connection and continuously returns progress, speech-turn, and transcription events. It produces text only and does not generate a spoken response.

Authentication and connection

After the WebSocket connection is established, send the first JSON text frame within 10 seconds. Place the Bearer API key in authorization.accessToken, not in the WebSocket handshake headers.

The first frame has no type and must not exceed 64 KiB. Wait for a successful response containing sessionId before sending audio. A successful WebSocket upgrade does not mean that authentication in the first frame has succeeded.

Session flow

1. Establish the WebSocket connection.
2. Send the first frame with authentication, model, and audio configuration.
3. Wait for the server to return sessionId.
4. Send raw PCM binary frames at realtime speed while continuously receiving events.
5. After the audio ends, send {"type":"endStream"}.
6. Continue receiving final transcription and close messages.

speechComplete means that one speech turn has completed, not that the entire session has ended. After sending endStream, continue reading server events until the connection closes.

First-frame parameters

Audio input

Audio must be headerless, mono PCM16 little-endian data and must match the 16 kHz or 24 kHz sample rate declared in the first frame. Each audio frame must be no larger than 1 MiB and contain an even number of bytes.

Use audio chunks of approximately 100 ms and send them at realtime speed: 4,800 bytes at 24 kHz or 3,200 bytes at 16 kHz. To preserve a continuous timeline during silence, send PCM silence in the same format.

# Generate raw 24 kHz, mono PCM16 audio
ffmpeg -i input.mp3 -ac 1 -ar 24000 -c:a pcm_s16le -f s16le audio.pcm

Python example

Install a WebSocket client and prepare audio.pcm before running the example. Sending and receiving occur in separate threads so that audio upload does not block server event processing.

python -m pip install websocket-client

import json
import sys
import threading
import websocket

url = "wss://api.agtcloud.cn/v1/asr/realtime"
stop = threading.Event()
sender = None
ws = websocket.create_connection(url, timeout=40, enable_multithread=True)

def send_audio():
    try:
        with open(sys.argv[1], "rb") as audio:
            while not stop.is_set():
                chunk = audio.read(4800)
                if not chunk:
                    break
                if len(chunk) % 2:
                    raise ValueError("PCM byte length must be even")
                ws.send_binary(chunk)
                if stop.wait(len(chunk) / 48000):
                    return
        if not stop.is_set():
            silence = bytes(4800)
            for _ in range(10):
                ws.send_binary(silence)
                if stop.wait(0.1):
                    return
            ws.send(json.dumps({"type": "endStream"}))
    except Exception as exc:
        print("Sending interrupted:", exc)
        stop.set()
        ws.close()

try:
    ws.send(json.dumps({
        "authorization": {"accessToken": "Bearer your-api-key"},
        "model": "muse-voice-transcribe-1.0",
        "audioEncoding": "PCM_24KHZ",
        "mode": "ENDPOINTING",
        "partialMode": "CUMULATIVE",
        "languageBias": ["Mandarin Chinese"],
        "emitAudioProgress": True
    }))
    ack = json.loads(ws.recv())
    if ack.get("type") == "error" or not ack.get("sessionId"):
        raise RuntimeError("Connection initialization failed: " + json.dumps(ack, ensure_ascii=False))
    print("sessionId:", ack["sessionId"])
    sender = threading.Thread(target=send_audio, daemon=True)
    sender.start()
    while not stop.is_set():
        message = ws.recv()
        if not message:
            break
        event = json.loads(message)
        print(json.dumps(event, ensure_ascii=False))
        if event.get("type") == "error":
            break
finally:
    stop.set()
    ws.close()
    if sender is not None:
        sender.join(timeout=2)

Event reference

audioProcessedMs is cumulative processing progress; do not add values from multiple events. With CUMULATIVE, each new partial transcript replaces the previous accumulated result. With DELTA, append each new text fragment.

{
  "type": "audioProgress",
  "audioProcessedMs": 1600
}

speechComplete.transcript contains the complete transcript for the current speech turn. During session finalization, a transcript event with final: true may not contain text. Do not overwrite a saved cumulative transcript or speechComplete.transcript with an empty string.

Session limits and error handling

A session can last approximately 60 minutes. A 15-second period without a client message may trigger input finalization, and network or server conditions may end the connection earlier. Set an overall duration, read timeout, and limited retry policy on the client.

Billing

Realtime transcription is billed by processed audio duration. See the model catalog for pricing on the current site. Millisecond values in progress events represent the processing timeline and must not be added as separate billing records.

References

• Muse Voice Transcribe model overview
• Meta Speech-to-Text guide
• Meta Voice API Reference