Realtime API

WebSocket sessions for streaming chat with an optional voice round-trip — speech-to-text on committed audio, text-to-speech on responses. The realtime surface lives on client.realtime.

It works with the browser's native WebSocket, Node.js ≥ 21 (global WebSocket), or any compatible implementation (e.g. the ws package) passed via connect({ webSocket }).

Overview

import { ConsoleClient } from '@cognipeer/console-sdk';

const client = new ConsoleClient({ apiKey: process.env.COGNIPEER_API_KEY! });

Connect and take a turn

The high-level respond() helper sends a user message and resolves with the full assistant response.

const conn = await client.realtime.connect({ model: 'gpt-4o-mini' });

const { text, audio } = await conn.respond('Give me a haiku about the sea.');
console.log(text);

conn.close();

Generator is locked at session start

The response generator — a model or an agent — is chosen when the session opens. You may still set it via updateSession before the first response, but it locks once the conversation starts. Pass { agent: 'sales-assistant' } instead of { model } to have an agent generate responses.

Event-driven usage

For full control, listen to server events directly. Use '*' to subscribe to every event.

const conn = await client.realtime.connect({ model: 'gpt-4o-mini' });

conn.on('response.output_text.delta', (e) => process.stdout.write(e.delta));
conn.on('response.done', () => conn.close());

conn.createItem('Stream me a short story.');
conn.createResponse();

Connection methods

Method	Description
conn.on(type, cb) / conn.off(type, cb)	Subscribe / unsubscribe ('*' = all events)
conn.send(event)	Send a raw client event
conn.updateSession(session)	Update model, instructions, voice, STT/TTS models
conn.createItem(content, role?)	Append a conversation message
conn.appendAudio(audio) / clearAudio() / commitAudio()	Stream input audio; commit to transcribe as a user turn
conn.createResponse(overrides?) / cancelResponse()	Generate / cancel the next assistant response
conn.respond(content, options?)	High-level: send a message, await the full response
conn.close()	Close the socket

Voice round-trip

Configure STT, TTS, and a voice (on the session or a realtime model preset) to get audio back. respond() returns base64 audio alongside text; in event mode listen for response.audio.delta.

const conn = await client.realtime.connect({ model: 'voice-assistant' });
conn.appendAudio(micChunkBase64);
conn.commitAudio();      // transcribe the buffered audio as a user turn
conn.createResponse();   // assistant replies with text + synthesized audio

Node.js without a global WebSocket

import WebSocket from 'ws';

const conn = await client.realtime.connect({
  model: 'gpt-4o-mini',
  webSocket: WebSocket,
});

Messages from outside the session

A voice agent often starts work it cannot wait for — a report, a deployment, a payment check. The session does not need to stay in your hands for the answer: when the work finishes, send the result into the running conversation from anywhere (a job, a webhook, an operator) with realtime.sessions.sendMessage. The assistant tells the caller.

await client.realtime.sessions.sendMessage(sessionId, {
  content: 'The quarterly report you asked for is ready: 14 pages, revenue up 8%.',
  label: 'report-bot',          // shown to the assistant as "Notification from report-bot"
  idempotencyKey: 'job-42',     // a retry adds the message once
});

Field	Default	Description
content	—	The text, at most 4000 characters.
respond	true	false adds the message silently; the assistant sees it on its next turn.
mode	'when_idle'	'when_idle' queues the answer behind the one in flight (and drops it, keeping the text, if the caller speaks first); 'now' interrupts the current answer.
label	external system	Who is speaking.
idempotencyKey	—	Same key within 10 minutes: nothing is added twice (replayed: true).

What to expect:

• The message enters as a user turn marked as an external notification. system and assistant messages cannot be injected, and the answer goes through the session's guardrails like any other user turn.
• The live monitor and the playground show it with an external notification badge. A websocket client of the same session also receives conversation.item.created with item.source === 'external'.
• The call needs the realtime service permission and the session must belong to the key's project. An unknown session, an ended one and another project's all answer 404; a session that has more than 30 accepted messages in a minute answers 429 with Retry-After.
• A session lives on one server process. With several replicas the message is relayed to the one that holds the session (this needs the same Redis the platform uses for queues or cache); without Redis it is delivered only when the request reaches that process. A relayed request that nobody answers within about 1.5 s is a 404.

The same call exists in the dashboard: the playground and the live-session page have a Send a message into this session panel.

Realtime model presets

A realtime model bundles chat + STT + TTS + voice + instructions under one key; connect with ?model=<key>. Manage presets via client.realtime.models.

const models = await client.realtime.models.list();
const preset = await client.realtime.models.create({ /* ... */ });
await client.realtime.models.update(preset.id, { /* ... */ });
await client.realtime.models.delete(preset.id);

Low-level URLs

// Raw WebSocket URL (for custom clients):
const wsUrl = client.realtime.url({ model: 'gpt-4o-mini' });

// Twilio Media Streams URL — paste into TwiML:
//   <Connect><Stream url="..."/></Connect>
// The model must have STT, TTS, and a voice configured.
const twilioUrl = client.realtime.twilioStreamUrl('voice-assistant');

Methods

Method	Transport	Description
realtime.connect(options?)	WS	Open a session, resolve a RealtimeConnection
realtime.url(options?)	—	Build the raw WebSocket URL
realtime.twilioStreamUrl(model)	—	Twilio Media Streams URL
realtime.sessions.sendMessage(sessionId, body)	POST /api/client/v1/realtime/sessions/:sessionId/messages	Send a message into a running session
realtime.models.list()	GET /api/client/v1/realtime/models	List presets
realtime.models.create(data)	POST /api/client/v1/realtime/models	Create a preset
realtime.models.retrieve(id)	GET /api/client/v1/realtime/models/:id	Fetch a preset
realtime.models.update(id, data)	PATCH /api/client/v1/realtime/models/:id	Update a preset
realtime.models.delete(id)	DELETE /api/client/v1/realtime/models/:id	Delete a preset