Callbacks and events
A voice client exposes callback hooks that can be defined on the constructor.
Callbacks can be configured as part of your voice client constructor:
const voiceClient = new VoiceClient({
// ...
callbacks: {
onBotReady: bot_ready_handler_func,
},
});
State and connectivity
Provides a TransportState string representing the connectivity state of the local client.
One of: idle | initializing | initialized | authenticating | connecting | connected | ready | disconnected | error
See transports for state explanation.
Local user successfully established a connection to the transport.
Local user disconnected from the transport, either intentionally by calling
voiceClient.disconnect() or due to an error.
Bot connected to the transport and is configuring.
Note: bot connectivity does not infer that its pipeline is yet ready to run. Please use onBotReady instead.
The bot has been instantiated, its pipeline is configured, and it is receiving user media and interactions.
This method is passed a BotReadyData object, which contains a config VoiceClientConfigOption[] array and the RTVI version number.
It is recommended to hydrate your client with the passed config in case the bot alters any configuration based on its implementation.
Since the bot is remote and may be using a different version of RTVI than the client, you can use the passed version string to check for compatibility.
Bot disconnected from the transport.
This may occur due to session expiry, a pipeline error or because the local participant left the session.
Messages and errors
If the client receives an unknown message type from the transport (see messages and events), it can be handled here.
Response error when an action fails or an unknown message type is sent from the client.
Error signalled by the bot. This could be due to a malformed config update or an unknown action dispatch or the ability to complete the requested action.
Configuration
Sent when the bots configuration changes. This is most likely in response to a user updateConfig action, but can occur within a bots pipeline.
It’s recommended to hydate the client config with the passed config.
A list of available configuration options for each service. Sent in response to a user describeConfig() call.
"config": [
{
"service": "llm",
"options": [
{ "name": "model", "type": "string" },
{ "name": "messages", "type": "array" },
...
]
},
{
"service": "tts",
"options": [
{ "name": "voice", "type": "string" },
...
]
},
...
]
Media and devices
Lists available local media microphone devices. Triggered when a new device
becomes available or in response to voiceClient.initDevices().
Lists available local media camera devices. Triggered when a new device
becomes available or in response to voiceClient.initDevices().
User selected a new microphone as their selected/active device.
User selected a new camera as their selected/active device.
Audio and Voice Activity
Media track from a local or remote participant was started and playable. Can be either an audio or video track.
Media track from a local or remote participant was stopped and no longer playable.
Local audio gain level (0 to 1).
Remote audio gain level (0 to 1).
Note: if more than one participant is connected to the transport, the participant property details the associated peer.
The bot started speaking/sending speech audio.
The bot stopped speaking/sending speech audio.
The local user started speaking.
This method is more reliable than using audio gain and is the result of the bot’s VAD (voice activity detection) model. This provides a more accurate result in noisy environments.
The local user stopped speaking, indicated by the VAD model.
Transcription
Transcribed local user input (both partial and final)
Transcribed bot input. Sentence aggregated.
Other
Pipeline data provided by Pipecat. Please see Pipecat documentation for more information.
Events
RTVI defines the following standard events that map to messages/actions from the bot to the client.
MessageError = "messageError",
Error = "error",
Connected = "connected",
Disconnected = "disconnected",
TransportStateChanged = "transportStateChanged",
ConfigUpdated = "configUpdated",
ConfigDescribe = "configDescribe",
ActionsAvailable = "actionsAvailable",
ParticipantConnected = "participantConnected",
ParticipantLeft = "participantLeft",
TrackStarted = "trackStarted",
TrackedStopped = "trackStopped",
AvailableCamsUpdated = "availableCamsUpdated",
AvailableMicsUpdated = "availableMicsUpdated",
CamUpdated = "camUpdated",
MicUpdated = "micUpdated",
BotConnected = "botConnected",
BotReady = "botReady",
BotDisconnected = "botDisconnected",
BotStartedSpeaking = "botStartedSpeaking",
BotStoppedSpeaking = "botStoppedSpeaking",
RemoteAudioLevel = "remoteAudioLevel",
UserStartedSpeaking = "userStartedSpeaking",
UserStoppedSpeaking = "userStoppedSpeaking",
LocalAudioLevel = "localAudioLevel",
Metrics = "metrics",
UserTranscript = "userTranscript",
BotTranscript = "botTranscript",
LLMFunctionCall = "llmFunctionCall",
LLMFunctionCallStart = "llmFunctionCallStart",
LLMJsonCompletion = "llmJsonCompletion",
Handlers for these can be bound on the client like so:
import { VoiceEvent } from "realtime-ai";
function handleBotReady() {
console.log("Bot is ready!");
}
// Bind an event handler
voiceClient.on(VoiceEvent.BotReady, handleBotReady);
// or
voiceClient.on("botReady", handleBotReady);
// Unbind an event handler
voiceClient.off(VoiceEvent.BotReady, handleBotReady);