​
Real-Time Voice and Video Inference (RTVI)

RTVI is an open standard for real-time AI inference.

The RTVI project’s goal is to make it easy to build AI voice-to-voice and real-time video applications.

  • Applications developers should be able to write code that can use any inference service.
  • Inference services should be able to leverage open source for the complicated, client-side developer tooling needed for real-time multimedia.
  • Any developer should be able to trivially stand up real-time AI infrastructure for small-scale use, testing, or prototyping.

​
SDKs

​
JavaScript SDK

API Reference

See the API Reference docs for RTVI Web

You are reading the docs for the RTVI JavaScript SDK. This is an Open Source implementation of the RTVI standard intended for use in web browsers.

Because RTVI is a vendor-neutral standard, this SDK does not implement the lowest level of client-to-cloud communication. We call this the “transport” layer. To build RTVI client apps, you’ll use both this SDK (realtime-ai) and a library that implements an RTVI transport.

For example:

npm install realtime-ai
npm install realtime-ai-react
npm install realtime-ai-daily

Once you have both this SDK and a transport library installed, a very simple Hello World app (using Daily as a transport) looks like this:

import { DailyVoiceClient } from "realtime-ai-daily";

function myTrackHandler(track, participant) {
  if (participant.local || track.kind !== "audio") {
    return;
  }
  let audioElement = document.createElement("audio");
  audioElement.srcObject = new MediaStream([track]);
  document.body.appendChild(audioElement);
  audioElement.play();
}

const voiceClient = new DailyVoiceClient({
  baseUrl: process.env.NEXT_PUBLIC_BASE_URL || "/api",
  enableMic: true,
  callbacks: {
    onTrackStart: myTrackHandler,
  },
});

voiceClient.start();

Of course, a production-ready application is more lines of code than that! You may want to look at the RTVI React SDK, which wraps all the functionality from this SDK in React helpers and hooks.

There are also RTVI SDKs for Android and iOS. Links to those are in the nav sidebar.

Pipecat