OpenAI using WebRTC in its Realtime API is obviously exciting to us here at webrtcHacks. Fippo and I were doing some blackboxing of this based on a quick sample the day of the WebRTC announcement so we could look at it in webrtc-internals and Wireshark. Some weeks later, my daughter was interested in using ChatGPT voice mode. That reminded me of the many old cardboard Google AIY Voice Kits I made with her. Could these be repurposed to use the new OpenAI Realtime API instead of Dialogflow?

Adapting that OpenAI Realtime WebRTC sample to run on a Raspberry Pi was actually pretty simple. However, as I got beyond the basic “say hello to the bot”, I noticed the documentation for using the Realtime API with WebRTC and Data Channels was not great. My initial attempt to vibe code something quick failed despite my Retreival-Augmented Generation (RAG) attempts with OpenAI’s Realtime docs. This failure forced me to do some old-school thinking and experimentation for myself. I ended up coding this the traditional way with logs, network inspection, and trial and error. That ultimately led me to this post to document it all.

In this post, I will:

• Provide a step-by-step guide for using the OpenAI Realtime API with WebRTC
• Highlight some best practices for working with WebRTC along the way
• Comment on the messages and flows
• Highlight how to use functions with the Realtime API’s data channel

Many thanks to pion founder and past webrtcHacks interviewee, Sean DuBois of OpenAI for his review and suggestions! I will be asking him to chat more about this.

If you want to see what this is before reading, try it below. You must click the Settings button and paste your OpenAI key for this to work. (ToDo: better warnings in the app on that).

This app is just an iFrame hosted from the webrtcHacks/aiy-chatgpt-webrtc, but if you are not comfortable entering your credentials here, then clone the repo or copy & paste the code from the source into a HTML file and open it..

Open demo

You can see my final example code on the repo.

Some notes on this before we begin:

• It is a single file example – just HTML with some vanilla JavaScript inside <script> tags
• This is a simple, but realistic example
• I broke this into 2 <script> tags
  • We’ll focus on the first with the WebRTC, API setup, and messaging logic
  • The 2nd is mostly UI logic – we won’t cover that much

The Flow

As with most things in JavaScript, most of the code is asynchronous so the exact timing and ordering may vary. In addition, this does not describe the many things WebRTC does for you behind the scenes. Stay tuned for more details on the WebRTC implementation from Fippo soon.

Getting the Microphone Audio

getUserMedia takes a constraints object as a parameter. The {audio:true} tells chatGPT we just want to capture (adding video: true would also capture video, but that is not supported (yet?). We are keeping this simple here, but there are a variety of options you could put here. Generally, you should use the defaults.

This will use the default microphone option configured in your browser settings. There is an enumerateDevices API that will list available devices and make those available for selection. If you go down that path, then you should also listen for device changes.

Also keep in mind, the OS and browser usually both require permissions to share the microphone. I did not do this, but you can check these permissions even before the user starts the session with the Permissions API. If we had a very short TTL on our token, then I would have done the media capture first, since the user could take some time to give the proper permissions.

The Media Capture and Streams spec puts the captured audio inside a stream object. That stream object can contain one or more track objects. We only need to use the track later. Unless the user has some kind of advanced audio setup, the stream should only contain a single track so we specify the first one in: [track] = stream.getAudioTracks();

WebRTC Setup

RTCPeerConnection

Adding audio streams

Note: It is common to show a volume visualization to give the user an indication their mic is working. See this WebRTC sample for an idea of how to do that. In our case, we display the audio transcription, which helps serve that function.

addTrack adds the local microphone audio to the RTCPeerConnection. Including the stream helps the peer connection maintain the track association with the stream, which is a good practice.

Data Channel Setup

I will cover the messages and interactions in the Realtime API Message Exchange section.

WebRTC Offer / Answer Process

This method builds connectivity options and media details known as an “offer”. WebRTC sends the offer to the remote party – also known as the “peer” – behind the scenes. Unlike the official OpenAI references, I use the preferred “implicit setLocalDescription style” here (the offer is created automatically).

In WebRTC, it is up to the developer to figure out how to send that offer to the other end and send back the response. OpenAI does that via their https://api.openai.com/v1/realtime endpoint. We send the local description SDP and the API token in the body.

Sidenote: this Offer/Answer exchange is similar in concept to the IETF’s WebRTC HTTP Ingress Protocol (WHIP) (video on this).

The API endpoint will then respond with something like this:

That is the archaic session negotiation format known as SDP. The SDP defines available networks, media codecs, and datachannel parameters. You shouldn’t have to worry about this, but if you are curious about the full gory details, see our SDP Guide.

Next you need to pass this to the peer connection’s setLocalDescription method:

That should create the bidirectional WebRTC connection with audio flowing in and out the peer connection with ontrack and data channel open events firing. The OpenAI docs don’t mention this, but to avoid asynchronous timing issues, you should listen for the peer connection’s connectionstatechange event to switch to connected before proceeding with any logic that assumes you are fully connected.

You do not need an ephemeral token

The OpenAI docs I saw only referenced getting an ephemeral token to authenticate the client. Ephemeral tokens make sense in most client-server apps when you want your server to authenticate and give temporary access to connected clients. In my case, there is no server since it is all client-side. I started out generating an ephemeral token (reference code for that here) because I thought it was the only way.

Sean DuBois at OpenAI pointed out that this isn’t necessary. You can use your OpenAI API Key to authenticate directly from the client, as you can see in the fetch code above.

But what if this was a more normal client-server application? In that case, you could send the pc.localDescription.sdp object to your server and have your server fetch the answer (the await fetch(`${baseUrl}?model=${model}`, {method: "POST", body: pc.localDescription.sdp, headers: { Authorization: `Bearer ${apiKey}`,"Content-Type": "application/sdp" },});  ). Then it could pass this back to the client. Depending on timing needs and your server’s responsiveness, this approach could make sense in many situations.

Also, consider the Time To Live (TTL) between the ephemeral and direct API token approaches. I found the expiration time is always 30 minutes from the creation time when passing the API Key directly. I did not see a way to change this. I got a 2-hour Time To Live (TTL) when I experimented with ephemeral tokens. That is also hardcoded to 2 hours with no option to set expires_at. 2 hours seems long and is a common complaint on OpenAI’s message board.

To Do: see what happens to the live session when these tokens expire.

Finally note, the ephemeral token option does let you establish session parameters before you do anything with WebRTC. As I will show in the next section, I chose to do that later via a session.update do demonstrate now non-user, system instructions and other parameters can be updated at any time.

Realtime API Message Exchange over the Data Channel

session.update & available session parameters

As mentioned in the previous section, I chose to defer setting up the session until after the WebRTC connection was established.

Here we start by sending a system message using type session.update:

Let’s break down the session object we send:

• • instructions – the session instructions taken from settings that guide the default model behavior – i.e., “You are a friendly assistant to a middle school girl. Speak in a slightly enthusiastic tone. Occasionally use gen alpha language and irony. Your knowledge cut-off is October 2023.”
  • voice – which speech synthesis to use from the options listed here
  • temperature – how random the responses should be on a range of 0 to 2 where zero is deterministic (you get the same output everytime) and 2 is entirely random
  • input_audio_transcription – the realtime model takes speech directly as an input without converting it to text first. To provide the text, OpenAI uses its Whisper speech-to-text engine to convert the speech input. I have noticed this transcription can differ from what the model seems to interpret. You can specify a language as a parameter for increased accuracy if you don’t care about multi-lingual support. You can also give it a prompt to help improve the output. I didn’t play with this, but I imagine adding proper spellings of names and additional context details would help with the Word Error Rate (WER).

• tools – I’ll cover functions in the next heading below
• tool_choice – this goes along with the above

Note you can’t change the voice once synthesis starts.

Examining session defaults

The session.created event (covered in Incoming Events) returns the default session parameters as a JSON object:

You can set all of these. I will cover the tools option in the Using a function call to respond before ending a session section.

Don’t touch the audio formats

Also note, input_audio_format and ouput_audio_format don’t change the codec used by the browser so you don’t need to set this. Fortunately, the WebRTC API uses Opus, which is fortunate because it supports wideband audio and has nice features like Forward Error Correction (FEC) by default in browsers for improved audio quality. The pcm16 (or g711_ulaw/g711_alaw) selection is only used by the internal media proxy.

Turn detection

One piece I did not include here is turn detection – i.e., how sensitive the API is to interruptions from the user and background noise. You can see the parameters for this here.

WebRTC includes some basic noise suppression by default and uses its own Voice Activity Detection (VAD) to minimize bandwidth during silence. The Realtime API needs to detect when the user is speaking and how quickly to interrupt. The defaults worked fine for me in a reasonably quiet environment – both on the browser and in my Raspberry PI AIY Voice Kit hardware. I see where a device manufacturer might want to tune some of these parameters.

Third-party noise suppression libraries like RNNoise could help clean noisy input. I am curious if that would make much of a difference on the LLM’s ability to understand the input. If you were doing this server-side, one could even use Silero VAD to control turn-detection manually.

Start Instruction via response.create

Incoming Events

All the other events directly update the UI with a transcript of the dialog except session.created and response.function_call_arguments.done which I will cover next. I only included session.created to examine the default session options. I’ll cover functional calls later.

Other events

Using a function call to respond before ending a session

When the model detects something that looks like what is described there, then a response.function_call_arguments.done event is returned. The function response can include parameters, which I did not need here. Function calling works the same as described in the documentation, except the messages are sent over the data channel.

As described in the previous section, if response.function_call_arguments.done matches “end_session”, I call the endSession() function:

endSession() checks to see if the data channel is still open. If it is, it sends a response.create to generate a good-bye message and then listens for the next output_audio_buffer.stopped before closing the session.

track.stop() ends the microphone capture. pc.close() closes the peer connection, stopping transmission and closing the data channel.

When the user clicks the end session button, I send a response.cancel to stop any in-progress responses. This logic is unnecessary for speech input because saying something while the AI is speaking will trigger turn detection and stop the speech output.

When I started this project I wanted to do everything in Python. Then I discovered I couldn’t get recent versions of anything to work with the AIY Voice Kit hardware and associated code that was archived 4 years ago. It is easy to load a browser on a Raspberry Pi and that has the advantage of being portable to any web browser, so I went down single-html-file path described here. I am still working on cleaning this up, but you can see my AIY Voice Kit chatGPT Realtime code with some hardware interactions in the same repo on the aiy_voice_kit branch.

Here is a demo of that: