Playing with the OpenAI Realtime API + Twilio

[@suyashsaxena7320]

Can we implement a caching layer here as the openai realtime api is too costly to use and I do not want to spend that much on a set of responses. Also I tested the realtime switching between English and Hindi and it worked flawlessly. I am open to any ideas.

[@TwilioDevs]

I asked around internally: "Caching on the Realtime API is hard if not impossible but also realistically OpenAI has their own caching built in already which is reducing the effective cost already. I don't know if you can do things on top of that." You may want to look at Twilio's ConversationRelay which should provide a bit more flexibility since you control the LLM side of it. www.twilio.com/en-us/blog/conversationrelay-voice-ai-made-human While we don't have a video for it yet, we do have a reference implementation using AWS: www.twilio.com/en-us/blog/conversation-relay-aws-reference-architecture

[@suyashsaxena7320]

@ Thankyou for your help! I'll look into it and share my implementation if I figure it out

[@Markmoody-dd6rz]

so how can i get this interrupt code?

[@TwilioDevs]

It's in the repo: github.com/twilio-samples/speech-assistant-openai-realtime-api-node

[@grzegorzgrodzki3855]

Hi! Great video. I have a problem with setting a language other than English. transcription does not work as it should. Sometimes it changes the language to another and shows different words. How can I fix this?

[@TwilioDevs]

Have you tried being very clear in the system prompt that it should always use the language you're trying to work with? I tried giving a prompt to only speak in Spanish and that the caller would always be Spanish and when I tried to insist to the assistant that I only spoke English, it wouldn't break out of Spanish.

[@Rohan-o9w]

Can you make a Video on Using Twilio with Azure OpenAI Realtime API?