Build an AI Voice Agent Using OpenAI RealTime API (N8N & Minimal Code)

[@AhmedMukhtarAI]

📚 Find all the Resources in my Skool community: linktw.in/QcndVy

[@JoshtheFifith]

my question is how were you able to trigger n8n while you have a streaming call and get back data on a streaming call?

[@AhmedMukhtarAI]

The call streaming is between the user and realtime API Agent. The agent can then do function calls to n8n via API call

[@JoshtheFifith]

@@AhmedMukhtarAI this is extremely advanced and complex programming this is not for the average user or even programmer

[@josiahbujanda]

Another banger. That overview chart giving me goosebumps.

[@AhmedMukhtarAI]

My guy 👊🏽

[@gioyouniverse]

Bro, listen. I just jumped on the Ai space and learning, doing my thing you know. YOU ARE THE BEST CREATOR i have seen in this space, I am here before the 100k sub bro hear me out

[@AhmedMukhtarAI]

Thank you bro, I really appreciate it. Keep pushing and learning and I'll see you at the top!

[@Bite-Size-Tech]

The ending was funny, your shushing as if it’s someone lmao

[@AhmedMukhtarAI]

I've just noticed in the demo the agent booked 9am instead of 5pm 😅 I had a look at the cause of this and did some more testing- Basically the Realtime API's TTS model can only be set to the whisper model currently. Which is not the best model at transcribing audio, so the agent must've read in 9 instead of 5 🤦‍♂ Some other notable issues with Realtime API: 1) The reasoning and decision making isn't the best either - even with good prompt eng, it is not following instructions and calling the tools correctly. 2) It is super expensive to run! but hopefully Like all the other openai models it would become alot cheaper over time.

[@BennuBirdPred]

Pretty amazing stuff. We started with Whisper, switched to Deepgram and ended with Assembly as it was the only one that did well with accents in my limited experience.

[@AhmedMukhtarAI]

@@BennuBirdPred That's useful to know, I haven't came across assembly but will defo check it out. Thanks

[@unime1988]

What's the over all costing of this set up, including api and other charges coz that the key, today , it a not a surprise or shock of what an AI and low code platforms can do. It's all about the final costing , This is the same reason that great invention never see the day light of success. So if possible, please do mention that all well in future video.

[@ahmedsomir]

Great Ahmed .. waiting for the next

[@benjaminbar4365]

🎯 Key points for quick navigation: 00:00 *The video focuses on building a voice AI agent using OpenAI's Real-time API for natural-sounding conversations, handling multi-location appointments, and quick responses.* 00:42 *The video addresses the main concern of clients wanting a more natural and realistic voice AI agent that can handle complex calendar appointments.* 01:11 *The system integrates Twilio, N8N, OpenAI's Real-time API, and Pinecone to manage inbound calls, chat history, and appointment scheduling.* 02:21 *Call management involves retrieving chat history, storing call transcripts, and scheduling appointments using N8N's workflow capabilities.* 03:17 *Code is hosted on Replit for fast deployment, and different variables, including API keys, are set for functionality.* 04:11 *Workflow in N8N checks Google Sheets for call history and manages initial response messages with JavaScript nodes.* 05:22 *Various functions, such as Q&A and schedule meeting, handle user queries and manage appointment bookings.* 06:04 *Integration with Pinecone Assistant allows for interaction with a knowledge base, streaming responses, and managing appointment schedule workflows.* 07:30 *Replit deployment requires specific URL configuration in Twilio; testing involves real-time interaction with the voice agent.* 08:12 *Demonstration includes making a booking, configuring details like name, email, purpose, and preferred location.* 10:05 *Alternative times are offered if the proposed slot is unavailable, showcasing system's adaptability.* 10:21 *The demo highlights the efficient response time, ability to interrupt, and previous call history recall.* 10:34 *The video encourages joining the creator's community for access to code and templates, appreciating audience support.* Made with HARPA AI

[@Androgamerzpro]

Can I use any other SIP TRUNKS alternative to tiwilo like fonoster? and I need to modify the web socket code?

[@massibob2004]

Nice job ! Why do we have to code if we can do all the pinecone stuff and implement the realtime voice in N8N ?

[@AhmedMukhtarAI]

@massibob2004 Because n8n only supports webhooks (request / response) so we aren’t able to have open live connections (websocket) needed for openai realtime API communication

[@massibob2004]

@@AhmedMukhtarAI Do you know the scale limit of an n8n project ? it depends on the power of the server only ?

[@contractorwolf]

excellent explanation, thx

[@JR-joren]

hey, i am a none coder, is it worth it for me to register to your community, or should i need coding knowledge? thanks

[@AhmedMukhtarAI]

Hey, no coding experience needed. My aim is to help non technical people from 0 to building these kinda systems!

[@massibob2004]

Hello, where can we find a cheaper deployment solution please ? because Replit seems so expensive

[@AhmedMukhtarAI]

@massibob2004 Its just a FastAPI backend. Could be deployed on any service you like. A good example would be Render or Railway.

[@gjsxnobody7534]

can you show this with Deepgram? seems Open Ai Whisper is quite slow.

[@AhmedMukhtarAI]

I believe currently openai only support whisper model for realtime API

[@gjsxnobody7534]

@ correct. That’s why Groq and Deepgram would be faster. I just don’t know how.

[@AhmedMukhtarAI]

@@gjsxnobody7534 oh if you mean just not using realtime api then yes, it would be possible to build it from scratch using deepgram groq and a voice model like 11labs. I'll try and see if I can build it

[@moses5407]

great work! but still not clear why you don't do all of this in n8n? It seems perfectly capable of handling this.

[@dabrodev]

I think it's because of the websocket protocol, which is needed for the openai realtime api.

[@AhmedMukhtarAI]

Yes exactly, n8n only supports webhooks (request / response) so we aren’t able to have open live connections (websocket) for realtime communication

[@dreamlyricist_]

where can I find the full code...?

[@AhmedMukhtarAI]

in the community

[@krisszostak4849]

Hi Ahmed, great video! One question though. Is there any way to protect the system from the "timewasters" I mean people who just want to troll it and spend a lot of time on the phone wasting money?

[@AhmedMukhtarAI]

its possible to prompt the Agent to professionally direct the conversation back if it goes off-course. However it's hard to mitigate against trolls that might just play along and pretend to want to schedule a call and keep going around in loops to just waste time. Another way is you can limit the call duration so it will automatically hang up after it reaches the limit.

[@LucasGabriel-jb9mx]

Hello, how are you? My name is Lucas and I'm from Brazil! We don't use Twilio here. Is it possible to adapt this agent to a WhatsApp agent that responds in text? And then save the information in the sheet and schedule it? I discovered your channel today, keep up the great work!

[@AhmedMukhtarAI]

Hey man, yes that's possible with N8N too. Actually, my next video shows this, so keep an eye out in next couple days

[@luizrodrigues807]

I'd love to be able to get this working and get in your skool, but USD is very expensive for me, as I live in Brazil. Sad, but true.

[@AhmedMukhtarAI]

I know, I wish skool did regional pricing. I just checked and it doesn’t seem to be the case unfortunately.

[@gnaneshnayak4592]

Is there any alternative for realtime api?

[@AhmedMukhtarAI]

Yeah you could use VAPI

[@data_scientist_vlog]

Can you provide code of this video ?

[@AhmedMukhtarAI]

you'll find all the code and templates inside the skool community: www.skool.com/ai-business-accelerator/about

[@tomthompson1198]

cool but i want localy ran everything free of monthly fees. so plz do something like or better to: local n8n, local llm, and so on. and not interested in the skool platform.

[@build.aiagents]

Phenomenal

[@aireferat]

you be broke fast, 1h testing this the cost is to high cant use daily