Client Side Tool Calling with the OpenAI WebRTC Realtime API

Рет қаралды 26,680

Cloudflare Developers

Күн бұрын

Пікірлер: 68

@cooliomdp 29 күн бұрын

I think this video deserves some kind of award. You are amazing.

@mudaserali1862 28 күн бұрын

agreed

@toddwmac Ай бұрын

Love the vid...nice delivery and fun example. It is wild to think that just a -few- couple years ago, it could have been a keynote demo that blew the lid off a convention center somewhere. It makes our days of demoing _drag-n-drop_ or _OLE_ , on stage in front of thousands a very, very distant memory. A great time to get into, or in my case back into, coding. Thanks!

@craigsdennis Ай бұрын

Honored to be a part of your journey back into coding! Keep me posted on your journey! 💪🚀

@sapiir 27 күн бұрын

I'm really glad this video popped up on my for you page. Already did star a few of your repositories. Awesome work keep it up!

@alex_strauss Ай бұрын

oh maan crazy.. cool that you demostrated first and then jumped into the code.. like a magician that reveals the trick haha

@craigsdennis Ай бұрын

🐰🎩

@DanielTalero Ай бұрын

Amazing video. Seriously for a small company like the one that I have, this video is so useful

@bennettwaisbren Ай бұрын

Whoa... Wild. This stuff is starting to resemble magic. Very interesting times

@craigsdennis Ай бұрын

Bonkers right?!

@maskman4821 Ай бұрын

Awesome, this hides the complexity of low level webrtc api, I would like to see realtime message and video chat and robot hand in the next episode🥰

@KManAbout 26 күн бұрын

You are the god of demos

@craigsdennis 26 күн бұрын

☺️

@o1-preview Ай бұрын

siiick!!!! I was just working on tools after setting up rtc, can't believe there's already a video on it

@craigsdennis Ай бұрын

Have fun! It's so incredible!

@FuzailShaikh Ай бұрын

Pretty cool, one suggestion - for tool calls getting number of fingers you can use enum 1 to 5 to ensure it doesnt hallucinate with results and use strict mode

@craigsdennis Ай бұрын

Awesome! Thanks for the feedback, I'll make it better!

@jmg9509 Ай бұрын

Structured Outputs for the win!

@suicidequad Ай бұрын

explain?

@AtThePeak-e4b 7 күн бұрын

Fantastic, it's very interesting

@mstx8047 Ай бұрын

Great vid. Did you post link to your sample code somewhere or I just missed it. Also, if you implemented strict mode like other comments suggested which you liked, it would be great to see how one can implement some sanity here. Thanks again - also love your enthusiastic tone and smiles. Happy holidays.

@craigsdennis Ай бұрын

Code is in the description of this video prefixed with Repo

@RussellDeming Ай бұрын

Love your stuff 🤙

@danielohkay Күн бұрын

How can we on the server side within the webrtc relay hook into the messages flowing through data channel?

@andrewcbuensalida 29 күн бұрын

How do you do function calling on the server side?

@alexdolbun Ай бұрын

Super informativ about WebRTC

@noelmena138 11 күн бұрын

Hi, do you have any idea how a conversation could be implemented between three actors, the microphone and output sounds of the equipment and the Openai realtime API?

@kenwarner 26 күн бұрын

what a time to be alive

@manoharants Ай бұрын

Is it possible to connect to Azure OpenAI?

@Dron008 25 күн бұрын

Can o1 call functions while thinking or just when it is finished?

@ahmeda.abouserie6096 3 сағат бұрын

Thanks a lot for the video! I noticed that when I reach the daily limit of openai the api will stop responding but there is not indication for that of the issue. Any idea how can I know this error? I tried to print the opeai API response but the status still coming as OK

@GrowStackAi 28 күн бұрын

Achieve more with AI by your side 💫

@ThiagoCoutinhos Ай бұрын

The Terminator is knocking at the door, one finger at a time.

@JacobIlin Ай бұрын

So cool!

@ABOfficial-YT 15 күн бұрын

very cool!

@oneworldplan6843 13 күн бұрын

This is wonderful! How do I reach you?

@jonas_ai Ай бұрын

Wow this is wild

@sfsadfsadfasdf Ай бұрын

this is huge

@Aryan_Mayya Ай бұрын

how to protect openai api key from user ?

@craigsdennis Ай бұрын

It's on the server using protected secrets, I relay it through. If you did it directly there are other workarounds.

@EricFriedman Ай бұрын

This is amazing thanks for sharing

@craigsdennis Ай бұрын

Thanks for watching!

@MarioCaronna 22 күн бұрын

I would like to a chatbot similar to Jarvis running a Raspberry Pi4. Online, many examples, but none using the advanced feature of a real-time API Could you assist me or suggest a tutorial? I believe it should be quite easy to create a on a Raspberry Pi but using the real-time API instead of the old one, correct?

@Trouvaille-Jack Ай бұрын

Cool!

@sitedev Ай бұрын

Insane!

@brando2818 Ай бұрын

Sheeeeeesh.

@craigsdennis Ай бұрын

Riiiiiiiiiiiight?!

@moresignal Ай бұрын

Fantastic! I got this working on my laptop in about 10 minutes and was able to add a new function which checks my battery level in about another 10 minutes. It's very satisfying when someone shares such simple code that gives a working starter platform . Have you played with anthropic's tools standard thingy yet?

@jmg9509 Ай бұрын

Love it. But technically, just like with your robot arm there, a person can "tell" the ai to fire a gun round when a certain condition is met. Like a person walking by, for example. This extends its capabilities into the real world, just like your robotic hand example. Don't get me wrong, i'm super stoked about this, and will be using it. Just giving some hypotheticals that are now possible for a dystopian set future. But i expect a utopia. I am optimistic!

@craigsdennis Ай бұрын

Someone would have to write the "shootGun" function in this example. Definitely keep the human in the loop!

@IceMetalPunk Ай бұрын

You can tell a person to fire a gun, too. These aren't new problems.

@jmg9509 Ай бұрын

@@IceMetalPunk @IceMetalPunk Yes, but the difference is that the person shooting the gun has to be present, even if it's from a sizable distance. And that's going to happen at an exact location and time, and a vantage point can be determined based on the trajectory of the shot. The surveillance and forensics nowadays is so good, that it is very difficult to get away with this kind of killing, even if it is not face to face. On the other hand, if a person instructs an ai to carry out a killing days in advanced so that the person can escape, this leaves new ways for criminal tactics and evasions. It can detect the exact face/profile of the person to carry out the act on with cheap equipment, and teleoperated weapons. You could create a gmail with a fake name and info, drive to a public location in a stolen tinted vehicle with free wifi on a cheap or stolen, dumpable machine, get an LLM api key, mask the functions as something benign that carry out the deed so that the LLM system doesn't flag it, throw the device out the window (with your mask on of course - smile for the cameras), drive off the grid where there's 0 surveillance and devices on hand, dump the vehicle, have weeks, months or literally even years before the AI carrys out the act without a trace because it can act and adjust almost as intelligently as a human. New laws will have to emerge, and those often take a while to pass in courts. More importantly, new ways to track these events. This isn't too convoluted a task for anyone to do. In fact, it's scarily easy now. It doesn't even have to be a gun-type weapon, it could be releasing poisonous gas, an explosive, or something less seeming and intricate; like having a drone with face detection activate, fly to the target 4 years later, and fire a lethal poisonous dart. How scalable is this? One dart...hundreds? The main asset that AI could buy you in these hypotheticals, is time. It becomes way more difficult to solve crimes the more time passes. And with it, you could have a lifetime before releasing the 'trigger', without you even needing to monitor or maintain it. I know this is exhaustive to read, but people are clever and I'm sure my hypotheticals aren't even as intricate and untraceable as it can get.

@jmg9509 Ай бұрын

@@craigsdennis It could be masked as simply as "pushButton()" or "sendSmiley()" to avoid flagging from the system. With its function body being to detect a specific face and then send json to a machine which uses its payload to know if to release the trigger. The activation of the machine could literally be a '🙃' smiley face that was received from the json . Its a seperation of concerns that makes the AI unaware of what it's doing.