Senior Engineer’s AI Assistant: Realtime API and AI Agents built to SHIP for you

[@moresignal]

It's coming along nicely. Thanks for taking the time to share. It's a lot of extra effort for you but it's appreciated.

[@2034-SWE]

Absolutely Perfect sequential and contextual display throughout this video. I love how you almost never have to explain anything to us directly, and instead let the pre-cursor screen focus (i.e. looking at certain parts of a page) do the talking, in conjunction with the subsequent verbal request & speech/file outputs

[@stanleylu3625]

You're building Jarvis from iron man. Love the videos and generously sharing your code. I can't code but with your channel and AI I'm able to create full apps with just prompts. Thank you

[@SamiSabirIdrissi]

EXACTLY

[@goforit5]

SAME!

[@IdPreferNot1]

lol. literally the 2 latest url's i copy/pasted to a .md for my claude Project Knowledge context. I borrowed your code for the tools, memory manager, utils and was able to fold them into the swarm agent framework i was working on. Im a new python coder so couldnt believe i was able to get it to work but its awesome. Thx!

[@zoltanabonyi3307]

Please continue developing this project. Absolute mind-blowing stuff.

[@higon99]

No it's not the tool that is impressive. It's you and the way you use it are impressive. I can not repeat long sentences until llms get my task done right eventually... but surely you have me excited for the future. Thank you for sharing the experience.

[@dwayneholmes5049]

Wow! This is exactly what I’ve been working on for Release Engineering and DevOps. I’m really impressed and will be reviewing your code. I agree ai agents is about doing tasks in parallel….didn’t think about the speech thing!

[@SamiSabirIdrissi]

this is F***ing wild, cant wait to try this. thank you for sharing!

[@Sub0x-x40]

this looks fun af. if only my dad was alive to see this, its like commander data from star trek. it would truly blow hes mind

[@ph4nt0mcz130]

Honestly, as an engineer you have to use this tooling to filter the information that you recieve. Imagine how many empty words you get now from “unskilled” product and management people. With the unsupervised power of AI on their side it will triple in size + be hallucinated. Funny thing is that we are getting into the encrypted channel and will have to filter out the AI with the AI

[@nathanstreger3851]

I'm giving an AI presentation to my dev department and interested parties about how we're currently utilizing AI in our workflows/products and what I see for the future. Going to get this up and running locally to "Wow" them at the end.

[@parkerrex]

Paused at second 29 to read. This is it man!

[@IowIy]

Awesome. I agree with what your thoughts are on personal AI assistants with memory management.

[@stonedoubt]

Dan… thank you for sharing this.

[@K.F-R]

Great work. Deeply appreciate you sharing as you go.

[@juliotriana4449]

Excellent work. Love your videos.

[@software_valen]

nice vid man! i'm doing something similar just to play around and i'm 100% sure that you can achieve the same results just using any whisper locally (easy to setup btw) and an llm surrounded by some agent logic (i applied the react paper here). in this way you'll cut off your costs to zero. but in case you choose to use the openai apis, you will also cut your costs in an unimaginable way.

[@aibeginnertutorials]

Brilliant as usual. Thank you..Looking forward to testing this.

[@TonyAlfredsson]

This is just awesome! I have been working on the real-time API integration into my Flutter project since they released it, but I am having some challenges. I am also using Aider to code and will fork this right away and start with some implementations. Currently saves the transcribed audio into Supabase, so something similar should be a fun project. Thanks!

[@YoneCortopassi]

aistructuralreview AI fixes this. "Realtime API AI Assistant Launch"

[@TeamDman]

great demo! gives some good inspiration for my own tooling

[@NLPprompter]

for someone who thinks randomly speaking is never been my forte, typing us, with typing i can re read what I typed. but yeah parallel building stuff... I'm in!

[@ibbbyscode]

Wow! Amazing stuff. Thanks Dan

[@k22marie]

hey Dan - I love these videos from a senior engineer's perspective. Videos are great, well edited and provide tons of content in an appropriate amount of time. My question: how does a junior engineer leverage this technology appropriately while they're skilling up?

[@indydevdan]

Hey @k2marie, great question. Right now everything is changing in the tech ecosystem. The best thing junior engineers can do is USE this technology and get your REPS in. Theory, research, and 'best practices' are only great after technology has been established. Right now the experimenters, tinkerers, prototypers and builders that are USING this technology for EVERYTHING they possible can will emerge winners. My advice is to spend as much time building, testing, and using the new ai tech, and try to pay attention to people who are ACTUALLY building - not just talking and speculating.

[@JonasLindquist]

This is really impressive, thanks a lot for sharing!

[@mindcastsoftware]

I'm curious if puppeteer/playwright could be set up as a tool for the assistant to perform actions on websites and assess results?

[@agenticmark]

an agent that runs on github issues and produces PRs and then deployment releases for me just to test each. a list of ports to match the PRs. cursor is bad ass, but it doesnt do that yet

[@johnMcKartney]

This is the product that will be shipped to the public and will create a new revenue stream for the big 7, justifiying the immense investments these companies made for their AI-training data centers. I'll give it less then a year and we will have atleast one official release, if not build-in assistants in the new generation of phones and PC's. 12 months ago i tried building something like this but i couldn't manage to do it, cause i'm just not good enough, so i'm glad that this will become a reality very soon. Exciting times to be alive.

[@stevebim000]

Another great video Dan. Could you build in the interruption instead of the push to talk ? Also the streaming output ?

[@RybkaZolotse]

Any thoughts on something akin to "admin" level privilege to the speaker? Could ADA recognize one person's voice giving directives that had "junior" level permissions as a non-compliant condition?

[@vastvitamins1966]

Are you planning on put together a paid course that goes step by step into setting this up I'd would love to see that.

[@uhtexercises]

Great stuff, as always. Really love how this is evolving. Can't thank you enough for sharing the code. So much to learn from it. Let us know how we can give back.

[@aerotheory]

I'd like to get a feel for cost/token over productivity.

[@ia7cast]

is uv compatible with miniconda? would be great a tutorial about how install this repo

[@TimNagle]

Insane. This is so good

[@mrpocock]

Do you have agents that can write,test,run,debug a small code project? Bail back to you if they get stuck.

[@IlkkaNisula]

Just curious did you track what was the total cost of this demo using the realtime api and models? I tried some simple realtime api for a just a few prompts and cost was whopping $2.5.

[@henno6207]

a relevant question, but also I imagine the price for this plummets rapidly in the near future.

[@2034-SWE]

Realtime API alone is approx $15 per hour - but as the person above commented, it’ll go down. Fast. Look at GPT-4 vs 4o costs for example View the current pricing as irrelevant. Instead, focus on how only

[@psinke]

very nice. Would this be possible as a plugin for something like VSCode? Or would you run into limitations for what a plugin is allowed to do?

[@cyberthugFi]

IDD is the new renaissance goat! I have questions tho, is the Model only able to help you on code? Can you make it see through the camera with Python scripts ;), can it have real world perception if i assign other tools using python or java, other languages, can I customize the ai personality ? How much it costed in tokens just to shoot this one video?

[@FurfelOfficial]

Hi, I really enjoy this series of videos! However, in the last one when I saw the pricing for realtime API, I thought: why not just use Whisper 3 turbo locally? The only thing needed would be to detect then when the speech was started and finished (with let's say 1.5 second silence delay). I just tested it on M1 and it works very well, and doesn't take that much memory.

[@jefferystartm9442]

Please we need a Live Kit version , also have you check E2B ( bet you have not 😂)

[@EdwinFairchild]

man i was so close to playing with your code till i realized i dont have access to the realtime API which is weird since they usually give me other betas

[@edzynda]

What are your thoughts on Anthropic's new MCP spec? You see this becoming the standard when adding extra capabilities to LLMs?

[@indydevdan]

BIG if USEFUL. Video is coming. Working through the value prop.

[@fbalco]

are you triggering on °go ahead and ... ° or is that just how you like to say it for videos?

[@cameronyking]

Can this be a VS Code extension? Would you build or could I port?

[@tharunt3342]

Can you provide a video for Complete guide

[@serafinalcantara5520]

Would the assistant be as responsive and efficient to use one of the large open source llms locally like llama3.2/quen/visavis or a quantized model with a speech2text -> inference -> text2speech? As always great content 👍

[@Aguiraz]

man this is hella cool, but can you help us give it a UI (streamlit?) as well for text input ? we often cant use mic and this tool is too powerful to leave bound to mic

[@numbah16]

Love it. It's Scrape* though dude, not Scrap lol

[@louisduplessis5167]

thx for the vid bro. what voice do u use ?

[@christianreid7529]

Thank you so much 🙏🙏

[@grymvision3094]

Forgive the question since I think you said it at some point, but what LLM are you using? Would this be viable with a local LLM? Obviously that depends on the complexity of the request, but in general, how viable would a local LLM be at this moment in time?

[@ix4564]

What is this exactly? What is the talking agent that can do these things ?

[@sephirothcloud3953]

Can you please tell EVERYTIME the AI COST of your video? Otherwise, I have to spend money every day for testing if it's affordable enough to use for my purpose

[@bukitsorrento]

Every AI content creator should know this basic principle.

[@d1m18]

Very good point. All these guys showing tech that we can't afford isn't very useful

[@bukitsorrento]

It will be useful for the future when costs go down by 90%. Dan is doing us a big favor by making this video, showcasing the technology despite the current high costs, using his own money.

[@cyberthugFi]

@@bukitsorrento well said

[@adamking6957]

​@@bukitsorrento Why can he not then just say how much it cost? I mean, he hearted the comment, but he has not updated the description to include the cost. Why not just do it?

[@alessandrofrau4196]

Great video. I'd love to make a solid use of Realtime API but it's so expensive, yet blocked in the EU 😮‍💨

[@tomaszzielinski4521]

Oh man, I'm already jealous :>

[@tlatoanimachi]

Beautiful

[@ScottzPlaylists]

👍👍👍 Can you put together an Open Source version, that duplicates this as best as possible❓❓❓ That the real future, but seems a little behind SOTA ❗

[@mrpocock]

I still want a better persona. I don't need it constantly asking me what else it can help me with.

[@pgg131]

Time to be another VSCode fork

[@cameronyking]

exactly what i thnk

[@aiplaygrounds]

Cost is to high to run it at the moment an average of $3 p/m

[@MdNaimulIslam-y9l]

wow wow

[@seadude]

kzbin.info/www/bejne/ZmqToIVjYtiSlZo "The future is already here - it's just not very evenly distributed." William Gibson ;)

[@Doodiligent]

This is hot 🤤

[@d1m18]

Too much fluff and politeness