Real time AI Conversation Co-pilot on your phone, Crazy or Creepy?

[@m4tthias]

Would be funny when both parties of an interview who uses the co-pilot finds out about each other.

[@jaredlalal]

Oh it'll happen, it already does to an extent, we use ai to help us find jobs, they use them to scim threw applicants

[@figs3284]

You just gave me an idea 👍

[@adolphgracius9996]

Lmao😂😂😂

[@jaredlalal]

@@figs3284 I love random ideas :0, can I have a spoiler? 🙂

[@TyFactorCGI]

this is so great!!! what fun! I picture the underqualified candidate who's code crashes mid interview and all of a sudden can't answer 1+1. (: But seriously, I have some cognitive disabilities, even something to keep me on track and succinct in answering my questions is absolutely invaluable. I am seeing this as a reminder tool for me, not an answer tool per se. so powerful and enabling for somebody like me. I imagine it helping my confidence which will allow me to be more of myself and let my natural attitude come out instead of being clouded with mumbling, with racing thoughts.. (as many of my other high-frequency colleagues have) thank you very very much for taking the time to share with us!

[@haraldlasshofer214]

this is crazy. Love your content. I will definitely use the end product if you publish one. Keep the great work up man!

[@Romathefirst]

Best AI channel hands down

[@Soniboy84]

Hey Jason, interested in the iPhone project (tho android would be better). How about adding an extra layer to this flow that translates the text to a different language and speaks it out loud? We could then have real-time conversations with someone who isn't the same language as us. Here's the use case. You hand over the left earbud to your Polish friend. You use the right earbud. Ideally both earbud would have a mic. The application on the phone would: listen to the speech in English => do a transcribe on the chunk => do a translate to Polish on the chunk => speak out the chunk in Polish. Then when the Polish person is talking, it'd work the opposite way.

[@HarpaAI]

🎯 Key Takeaways for quick navigation: 00:00 *🎬 Introduction to Real-Time Conversation Co-pilots* - Introduction to the concept of real-time conversation co-pilots. - Overview of the challenges and potential benefits of using AI in conversations. - Discussion of past attempts and the need for low-latency solutions. 01:39 *🚀 Real-Time Co-pilots in Professional Settings* - Examples of real-time conversation co-pilots in professional contexts, such as aerospace engineering and job interviews. - Consideration of the value and ethical implications of using AI in interviews and professional settings. - Potential for improving interview processes and enhancing communication skills. 03:29 *🤝 Real-Time Co-pilots for Social Interactions* - Exploration of the benefits of real-time conversation co-pilots in social interactions. - Personal anecdotes about the challenges of social interactions and the potential support AI could provide. - Discussion of the broader applications beyond professional contexts. 04:24 *📱 Building Real-Time Conversation Co-pilots: Web and Mobile Apps* - Overview of building real-time conversation co-pilots, including web and mobile applications. - Introduction to the technical components required for such applications. - Step-by-step guide on using platforms like Replicate for deploying AI models. 04:53 *⚙️ Technical Challenges: Real-Time Transcript and Fast Inference* - Discussion of the technical challenges involved in achieving real-time transcription and fast inference. - Solutions for real-time transcription, including recurrent loops and optimizations for accuracy. - Strategies for achieving fast inference with large language models, such as model selection and optimization techniques. 10:19 *🛠️ Implementing Real-Time Conversation Co-pilots: Demo and Iterations* - Overview of the iterative process of building real-time conversation co-pilots. - Demonstration of a web application prototype and its functionality. - Integration of AWS services and Replicate for deploying and running AI models. 17:30 *🛠️ Backend service setup and frontend basic structure* - Setting up the backend service involves defining routes and handling requests. - The frontend structure includes defining HTML elements and basic CSS styling. - Functionality such as recording audio and fetching suggestions is outlined. 23:18 *📱 Exploring Whisper Kit for mobile deployment* - Whisper Kit, an open-source fine-tuned model, enables deploying speech-to-text models on mobile devices. - Real-time transcription with minimal latency is demonstrated on devices like the iPhone. - Whisper Kit's optimization allows for efficient use of resources on mobile hardware. 24:01 *🎁 Unboxing and setup of Apple Vision Pro* - The unboxing and setup process of Apple's Vision Pro headset is showcased. - Details about accessories, boot-up procedure, and initial impressions are provided. - The demonstration highlights optimizations for improved accuracy in AI-generated transcripts. 25:14 *💻 Development using Whisper Kit for iOS apps* - Utilizing Whisper Kit Swift package to develop iOS apps for real-time transcription. - Setting up the project in Xcode and configuring model selection and streaming options. - Overview of the app's structure and explanation of key functionalities for transcription and model loading. 31:03 *🚀 Integration and testing of conversation co-pilot features* - Defining variables and functions for user input, API interaction, and response handling. - Adding UI components for user prompts, transcription display, and interaction buttons. - Demonstrating the setup process and testing the app on an iPhone for real-time transcription and suggestion generation. Made with HARPA AI

[@asbjborg]

I need this for screening consultants. They are very smooth talkers, the minute you scratch their gloss they fall through, unless they actually know what they are talking about. Can't wait to see your app. Thanks for sharing!

[@mmarrotte101]

I've been thinking about this idea for the last 8 months - so pumped to give it a shot, thanks a ton for sharing!

[@bassamel-ashkar4005]

Are you trying to build a phone agent?

[@mmarrotte101]

@@bassamel-ashkar4005 phone is cool but just generally a conversational agent to utilize when having any conversation anywhere. It seems to be an extremely useful concept in so many ways.

[@RatherBeCancelledThanHandled]

Awesome Job. You really need to go commercial with your ideas .

[@matten_zero]

Every video you drop is a gem. Deepgram STT and Groq's LPUs make this possible.

[@brianhe2690]

There can be lots of good applications for this. Happy to explore and share. Keep the good work 👍

[@Jim-ey3ry]

Holy, whisper kit is insane, running the model on mobile device directly gonna be the future; Also thanks for sharing Replicate, didn't know it is free to use!

[@nftawes2787]

"If you’re new to Replicate, you can try us out for free, but eventually you’ll need to enter a credit card."

[@Scheevel67]

All your videos are amazing! One gotcha, if AWS give you an "access denied" error you may need to update S3 policy to add a "/*" onto the back of the resource ARN - the wildcard permits access to all bucket objects

[@haowu8448]

thanks bro it worked!

[@carterjames199]

Awesome video again Jason good stuff

[@lakergreat1]

Yes definitely interested in the app, please notify when done!

[@jpgallegoar]

This running on the new Groq arquitecture would be awesome

[@Royaltea_Citizen]

I look forward to you rolling out you app Jason! It would be amazing to run that locally on my phone!

[@the3rdworlder293]

tremendous work dude

[@NatGreenOnline]

This is amazing Jason. Just subscribed to your channel and am very interested to see the iPhone app you build and other AI projects that you're working on. I see a lot of amazing use cases for this like overcoming objections on sales calls, asking better questions on podcasts / interviews, etc.

[@kate-pt2ny]

Great production, thanks for sharing

[@surfkid1111]

Don’t have enough thumbs for that, great content.

[@enlaichu]

Thanks!

[@sandrofelder]

Yes would be highly interessted to see this app in the store!

[@jasonfinance]

I tried to build a similar interview co-pilot before too, but the latency made it not usable; Can't believe how far we went with those model performance past few month!

[@SahilP2648]

You do know that you can use OpenAI API to get sub 2-3 sec outputs right? The only thing not possible on GPT is a system prompt (I think, I have never needed to use OpenAI's API). But on my Mac with a capable 7b parameter like Mistral, or Mixtral model, the output is also within 5 secs (especially when loaded in GPU memory). I prefer local generation vs online since in local you can modify the system prompt and you can customize the output a lot more.

[@kenchang3456]

Excellent video and very timely for my interests. Thank you very much.

[@mackroscopik]

In the future, Neuralink will be wired directly to the brain activating the vocal chords so that the interviewer is mind blown on how you're answering the questions even though it appears you fell asleep during the interview.

[@luishiluy]

I would be super interested. Thanks for your magic!

[@webinnovationspartners9293]

Love your work. Great content. Yes, please let me know about the end product once you polish it please.

[@akellasoumya3432]

Excellent content

[@automatalearninglab]

Love your videos, you do an amazing job of packing high quality information into a 30 minutes ish video. 🎉 thanks a lot!

[@AIJasonZ]

thank you so much for your feedback!

[@nexuslux]

Thanks for sharing. Don’t sit on the translation potential for this as well ;)

[@VaibhavShewale]

so in real life f2f talk we have to hold mobile to have a convo with other?

[@Paktalkuncovered]

Could you make a detailed video on how to make this?

[@mikew2883]

Very cool! 👍

[@Silberschweifer]

oh, another step to local speaking AI Assistant like Cortana or Jarvis

[@marcus_AI_Advisor]

Definitely interested

[@augmentos]

Why use small model when you can use large model and Qroq?

[@mr.mikaeel6264]

Ok now i want to build an agent that can listen to videos, copy and build the apps. There is way too much cool AI stuff to try and i have other hobbies and a life too xD

[@senzz97]

This is amazing, thank you for great content. I wonder, I tried this (i'm a beginner with a newly found passion for learning python). I don't have the same amount response on the web app like you have, I get a 8-10 second delay both with the transcript and suggestion. How can I fix this?

[@flankz2950]

quick question, If I were to run this offline what would the token speed look like?

[@magic-4-ai]

When your app will be published in istore? Or maybe it is already?

[@build.aiagents]

Phenomenal

[@Qwerty-ff1cr]

Why can't I see this video from your channel on my laptop? Lol. Im on my phone now but is anyone able to see this video from the computer?

[@danielmacbride525]

hell yeah im interested in the app

[@Silberschweifer]

why no search or/and RAG func call? with thsi even the small fats model can become more knowledge

[@blackhat856]

Is it possible to have an AI copilot real time in game ,steamvr rec room ?

[@YipMilk]

It's not going to work if the interviewer is able to track your eye movements through AI which can tell you are reading from a script.

[@free_thinker4958]

Connect it then to a suitable glasses

[@peterparker7146]

Have you heard about eye tracking by nvidia

[@aldousd666]

This is a great tutorial and illustration of how to use services, but your bucket policy on Amazon needs to be locked to just your user so nobody can mess with your bucket and hijack it. It's one of the most common ways people get their data leaked.

[@nexuslux]

Imagine using this with Groq api inference speeds

[@messostuff6829]

exactly my thoughts.

[@brandonheaton6197]

For sure- two orders of magnitude faster inference is bringing us a whole new world and fast - by the end of march it will be evident

[@csepartha]

Kindly make a tutorial to fine tune an open source LLM model on many pdfs data. The fine tuned LLM must be able to answer the questions from the pdfs accurately.

[@saiaditya4397]

Can we use this model on ESP 32?

[@arixerchan3807]

this is what the deaf waiting for long time👍🏻

[@AIJasonZ]

true!

[@TaktAkira]

Is there something like this for the android?

[@NatGreenOnline]

Me: "I think I can do this. I'm going to give it a shot!" Tries executing this following Jason's steps. Gets error message at first step when installing replicate into VSC. "command not found" . Watches 3 videos to see if I can figure out why VSC is giving me this error. Still not working. Feels defeated and quits :(

[@marcc0183]

Can we do this but in Google meet or similar?

[@elskipvers]

Yes! I need this for zoom, teams and meet

[@RealLexable]

Horrorfying😮 Terminator has arrived i guess. Better to late than never.

[@JohnSteiger-ey9bi]

I don’t do the ‘KZbin’ other than to watch. I liked. Subscribed. And now I am kindly asking you how can I give you money? What you have here can help so many people. I eagerly await for this blessing to come to fruition.

[@AIJasonZ]

hah thanks bro!

[@tiberiumihairezus417]

What's the point of passing the interview when there is a probation period in which real tasks need to be accomplished. And if those tasks are still doable by LLMs, it is just a matter of time until that position will be completely automated.

[@chivesltd]

lol cheesing interview

[@abhijeetkumar1552]

seeing this and thinking google audio recorder transcript and gemma

[@alvintohw]

Please publish as an Android app too!

[@harisonfekadu]

👏👏

[@jessedbrown1980]

interested!

[@dawn_of_Artificial_Intellect]

Hi i am very interested in this development

[@Ho-Lee-Chit_Fu-Kin-Fast]

I will only do AI on my mobile if I can use it in Airplane mode.

[@AIJasonZ]

this model load locally so yes it works in airplane mode!

[@jaredlalal]

Ok so what if i want to use this instead to argue with ppl and win every debate always forever. I gotta grind them KZbin comment wins or something

[@teensounds]

what if interviewer ask to share the screen😅

[@NatGreenOnline]

If you get a teleprompter like the Elgato Prompter it acts as a 2nd monitor so the other person will never see it, plus you can be looking directly into the camera (while reading the info) at the same time so it looks super natural!

[@Generouslife153]

I’ll find anyone who is serious about building a ai call software

[@Mr.JOG-]

just make sure you throw in a "right" every 6 to 9 words and your interviewer will never know your full of shit and not reading.

[@cutthecheck]

I'm high

[@bloomflora1105]

hahaha so funny

[@laif9857]

30 sec. and i find so pathetic the use that some people give to the tools , faking an interview , if you suck at work pls , dont do an interview imgonna fired you after a month , why you are gonna lie for a month of pay , if you suck at one job maybe you can spend the improving your skills , but young people of this days really Suck so badly

[@ComicMasta]

Thanks!

[@AIJasonZ]

thank you sir!