Unreal Engine 5 - Ultimate Voice AI Tutorial - Masterclass from scratch

Because there has been excessive demand for a tutorial, there it is now.
Explaining how to use a microphone for communication with an NPC/AI and get an voice response using ChatGPT and Elevenlabs TTS from scratch!
Project Files downloadable for Patrons:
9sj9.short.gy/UE5AI
Plugins need to be installed for UE5.1
____________________________________
www.unrealengine.com/marketpl...
www.unrealengine.com/marketpl...
www.unrealengine.com/marketpl...
Websites:
openai.com/
API: platform.openai.com/docs/api-...
beta.elevenlabs.io/
API: docs.elevenlabs.io/api-refere...
Voice IDs:
Name Voice ID
Rachel 21m00Tcm4TlvDq8ikWAM
Domi AZnzlk1XvdvUeBnXmlld
Bella EXAVITQu4vr4xnSDxMaL
Antoni ErXwobaYiN019PkySvjV
Elli MF3mGyEYCl7XYWbV9V6O
Josh TxGEqnHWrfWFTfGW9XjX
Arnold VR6AewLTigWG4xSOukaG
Adam pNInz6obpgDQGcFmaJgB
Sam yoZ06aMxZJJ28mfd3POQ
Don't forget to like, comment, and subscribe for more exciting showcases.
/ discord
_______________________________________
Want to support me?
/ marvelmaster
or
www.paypal.me/marvelmasteryt
All donation money flows into better software, equipement, assest and cloud space.
_______________________________________
Time stamps:
0:00 Introduction
1:39 Requirements and limitations
4:51 Setting up the project
6:21 Speech recognition
16:07 ChatGPT communication
34:46 Fixing ChatGPT bugs
37:47 Make ChatGPT remember
44:36 Elevenlabs TTS
1:04:44 TTS Bug fixing
1:12:00 Outro

[@miliv_eth]

kind reminder for any security phreak who is bumping her head against the wall after an hour of debugging: please enable your mic access in win under Privacy Settings > Microphone : D. I almost fell down the chair when I figured. Many thanks for the tutorial btw, its truly amazing and that you published it open source. People like you are the reason the unreal community flourishes (*^ ‿ ⁢*)♡

[@daniellopez9582]

I highly recommend a mic that allows to be muted from the device itself.... I don¿t trust that microsoft respects any privacy settings

[@diluciousOK]

Thanks for creating this amazing tutorial! Subscribed, and I'm looking forward to see more of your creations :)

[@anthonymannwexford]

Excellent. Thank you. I've been looking for something like this for a while. Well done.

[@diadorchak9767]

For people wondering about the {Blank Audio} error I found mine was capturing the audio from this youtube tutorial. Go to the Start capture function that is copied an pasted at 8:28 in the video and set the Device Id from 0 to maybe 1 or 2 depending on your set up. For me 0 was my audio from my PC and 1 was my microphone. I hope that clears some confusion.

[@benblaumentalism6245]

Not here. I'm getting [Blank Audio] every time, no matter how I set up the Device ID. I know my mic signal is getting into Unreal because I made an input level meter and used a channel from the Audio Capture component. I thought maybe that was interfering somehow, so I set up a blank project just to test it. Still nothing. I'm using UE 5.3.2, in case anyone happens to have any insights into what's going on. The Marketplace says the plug-in works up to 5.4.

[@badashphilosophy9533]

Having the same issue. Did you manage to solve it?

[@sahurows]

Thank you for this tutorial, it's some amazing knowledge you are sharing.

[@Smartmoufdeity]

Genius! Pure genius!!

[@FluffyMrB]

Superb tutorial!

[@ganzukc2676]

Thanks for the tutorial!

[@shaheenal-asadi]

Very nice tutorial, thank you for creating this. The future of dynamic NPC interactivity is here today :)

[@MarvelMaster]

indeed

[@englishwithphil42]

This is just magic, thank you, time to finally install UE5 since there's no plugin for UE4 :)

[@harithsami843]

Great tutorial thanks heaps

[@user-nk1wk1gc3j]

Great tutorial, I want to understand how you set up the digital character to change from a casual animation to an intro animation

[@algorithman2129]

Nice nice :) Now we just need a AI like dolly locally and quite a bit smaller, fed with only game specific content. Hmmm, my todo-list is growing.

[@jang4765]

awsome

[@aurelianobuendia24]

Amazing work, how can i Connect animations to it?

[@aika9938]

Hi, this is amazing. You are a life saver. I'm facing a little problem, I've been trying to 'Get Response Content' but that node does not show, instead i'm seeing all others - 'Get Response Value' , 'Get Response Content as String', etc. What can I do and what could be wrong? I am using UE4.27

[@istealpixel7371]

Genau das hab ich gesucht^^

[@MarvelMaster]

👌

[@marijnfuhler7246]

Lovely tutorial, thank you for creating this. Just one question, because you make use of the VaRest plugin do you need a internet connection in order to use chat-gtp? And if so, how is the ai going to work on a build when the api is dependent on wifi connection?

[@MarvelMaster]

yes gpt requests are made online... as well as text to speech... I think computation would be too intense locally at the moment.

[@SK-hj1xh]

Hi. Can I use my RVC voice model in this pipeline? so that my character can speak with special voice

[@SuperNikpro]

very good tutorial. ijust have one problem my engine crashes when i hit # multiple times it crashes with an access violation at 0xfff... any ideas how to fix this?

[@wolfgangschwarz9415]

Thanks so much for the effort and clearly structured tutorial! Eventually I made it work for me :) I guess you are right and this is still basic with lots of things that can be tweaked. My biggest wish: How can I find an ID for generated voices to implement in UE? Btw, the German language model works quite well and I do not see any latency difference to English.

[@MarvelMaster]

different voice ids are in the description ir you create your own one if you habe an elevenlabs subscrption

[@user-ug6kk5ux5q]

I really love what you do, in my app I need to download the audio file and storage it at a specific file path. Do you know how could I do that ? Any small advice would be really helpful!! It is not clear to me what the runtime importer does with that file.. I need to turn it into an asset, because I have another API that requires the audio, but also the asset of the audio. Thank you!

[@MarvelMaster]

did you check the documentation of the audio importer... there is even a setup for non streaming

[@user-yv8dt9hi3h]

Thank you for the content!! It helped me a lot! Is it available on unreal engine 5.2??

[@MarvelMaster]

i think. the plugins are not available jet fir 5.2

[@diegobevilacqua4407]

Thank you! finally it worked, now i m trying to add a metahuman with Quixel bridge, i can add but i don t know how to start O_o using also the lipsynch like the "text to speech" function...

[@MarvelMaster]

jeah maybe thwre is some realtime plugin for that

[@joellegalloni6689]

Thanks for this! I am not so clear about what the streaming option is doing though. Isn't this supposed to output a stream that we can play through before the request is completed? Like we start reading the stream and continue updating the playback until the stream is over? The way it is now I still have to wait till the audio is completely through before playing so I don't see the streaming part of this?

[@MarvelMaster]

yes the way it is now is just wait till end... but in theory with streaming you can split inputs and outputs to make it faster... if one knows. how to do that

[@1RNik]

Thanks for the tutorial! For your patreon what tier would we join to gain access to the project files?

[@MarvelMaster]

as minimus supporter 😁

[@1RNik]

@@MarvelMaster Thank you!

[@estaling3802]

Hello, I followed the tutorial, and it only allows me to speak once. No error or string printed out after the second time voice recognition. Do you know why it's happening?

[@tuargames5357]

Is there anyway to setup a memory for the character like give him a backstory?

[@rotv6492]

Hello does anyone know how to fix the "the audio data could not be processed to the recognizer since the thread is stopped" problem with this plugin?

[@Hunty.]

any way to do this with a local LLM instead so that it doesn't have all the filters that chatgpt does?

[@user-ug6kk5ux5q]

Hi, great tutorial! In my case the imported soundwave didn't work, so in order to find out why, I added a "Switch on ETranscodingStatus" attached to the "OnResult" event that continues from "Status". Well.. this switch returns "Failed to read". Any idea why would this happen? I did correctly putting the switch there? (like, should I trust it? If yes, why would it fail to read the audio file? I did exactly what you did!). Thank you!

[@MarvelMaster]

maybe there is no audio file... did you printdebug the response?

[@CanCan-gy5hh]

Hi, thanks for the tutorial! I have a question. I created my voice in ElevenLabs but can not find voice ID. Where can I get it? Thanks for help.

[@MarvelMaster]

the elevenlabs website surely can tell that

[@taimur879]

Hi . can This be Used to generate Lip sync for non metahuman Characters

[@JesseThaBest]

Hey! So I've been looking into this Speech Recognition, is there any way to make it more efficient or faster? I'm noticing it takes minimum of 3 seconds for the Model to recognize what I say and return an answer, is there a way to make it "instant"? I already use the tiny model size and put english only, so can't reduce the sample size. :D

[@MarvelMaster]

hm maybe better hardware? And maybe there is a way to make it recognize to get streaming answer from chat gpt and elevenlabs... meaning not waiting till the whole sentences are processed but getting the responses while they are generated partly?

[@georgydev]

Hey, I just wanted to mention that since then, the plugin has undergone significant improvements, and its speed is now much faster compared to half a year ago

[@nBodyResearch]

Hey, in your waterline video it was for ue4 and needed world displacement and tesselation. Is there anyway I could show me how to do the waterline without those options in ue5? Thanks

[@MarvelMaster]

I read that you can use virtual heightfield mesh insteafmd of tessellation in ue5

[@nBodyResearch]

@@MarvelMaster I figured it out!! Though now I just need to figure out how to get swimming and buoyancy to work in a custom water mesh!

[@wolfmarkstudio]

So cool man can you plz tell me how to mzke the animation while talking

[@MarvelMaster]

just start animation when sound starts

[@wolfmarkstudio]

@@MarvelMaster ohh i was thinking your using npc meta plugin or sdk meta plugin thank you

[@ai_and_chill]

i'm having a lot of trouble with creating save game slots of the chat array history. would you be able to help with that please? this video is great thanks again.

[@MarvelMaster]

isnt it just a string to save?

[@ai_and_chill]

@@MarvelMaster i have it connected to twitch and twitch messages trigger this whole process to happen instead of a microphone. i'd like it so that when I load it only loads based on who just sent the message. so it would pull the array save game slot associated with that username. what's driving me mad is that I figured this out earlier this week and forgot to back it up. running in circles trying to remember what I did. lol

[@ai_and_chill]

@@MarvelMaster oh right i didn't explain the problem. it's saving all messages onto one array. even though I set the username variable as the save game slot name.

[@ai_and_chill]

@@MarvelMaster chatgpt and all them have been completely useless in getting this to work XD

[@MarvelMaster]

@@ai_and_chill you can try use something different than array

[@AICineVerseStudios]

great solid tutorial. seems like they have update their github page and how to use the plugin page is not there anymore. We have to create all the nodes by ourselves for the blueprint

[@MarvelMaster]

no the copyable codes is in the runtime speech recognizer doc not audio importer doc

[@AICineVerseStudios]

@@MarvelMaster Sorry my bad! you are right. Followed the eleven labs integration. All is going well but from the eleven labs whatever is returned, its not playing from the buffer. I am not using chatgpt response but actually just passing the text generated based on voice recognition to the custom event of playing voice reply. Any help here would be really appreciated.

[@MarvelMaster]

@@AICineVerseStudios you can debug print the resonse from elevenlabs... if its cryptic the itsa sound 8f its text that says some error then something is wrong

[@AICineVerseStudios]

@@MarvelMaster It's working! Eureka! 😀😀😀😀

[@AICineVerseStudios]

@@MarvelMaster Ok actually I'm building this to run on a cellphone as a person avatar. Is it recommended to follow your approach to building the same for Iphone or Android ? Can there be processing issues ? Or would it even be able to use the plugins for voice recognition ?

[@titan3rd474]

Great content, How to do animations when voice is playing

[@MarvelMaster]

you can get when audio is playing and the start an animation

[@titan3rd474]

How can I mix the idle animation and the talking animation, is there more tutorials coming up, because this has been a great help 😀

[@MarvelMaster]

@@titan3rd474 Animation is a topic itself...you can use an animation BP or just Trigger animations in the character BP

[@badashphilosophy9533]

When i press # in the blurprint search, keyboard elements isnt appearing as an option. Today is my first day in unreal engine so am asking here for everyone who has the same issue in future doesnt have to leave the video to debug. Thanx whoever provides the answer to this issue :)

[@badashphilosophy9533]

Found it, though it may be specific to the latest version of UE (5.4.1) You have to type "keyboard event" and then the key you want ie keyboard event T in order to get the precise key in the search. If you just type the word keyboard and then #, youl have to scroll up and locate it manually, or alternatively if you just type the word key and then #, youl have to scroll down and find it manually

[@wizzard_design]

Is it possible to make microphone work without clicking any button but when you start game? Thanks!

[@MarvelMaster]

you mean voice activated? Not sure if the plugin has a option for that or another plugin does that

[@badashphilosophy9533]

Did you find your answer, id like to do that too but a bit like how we say hey google to activate google assistant, but ive got a god game so u have to pray dear god lol.

[@CanCan-gy5hh]

Hi! Everything works fine in project. However, when I packaged project, only "VoiceRecognitionStarted" and "VoiceRecognitionStopped" are writing. So it does not work when I packaged. Do you have any suggestion?

[@MarvelMaster]

iirc plugins need to be packed manually in project settings

[@user-pf2se2df8v]

Great tutorial! For some reason, when it prints out what it processes from speech recognition (step 1 of the video), it always returns a series of exclamation points (!!!!!!!!). Any idea why?

[@MarvelMaster]

step 1 is voice recognition... make sure your mic works and is not set too loud or quiet

[@user-pf2se2df8v]

@@MarvelMaster right now I'm just using the mic on my Mac and am having issues - would this only work (well) with an external mic?

[@user-pf2se2df8v]

@@MarvelMaster bump on this -- is this a Mac issue? Would I need an external mic?

[@MarvelMaster]

@@user-pf2se2df8v try it on another pc then

[@georgydev]

Hey, that mentioned problem occurred some time ago due to a resampling issue on the engine side caused by audio chunks being too small for proper resampling. But it has been fixed since then, and now it shouldn't cause these problems anymore

[@vaibhavdevadiga9807]

where should I connect Make SpeechRecognitionParameters, in order make it work? 14:40

[@MarvelMaster]

there is probably a set parameters node or something

[@vaibhavdevadiga9807]

@@MarvelMaster can you let me know me know how to do it?

[@gscsdjoe7152]

i have one question, when i change language in setting, it wont work, if i change it back, the issue will be fixed, why, anyway, tks your video, its help me a lot. if u can rep me about how to change language, its will be great. and it will help a lot people just like u and me who are not the native english speaker.

[@MarvelMaster]

depends on what does not work... you can set voice recognition language in plugin and the also use the multilingual model of elevenlabs

[@rofethegeek5820]

how did you manage to animate her when she talks?

[@MarvelMaster]

just start a random animation when audio plays

[@dave0602]

Does it work for only 5.1? or include above?

[@MarvelMaster]

havent tried other depends on if the plugins are available for newer version

[@MemeIsAllreadyInUse]

how to change the mic input?

[@MarvelMaster]

dont know maybe in projext settings or windows settings... usually it takes the standart mic input from windows afaik

[@toras99]

What should I do if when I start recording sound, I get the following message: you

[@MarvelMaster]

new ue5 versions dont set mic source automatically, on the blueprint somewhere there is a node where you can set the audio source as number standart is 0 try 1 2 or e

[@toras99]

@@MarvelMaster I changed numbers from 1 to 7 in start capture. Did not help

[@MarvelMaster]

@@toras99 you can open the audio capture plugin level there you can see and select your audio devices folder should be somewhere hidden in plugin folder

[@bagusdai3684]

the audio wont play, this is the log "ogRuntimeAudioImporter: Warning: Imported sound wave ('CapturableSoundWave_1') data will be cleared because it is being unloaded LogRuntimeAudioImporter: Warning: Imported sound wave ('CapturableSoundWave_0') data will be cleared because it is being unloaded"

[@MarvelMaster]

did you try google it or look into the documentation... andengine version?

[@bagusdai3684]

@@MarvelMaster i use UE 5.2

[@eunachaanakya]

hello can you help for solving about "Audio_Blank"?

[@MarvelMaster]

never hewrd of that...

[@diadorchak9767]

I made a comment, it might help you

[@diegobevilacqua4407]

Im at 20%, everytime i speak it print me "you", any ideas?

[@MarvelMaster]

you have to hold the button

[@BR-hi6yt]

Good except the lips do not move when she speaks.

[@MarvelMaster]

jeah unfortunately did not find an easy solution for that

[@calvinwayne3017]

@@MarvelMaster id like to see u do it though, metahuman comes with the visemes, so text to visemes then set the visemes, would be the logic, but thats way over my head

[@Delv3r]

When I restart the project, the VArest json nodes get deleted and speech recognizer nodes break. Know what thats about? I could be dumb

[@Delv3r]

oh, I think it was because I created a cpp project instead of a blueprint project

[@MarvelMaster]

ok 😬

[@maxschweder704]

Awesome tutorial!