Integrate AI Characters with MetaHuman Avatars in Unreal Engine 5 | Convai Unreal Engine Tutorial

In this tutorial, we’ll show you how you can use Convai to add AI functionalities to Metahuman Characters in Unreal Engine 5. Convai enables these Metahuman AI NPCs to perceive their surroundings, perform complex actions and even talk with you or other AI NPCs within your games and virtual worlds. Create amazing virtual experiences in UE5 with Convai today!
Sign up at convai.com to try it out.
Also, learn more about Convai's Safety Guardrails for AI Characters in our latest blog: convai.com/blog/safety-guardr...
Write to us at [e-mail] for any queries, bug reports, collab, or business-related engagement.
Convai plugin for Unreal Engine: www.unrealengine.com/marketpl...
Reallusion Guide: • AI NPCs with Reallusio...
Ready Player Me Guide: • Transform Ready Player...
Custom Character Integration: • AI Characters with Cus...
Time-Stamps:
00:00 Intro
00:45 Setting Up the Convai Plugin
01:25 Adding Metahuman Character to Scene
02:05 Adding Animations
03:18 Applying Lip Sync
05:30 Creating Characters AI Personality
08:13 Enabling Actions
09:16 Scene Perception
10:57 Changing the Character
13:52 Outro
Follow us on your favorite platforms to stay in touch:
Discord: / discord
LinkedIn: / convaitech
Reddit: / convai
Twitter: / convaitech
Instagram: / conv.ai
Facebook: / convaiofficial
TikTok: / conv.ai
ArtStation: www.artstation.com/convai

[@uno27heaven]

Was legit JUST TRYING TO DO THIS TY

[@convai]

We are glad you found this helpful! Happy development :)

[@robot_collective]

Thank you so much for sharing this tutorial and for your work on Convai! I am so excited to try this out in the next few days :)

[@user-yh3rk6no2s]

I hope you guys come out with more tutorials on integrating into VR templates!

[@convai]

Hey! Thanks for your comment. We have a tutorial out for setting up Convai into Unity VR projects! You can check it out here: kzbin.info/www/bejne/l4Orm2mIjct4qNk

[@robot_collective]

So awesome, the body and face animations work great on my Metahuman in UE 5.1. and also the conversation works like a charm. You got a new fan here :)

[@Augmented_AI]

Please add in the ability for gpt 4o vision. So it can see dynamically what's happening in the scene

[@GES1985]

I wish there was a way to "teach" them actions while in game. I.e. tell them to go into a state where they observe the player and record things like player position, direction player is facing, actions taken, etc. Dynamically adding things like "axe" and their respective descriptions, and giving the NPC the ability to go do tasks that you can do with your own character, to save time grinding resources, etc.

[@RizwanYe]

Thank you Convai team and the hard work you do! Please make more of these tutorials! Ill watch all of them!

[@convai]

Hey! We are glad you like the tutorials. Happy development! :)

[@mattbeets]

As a total n00b in this, I thought I was going crazy because my ConvaiBaseCharacter's edit blueprint looked nothing like what's shown on the screen @3:04. But somehow I missed that I had to click on "event graph" (Graphs section on (my) left part of the screen) under graphs inside the edit screen. Just if anyone else was confused like I was :)

[@mattbeets]

Maybe also worth mentioning that using ElevenLabs voices currently requires first a subscription of 6 or 9$/month for Convai, and also a 99$/month subscription to ElevenLabs

[@NostromoVA]

Amazing!

[@schmutz06]

nice work! i spent a while playing with this earlier with metahuman, in Unreal 5.4.2 (latest stable). I couldn't get the face sync to work unfortunately. I am certain the correct anim class is set to the face, and I have the ConvaiFaceSync component in the stack (tried it in all hierarchies; under ConvaiChatbot, under Root, under face). I was unable to get the navmesh / follow action to work either. The character responded but did nothing. It would turn and face me half the time, and otherwise only look forward. After around 1 hour of starting out, I hit the usage limit. Is it possible to use Ollama / Llama3 local with this to avoid the limit by running all the generative parts locally?

[@SethTodd-rl6hr]

I am also trying to get a ConvAI metahuman operable, but in 5.3 . Mine also agrees to do some action, like move, but then never does it. Object specified in the details panel so not sure what the issue is.

[@LivioAscione]

As soon as I run the convai game mode and try to send a message (vocal or text) ue5 crashes on Mac OS. Maybe a problem with mic permissions?

[@xgtwb6473]

Can i run this all on a server using python and render to video?

[@SethTodd-rl6hr]

I have been trying to get a working MetaHuman with ConvAi in UE 5.3, but with VR. Firstly, I notice that the text box does not appear in the view inside VR (Vive Focus 3). The convai chatbot can respond to input, but the input doesn't show up. Any idea why this might be?

[@zimingpeng8418]

So exiciting! this is a very brilliant tutorial and plugin! I was wandering if this funtion could be packed like a Dektop-program or game version that user can play it with interactive hardware?

[@convai]

Hi @zimingpeng8418 this is definitely possible, simply package the project as normal and you get a playable app.

[@zimingpeng8418]

@@convai cool!!! Thanks! i can´t wait to try it.

[@aw92-v1]

Hello - great video! Previously, I remember there was a built-in mechanism where the actual SB_Main chat appears only as you're nearing or facing the ConvaiBaseCharacter we added in the scene, then it disappears when you face away. How can I get that feature back, please? I do not wish to continue seeing the dialogue box as I move around and interact with the rest of the 3D scene, it clashes with my existing UI widgets. Can you please help me here or guide me to a video or a thread if this has already been addressed in the past? Much appareciated.

[@convai]

Hello, We are sorry for the issue you are experiencing. If you wish, please join our Discord server, and our developers will be able to assist you in resolving the problem. discord.gg/Tm9b3FWFJG Thank you!

[@michaelb1099]

is there a way to fix lipsync as it is really off? i am using a reallusion character and not sure how to fix in convai

[@convai]

Hello, We are sorry for the issue you are experiencing. If you wish, please join our Discord server, and our developers will be able to assist you in resolving the problem. discord.gg/Tm9b3FWFJG Thank you!

[@NetZeroEarth]

🔥

[@frunobontes]

Are the actions not functional on 5.4? I cannot make it move or follow me.

[@convai]

Hey! Thanks for your comment. Actions are definitely functional on UE 5.4. You can check out our detailed tutorial on AI NPC actions to learn more: kzbin.info/www/bejne/ppmknHh8lpuHnsU

[@Espinosa_render]

why my metahuman when talks just has an idle pose? he doesn't use the talking animations...why? Thank you for the plugin, is really amazing

[@convai]

Hi Esinosa_render, thank you for your kind words, the talking animation is a bug that we solved in the latest 3.2.1 version of the plugin, it is not yet out on the marketplace but will be out along with other features and fixes.

[@zhengdonghou178]

In today's test, I found that ConvaiFaceSync in metaHuman doesn't work, and as a beginner I don't know why this is?Maybe someone knows?

[@convai]

Hi @zhengdonghou178, if the ConvaiFaceSync component is not correctly causing lipsync to work, then double check that you have added the correct face animation blueprint over the face skeleton in the character.

[@twjohnsontsai]

Hello, after setting up the Metahuman character, in the game, his eyes always wander and cannot focus on the other person. How can I fix this?

[@convai]

Hello! We have a tutorial dropping on customizing metahuman animation blueprint dropping very soon, which dives deeper into the character interaction logic, eg: Character looking at user while talking, character looking away while being in an idle state, etc. Will share it here, and it should be able to help you out! Meanwhile, we recommend you to join our Discord server, and our developers will be able to assist you in resolving any challenges you are facing: discord.gg/Tm9b3FWFJG Happy development! :)

[@mikebert]

is there a way to let the conversation box disappear while talking?

[@convai]

Yes! the conversation UI supports a wide variety of customizable options, and it's even possible to hide it entirely while talking.

[@NotFedBait]

@@convai i cant find this option, do you have a quick guide? please and thank you.

[@damingy]

Looks really cool, but I'd imagine just a few players actually utilizing this within a game could easily hit the monthly interaction limit.

[@ErikMalkavian]

Is this in Real-Time?

[@convai]

Hello @ErikMalkavian, Yes!

[@DIVERGEsound]

lol the second comment at the end

[@devasCabral]

When will plugin compatibility for Unreal Engine 5.4 be available?

[@convai]

Hello @devasCabral, Our plugin is compatible with Unreal Engine 5.4. You can download it from our Unreal Engine Marketplace page or access it via this link. www.unrealengine.com/marketplace/en-US/product/convai

[@MrBastos2010]

Is it possible to use another language besides English?

[@convai]

You can select a language for your AI NPC from a wide variety of options that are available on the Convai Playground. Happy development!

[@chris09423]

I wish you dont need a metahuman for this, Is there no way to use your own character?

[@convai]

You can definitely use your custom characters! Check out our tutorial on integrating Convai with Custom Characters: kzbin.info/www/bejne/bGrYlqSjiribp5Y&t

[@SanjeevKumar-be4dd]

i'm not receving any response from metahuman after asking questions

[@convai]

Hi @SanjeevKumad-be4dd, is there any errors that pop up in the chat widget? you can share this issue on our Discord channel: discord.gg/uTfQqzUnhf

[@SanjeevKumar-be4dd]

@@convai actually i did receive response from face but hand animation not working tried so many times

[@mrbosstv7083]

You guys hiring? Im a game dev, tech art, with a background in generative ai with python

[@angryyoungman4389]

Wanna join me for AI Project in my Startup, I can assure you it's something great! Please connect.

[@itzDrizzyyyman]

hmmm i still dont think its it!!!! you guys went the wrong way about it actually.

[@renanartist3d216]

why?

[@itzDrizzyyyman]

@@renanartist3d216 you can easilly do the same with a chat gpt subscription!