Creating Talking AI Characters in VR and Android | Convai Unreal Engine Tutorial

Learn how to add AI NPCs in Unreal Engine for Android and VR applications using the Convai plugin. This tutorial walks you through the installation of the plugin, creating interactive characters and integrating them, adding actions and scene perception, and finally creating the application for Android VR.
GDrive link for the ConvaiReadyPlayerMe and ConvaiOVRLipSync plugins:
drive.google.c...
Subscribe for upcoming demos and tutorials, and join our Discord channel. / discord
Sign up at convai.com to try it out.
Write to us at [e-mail] for any queries, bug reports, collab, or business-related engagement.
Follow us on your favorite platforms to stay in touch:
Discord: / discord
LinkedIn: / convaitech
Reddit: / convai
Twitter: / convaitech
Instagram: / conv.ai
Facebook: / convaiofficial
TikTok: / conv.ai
ArtStation: www.artstation...

[@amirorozobekov2490]

This plugin opens so many possibilities to create the ̶M̶A̶T̶R̶I̶X̶ metaverse real. You are the best guys!

[@bobhawkey3783]

I'll second the request for a Unity version for Quest 2. These recently posted videos are very much appreciated! ❤🎉

[@malorum]

I'll third the request for a Unity version! Most of the VR developers are using Unity not Unreal Engine. Would be super helpful!

[@convai]

@@malorum Hello, great to hear about the requests, I would like to understand more what would be good for all of you. Please set up some time in my calendar to discuss: calendly.com/nyla-convai or email us at [e-mail]

[@bauervision]

everyone has been pretty clear, we just want to see these same videos but using Unity@@convai

[@CCGS]

well this looks freakin dope, really awesome plugin!

[@ih4286]

This is great, I've got to try this out!

[@convai]

You should! Happy development! :)

[@onametaquest]

Thanks! Looking forward to see whats next!

[@convai]

Thank you! We'll make sure to make a lot more exciting content!

[@penguin-i9f]

When are you going to update some tutorials on UE5VR?

[@Hackarchy]

Do you have something like this but for Oculus in Unity? These videos are great! thank you.

[@convai]

Hello, for the time being we do not, but the base plugin is compatible with Oculus and you only need to change the character controller to make it work

[@bauervision]

Whenever I try to record my audio using Unity and the Oculus, the whole headset freezes and 95% of the time it doesn't record anything@@convai a verified project would be an excellent, simple thing to add to make everyone's life easier and keep people using this plugin. as it stands right now, I have emailed support ( 3 days ago with this issue and no one has gotten back to me. I am now considering other options)

[@eagar2333]

Hi, thank you for this tutorial! But there is a problem, once I change the Parent Class to Convai Base Player, the Snap Turn function will be disabled, then I can't turn my view with my joystick in VR. Could you please check this issue? Thank you very much.

[@eagar2333]

Oh! I found a solution. Just need to go to VRpawn details and change Use Controller Rotation Yaw to Use Controller Rotation Pitch.

[@dankus_t]

bro the tutorial is good, i am new in the comunity but is too easy understand❤

[@gavzbetzy]

Wow!!! Great video

[@convai]

Thank you!

[@enigmaticimagination01]

Would you please do a tutorial of the same topic for Unity

[@saiprasadpandilwar5935]

my character is not moving tried everything that you said in the video

[@ricardomontenegro4069]

you found a solution?

[@HowTuts]

Tried to package the project to exe but it did NOT launch I got this massage: Plugin 'ConvAI' failed to load because module 'Convai' could not be found. Please ensure the plugin is properly installed, otherwise consider disabling the plugin for this project. ANY FIX

[@convai]

Hello, sorry to hear about your issue. Please email us at [e-mail] for help

[@lucacorsi7175]

hi, I'm trying to do the same thing on unity, but it doesn't seem to work, how can I connect oculus quest 2 to convai on unity?

[@christosioannou9990]

Amazing! Does tutorial relate to an Oculus Quest HMD? How would this technology package onto such a headset and still work remotely? Or do you need to have the Oculus Link attached?

[@convai]

Absolutely, there's no need for an Oculus Link. You can simply package the project and install it directly onto your Oculus Quest HMD with no further steps required!

[@llupiy]

Hi, thanks for this video. Should we follow the same instructions with plugin v3.0.1? In my case (UE 5.2.1 on Windows 11), the chat widget doesn´t show in the viewport preview and I cannot trigger voice recording with the T key or text with Enter, pressing F10 also doesn´t work. Tried with controller input action wearing a Focuse 3, but also didn´t work. The character updates fine with the one from playground.

[@convai]

Hi @llupiy, yes the same instructions work for all plugin versions and UE versions, what you're explaining indicates that the player blueprint is not inheriting from the ConvaiBasePlayer, please double check that you followed the steps correctly. Additionally, we always encourage developers to understand the plugin on a lower level, so I would recommend going through this guide here which explains how to set the components without having to change the parent class: docs.convai.com/api-docs/plugins-and-integrations/unreal-engine/guides-v2-under-development/getting-started/simple-talking-cube

[@SethTodd-rl6hr]

@@convai Hello, you guys make a great product but I am running into a few problems similar to @llupiy. I am also using Vive Focus 3 in UE5.3. When I hit "play in VR" the scene shows up in the headset but the chatbox is not there. I have successfully used a button on the controller to activate talk function but its like the chatbotAI doesnt receive the message. I have watched the video over and over, along with many of your other videos and I am driving myself crazy with this, please help!

[@DjOmiBitOfficial]

Hi there, excelent tutorial and product. I am working with UE 5.3.2 but at the moment I copy the plugins for 5.3 but it doesnt work and I have to delte it manually becaduse the project doesn't launch. Any solution?? Thank you in advance

[@convai]

Hi @DjOmiBitOfficial, could you elaborate more on the issue, do you mean the ready player me plugins or Convai plugin? did you try installing directly from Marketplace? It would be easier for us to track and investigate the issue further via our Discord server: discord.gg/xbmCBb2vSh

[@haotinggao]

please tell me how to add the chatbox in the VR mode, I want the user can view the text.

[@convai]

Hello, please email us at [e-mail]

[@VuTruong-g6d]

HI, can you do it for AR also please?

[@korra50000]

Could this be used with a meta xr mixed reality template?

[@michaelb1099]

cant get the character to show dont have convai rpm character indrop down

[@convai]

Hello! Could you elaborate more on the issue you are facing? If it's something related to the RPM character on the Convai playground, you can write to [e-mail] and get a speedy assist on it. Also, we recommend you to join our Discord server, and our developers will be able to help you in resolving the challenge you are facing: discord.gg/Tm9b3FWFJG Hope this helps! :)

[@vuquangtruong5950]

Can you please have AR NPC in unreal ?

[@michaelroberson3971]

How can one implement this for Meta Humans instead of ReadyPlayerMe Avatars? Will a new tutorial come out?

[@convai]

We have this metahuman tutorial: kzbin.info/www/bejne/fnmtp4xvmdKqrck

[@simonperezgarcia]

Pleaseee a tutorial for Unity with the quest 2

[@convai]

Hello, we would love to hear more about your use case. Mind setting some time to chat on my calendar: calendly.com/nyla-convai

[@NeuroMatrixGame]

And Unity?

[@jaredredmer6229]

Mostly helpful! Only one problem im getting. My AI wont turn towards me. So if I say to follow me, he does, but when i run behind him, he runs backwards. seems his rotation is locked but nothing in constraints is checked

[@convai]

Hello, this sounds like the animations may be backwards or something is off with the physics.

[@convai]

Hi, we have made a very recent update to the Convai ReadyPlayerMe plugin which fixes this issue, you can find the updated plugin in the following GDrive link: drive.google.com/drive/u/2/folders/11jAzxkrjUj7FBMeM8w3ENii6KKH-gB5K

[@abhaychavan5614]

My UE 5.2 dosent start conversation with AI character after speaking t. Even after typing in textbox and hitting enter it doesn't reply (4:48)

[@convai]

Could you please check the output logs for any errors? It might be something straightforward like a missing API key or character ID. If you were not able to tell what the error is from the logs, then please share with us the output logs after interacting with the character over our support email [e-mail].

[@michaelb1099]

i dont have push to talk in vrpawn

[@convai]

Hello, We are sorry for the issue you are experiencing. If you wish, please join our Discord server, and our developers will be able to assist you in resolving the problem. discord.gg/Tm9b3FWFJG Thank you!

[@michaelb1099]

@@convai just wondering will you have any tutorials on npc2npc in unreal from start to finish as i am having a hard time for it to work in my museum enviroment and getting my custom characters to talk to each other like the sample you have. In the sample i got them using my knowledge bank so i know its working but when i use the custom characters and enviroment it doesnt.

[@trolatrolatrola]

After making the suggested adjustments to the VRPawn blueprint, the VR navigation is still broken. The left stick used to turn does not respond as it should.

[@convai]

Try to undo the changes one by one until they start working again, and then you will be able to debug which step caused the issue

[@trolatrolatrola]

@@convai Thanks for the response. The issue starts right after changing the VRPawn's Parent Class. None of the changes after change or fix not being able to turn with the left stick. I'm using Unreal 5.2 and a Oculus Quest Pro

[@mehmetenesgulcan2870]

If you close rotation yaw in vrpawn blueprint class default, it is work@@trolatrolatrola

[@danielpe6132]

@@trolatrolatrola hi there, I am having the same issue it stop the turn control, i have try in two different projects one is a default template and still happening, not a clue of what make it, did you realize what was happend? thanks in advanced

[@danielpe6132]

answering to my self, in case someone need it. the problem is that when we change to convai Base player, our "use controller rotation yaw" become true and it need to be False, just in the blueprint character click in vrpawn and in the Details section look for "use controller rotation yaw" and make it false

[@commanderskullySHepherdson]

It works but my metahuman keeps looking at the floor, occasionally she looks up

[@MrrPancake]

This didn{t work for me sadly as soon as I installed the plugins the memory no longer register my voice properly and they just repeat themself endlessly any tips or help with this ?

[@convai]

It seems as a microphone issue, you can test your microphone by pressing F10 while in play mode. Let us know how it goes!

[@MrrPancake]

@@convai I mean it was working fine with the mic, installed the pluggins and now I cant not even see what I say in the register of the memory which is a bit confusing. is just a blank space of dialog I said and then her answer, bit weird gonna try in a clean project.

[@MrrPancake]

@@convai did what you said but is not working idk what happened tbh, it was working and now all it says is "how can I help you" doesn{t matter what I say

[@MrrPancake]

Basically, when I Speak/Talk it doesnt transcribe into text as it should she just blindly answer the same over and over. @@convai

[@convai]

@@MrrPancake To troubleshoot your audio recording issues, first press F10 to access the settings menu, then click on "Test Record" and speak into the microphone. Once you've finished speaking, press "Stop" and listen to the playback to see if your audio has been successfully recorded. If you can't hear your audio, go back to the F10 settings menu and select a different microphone before repeating the test. Should the issue persist, it may indicate a problem with your microphone hardware; double-check to ensure it is functioning correctly. For further assistance, you can send your output logs to our support team at [e-mail].

[@EarthlyF0rg0ten]

Anyway of doing an open mic?

[@convai]

Hello, do you mean having it so that it can always pick up a voice? Not having to activate it with the keyboard?

[@EarthlyF0rg0ten]

@@convai exactly, I was looking around but no avail.

[@convai]

​@@EarthlyF0rg0ten There is a way, but it is still in beta with not much guarantee that it will work well. At 2:22, instead of changing the player class to ConvaiBasePlayer change it to ConvaiPlayerWithVoiceActivation.

[@EarthlyF0rg0ten]

@convai thank you, I see it's kinda buggy if I found out anything I'll report back.

[@tank_antivirus]

How much does it cost

[@convai]

Hello, feel free to check the pricing page from here: convai.com/pricing

[@tank_antivirus]

Do you have a limit of 100 questions per day on your website? Do I also have the same restrictions in my game?@@convai

[@cugilbert3799]

Promo`SM

[@hasstv9393]

UE5 in visual studio 2022 when i build my game, it's showing Severity Code Description Project File Line Suppression State Details Error C1083 Cannot open include file: 'ConvaiDefinitions.generated.h': No such file or directory Ai_NPC D:\UE5Main\UnrealEngine-5.1_Source\Engine\Plugins\Marketplace\Convai\Source\Convai\Public\ConvaiDefinitions.h 7

[@convai]

Hello, please follow our guide here over github for building the plugin from source github.com/Conv-AI/Convai-UnrealEngine-SDK?tab=readme-ov-file#installation

[@hasstv9393]

@@convai Thank you

[@convai]

You're welcome! @@hasstv9393