Tutorial: Unreal Engine ChatGPT with Metahuman SDK

You can now use the GPT artificial neural network, which belongs to the same family as ChatGPT, to communicate with MetaHuman SDK!
Chat GPT - a tool that uses artificial intelligence to create a more natural and productive dialogue between the user and the system.
In this tutorial you can see how to use chat GPT with our plugin.
Here you can find the blueprint from this tutorial:
/ discord
And by the way this video is also written using GPT chat :)
Link to our discord: / discord
Link to our website: metahumansdk.io/
Get the plugin for free from Unreal Engine marketplace: www.unrealengine.com/marketpl...
#unrealengine #metahuman #MetaHumanSDK #chatgpt #openai

[@agentp9900]

How did you make the metahuman act like a human or how to animate the body gestures? Thanks!

[@metahumansdk]

Hi! In this scene we use prepared animation asset directly to the body.

[@estaling3802]

Awesome video. I finally got the metahuman SDK plugin and openai plugin installed. How can I access the blueprint? The Discord link doesn't work, and the example level is also not showing up.

[@metahumansdk]

Hi! That level was created in UE 5.0. You can try to repeat blueprint from the video because file from the discord is the same blueprint.

[@petertsai2801]

hello ,may i ask you ,After converting text to speech, do you send the generated voice file to the MetaHuman SDK to generate LipSync Animation, or do you directly send the voice data from memory to the MetaHuman SDK to generate LipSync Animation?

[@metahumansdk]

Hi! It depends on the scenario. If you use TTS+ATL you just send text then receive wav file and send it back to generate an animation. If you are using combo request you send text then you receive wav and animation in the same time. Combo requests should work faster then TTS+ATL just because you skip part where you need to send received audio. You also can use Talk Component as Combo request kzbin.info/www/bejne/oKPTcn16fs12fKc

[@mumuhmuhaemin997]

Awesome. Is this tutorial using Unreal Engine version 5?

[@metahumansdk]

Used Unreal Engine 5.0.3 and OpenAI pligin from february 4 commit 9ea09256e4dc82a8f62846d100f0daef5598ae98

[@hangxu7281]

Thanks for making this fabulous feature, wondering can it link to private owned/hosted large language model rather than chatGPT?

[@metahumansdk]

Hi! It depends on OpenAI dev team, we use their plugin and connect text answers to TTS node then use our regular pipeline with ATL nodes. So you can connect any text inputs from any sources to our plugin.

[@vaibhavdevadiga9807]

Hii buddy. Did you try linking privately owned language model rather then GPT? did it work?

[@metahumansdk]

Hi! You can use any solution and send to our TTS node text or to our ATL nodes sound wave 16-bit PCM. So any chat-bot should work if it can send correct data to the TTS or ATL nodes.

[@benshen9600]

Excuse me, can you make a tutorial that supports GPT4? In addition, after the blueprint is built, is there a part of the video tutorial in the DEMO missing?

[@metahumansdk]

Hi! We create this tutorial as example of external chatbot integration for your project to understand how our plugin get information from it. You can get this scene in our discord and update it with any other plugin for Open AI technologies or other chat-bots that you want to use in your project.

[@user-ql2uj3to8y]

Can we use GPT3.5 or 4? Also, when I have conversations, the transcription of the conversation is always delayed by one turn.

[@metahumansdk]

Hi! Can you share little more details about delays issue to the [e-mail] or our discord server?

[@SuperAbdulJabbar]

Hi, you have not mentioned in this video that how you have created animation of movement of metahuman person in last, and how its lipsync is integrated with the chatgpt sound output. Can you kindly provide that tutorial video. I have tried different animation tutorials but they didn't help me. Since I have followed the exact procedure of this video, maybe yours way of animation in this video will help me. Thanks.

[@metahumansdk]

Hi! We have tutorial for our plugin where we show some cases of animation generation: kzbin.info/www/bejne/rqCXaGetbZJnl8U&ab

[@SuperAbdulJabbar]

@@metahumansdk the animations shown in the link provided are on stationary head which I have already done by following this video. I want to integrate that metahuman lipsync animation to the already made animation which is running some body and face movement as shown in the end of the video i.e. Cooper speaking while moving his neck or body at the same time, something that looks real.

[@metahumansdk]

To make neck movements you can use body skel, because root bone shared for body and head skeletons. So you can use blend per bone node to add neck movements to the body animation. Here is tutorial that shows how we do it kzbin.info/www/bejne/pYrCkIKQdsZjf5Y

[@rezafeng7301]

Hi，I wonder how to create the animation of the movement? where can I get this sample? Thank.

[@metahumansdk]

Hi! You can generate face animation with MetahumanSDK plugin, here is tutorial with examples of use kzbin.info/www/bejne/rqCXaGetbZJnl8U

[@diegobevilacqua4407]

someone can help to find the metahuman demo to install? is not on the project of metahuman (ada and taro) and the meta with quixel have different blueprint, on mac

[@metahumansdk]

Hi! You can download common metahumans in UE 5.0 or higher. Just open Windows>Quixel Bridge and find MetaHumans tab. If you need more details please contact us in the discord server discord.com/invite/MJmAaqtdN8

[@justinjin5675]

I gave your method a shot, but when the metahuman start talking, the head and body seem to disconnect. Also checked out your other tutorial on how to address this detachment issue, but I'm still a bit unclear on the solution. Especially on cached body pose.

[@metahumansdk]

Hi! This tutorial is too hard for this fix and we understand it now. I try to five easier way that works for us. Just add to the Face_AnimBP node Slot Default Slot between Modify Curve and Blend Per Bone nodes. In our discord you can find some screenshots with this example cdn.discordapp.com/attachments/1155594088020705410/1155844760708337704/image.png?ex=653fa2d1&is=652d2dd1&hm=6bf8056b06704dbdc1f3762d0f84916ed27bf4d6c277ac0b7fe280444285ffbf& Message link discord.com/channels/1010548957258186792/1155594088020705410/1155844761056460800

[@user-cf7ti5mc5j]

Hi, I really like this project. I've tried it 4 times with completed step by step but it never worked. the problem exists when I finish typing in the chat column and press Send, then "Generating response" appears in the top left screen and then nothing happens..(so sad), can you tell where the error is? Thank You

[@murfmusik3407]

Same!!!

[@murfmusik3407]

ok the thing is dont do it this way, just use the method af attaching metahuman face to the beggining of the blueprint where it asks you to attach metahuman face. Then add the open AI section as shown. So they have just simplified it!

[@metahumansdk]

Please share your log file in your discord because in unreal we have too many options that can break the pipeline. Also it will be great if you share a blueprint.

[@stephenrose1902]

when I tried to run the project things went sideways - I got the same first question about the rebuild, but then it came back with "can't compile try rebuilding from the source" I have no idea what that means or why I got that message.

[@metahumansdk]

Hi! For rebuild you need installed VS 2019 or higher with recommended parameters for Unreal Engine. Visual Studio visualstudio.microsoft.com/vs/older-downloads/ Unreal documentation about VS docs.unrealengine.com/5.2/en-US/setting-up-visual-studio-development-environment-for-cplusplus-projects-in-unreal-engine/

[@HEFTY-LNDN]

Will this work on 5.1? I got a new computer but I can't go back and install UE 5.0

[@metahumansdk]

We have created an example of level on the UE 5.0. You can use our plugin on 5.1 and about OpenAI plugin you can ask to OpenAI plugin team. OpenAI plugin used in this video is from commit 9ea09256e4dc82a8f62846d100f0daef5598ae98 on github.

[@ai_and_chill]

could you do a video on GPT4 integration please? thanks!!

[@metahumansdk]

@simplex5 We create this tutorial as example of external chatbot integration for your project to understand how our plugin get information from it. You can get this scene in our discord and update it with any other plugin for Open AI technologies or other chat-bots that you want to use in your project.

[@user-in7xc8vx3v]

how this easily to connect the Dialogflow chatbot and several voice AI like Azure, GG wavenet?

[@metahumansdk]

@user-in7xc8vx3v you can use our tutorial kzbin.info/www/bejne/rqCXaGetbZJnl8U or open demo maps in the plugin files like 05_Combo to make your own chatbot

[@user-in7xc8vx3v]

@@metahumansdk thanks

[@boyce-wei]

Hi, at 0:36 of the video, where is this WORK folder from

[@metahumansdk]

Hi! Work is root folder of some projects that in work now. So this is just a folder and its name is not important.

[@julienrenaud8897]

Hi! allz good but the AI respond only: , or respond nothing but it say ''success and he make face anim. Did you know why? thx

[@metahumansdk]

@julienrenaud8897 we will try to help but we need more details about your issue, can you share log file to [e-mail] or our discord?

[@kennedykimeu4618]

This tutorial needs to be updated.

[@ironform8263]

Is it correct that it is currently in service? I think the answer to the problem is too slow. I've even processed the payment, but I don't think the token is activated. Please take quick action.

[@metahumansdk]

Hi! Plaese contact us through mail [e-mail] or in our discord discord.gg/MJmAaqtdN8 we will try to help you with your issue.

[@4.22sy]

why my project doenst open.. my project is version 5.1 is that any version issue?

[@metahumansdk]

@4.22sy Hi! If you try to run our project, please try to do it on 5.0

[@charleneteets8227]

This video does not work for 5.1. Would love to see a updated video soon! (After you add the zip to the plugins folder it refuses to open).

[@metahumansdk]

Hi! This tutorial created on the engine version 5.0. The main point is connect text output from any source to TTS node or send 16-bit PCM wave sound to ATL node so you can use any chatbot you like if you can send correct data to nodes on MetahumanSDK.

[@diegobevilacqua4407]

Hi, i don t have the OpenAI call chatgpt3, i installed but it give me other options but not gpt3 , any tips?

[@metahumansdk]

We used OpenAI pligin from february 4 commit 9ea09256e4dc82a8f62846d100f0daef5598ae98 Perhaps now OpenAI plugin dev team change something in the newest versions. This tutorial shows that you can use any chatbot and send text to our TTS node or send 16-bit wave PCM sound to our ATL node and receive animation for your own chat solution.

[@diegobevilacqua4407]

@@metahumansdk sorry i don t understand, i cant find videos or courses where to learn, tutorials are incomplete, materials appear from nowhere, i don t know how and where to learn

[@metahumansdk]

@diegobevilacqua4407 here you can get the level from this tutorial: discord.com/channels/1010548957258186792/1010557901036851240/1089885618164744202 We try to show a lot about UE and give best to our users. We can help and explain detrails about our plugin functions. We can't get responsibility about products from the other developers but we beleive that you can get more advice for the OpenAI plugin from developers of the OpenaAI plugin.

[@andreiviievskyi2838]

The discord links doesnt seem to go anywhere! - even from the comments

[@metahumansdk]

Hi! We tested this link from few countries and this one works fine discord.gg/MJmAaqtdN8 Also you can try to use this new one discord.gg/csPFAsFBb8

[@user-oy8np3ip1w]

Hello, How can we integrate with Whisper speech to text ?

[@metahumansdk]

You can find instructions on how to use Whisper STT on the official OpenAI website. It would be great if you could share your final project with us.

[@solnone]

@@metahumansdk kzbin.info/www/bejne/bWW6aWeDoN-kabc

[@CanCan-gy5hh]

Hi, where can i reach this project file? Could you share link or can we reach on discord channel?

[@metahumansdk]

Hi! Join oour discord, you can get the file for OpenAI here: discord.com/channels/1010548957258186792/1010557901036851240/1089885618164744202

[@jackpops]

@@metahumansdk god its just the umap file has nothing in it

[@denizalbayrak9844]

Hey I love this project but Discord link is not working could you please update it ?

[@metahumansdk]

Hi! Thanks a lot! Yes, sure. Please, try this link: discord.com/invite/kubCAZh37D?_gl=1*ctxiy9*_ga*MTMyODEzMDY0NC4xNjc4MDMxMDAy*_ga_L1M1F0N7HH*MTY4NDI0NDI1NC4xNy4xLjE2ODQyNDQ4OTcuMC4wLjA.

[@user-cf7ti5mc5j]

I have completed step by step, when the play button is pressed, the message "Set Metahuman face in the level blueprint" appears, please help

[@metahumansdk]

If you using an metahuman with demo scene from our SDK you need to attach face of your metahuman to the nodes in the level blueprint

[@arifalbastomi2303]

@@metahumansdk how to attach?

[@metahumansdk]

Download MetaHuman, add it to your level and add it to the blueprint from the Outliner of your level. You can find more informatiion about it in our tutorial about MetahumanSDK plugin kzbin.info/www/bejne/rqCXaGetbZJnl8U&lc=UgwG3kT3WdO

[@xXArtCharXx]

This method of getting the demo file no longer seems to work. Please update the video when you can. Thanks!

[@metahumansdk]

Hi! The main point is connect text output from any source to TTS node or send 16-bit PCM wave sound to ATL node so you can use any chatbot you like if you can send correct data to nodes on MetahumanSDK.

[@RaedAbbas]

hi, the level blueprint doesn't look anything like yours. I'm on UE5.2. Can you kindly update your video?

[@metahumansdk]

Hi! We complete an example project for UE 5.2 with general cases include ChatGPT and you can download it in our discord server discord.com/channels/1010548957258186792/1068067265506967553/1143934803197034637 All blueprints in folder with ChatGPT so you can see how we use it. This project include MetahumanSDK and OpenAI plugins so it require Visual Studio 2019 to rebuild them and you need to turn off same plugins from the marketplace or project will not work correctly. Hope this one will help you much more then this video 😉

[@Nova-tf3es]

@@metahumansdk how to rebuild plugins ? any video or tutorial ?

[@metahumansdk]

To rebuild custom plugins you need to install VS 2019 with required parameters from Unreal Engine documentation and accept request for rebuild plugins when you run the project docs.unrealengine.com/5.2/en-US/setting-up-visual-studio-development-environment-for-cplusplus-projects-in-unreal-engine/

[@Nova-tf3es]

@@metahumansdk Thanks, it worked finally!! looks like you have to rebuild the plugins manually

[@bahtitashpaev7285]

after putting souce from github to the file Pluguns, project is not opening((.. Error is: could not be compiled can you help please?

[@metahumansdk]

Hi! Perhaps you need to install VS2019 or higher because plugins that you put directly in the project should be compiled at the first project run. Please use recommended parameters from the UE devs when setting up Visual Studio dev.epicgames.com/documentation/en-us/unreal-engine/setting-up-visual-studio-development-environment-for-cplusplus-projects-in-unreal-engine?application_version=5.2

[@bahtitashpaev7285]

@@metahumansdk thanks for the quick response. I will try it

[@meme-tn3vz]

Help i cant log in to metahumansdk personal acount, i want to get the API token.

[@metahumansdk]

Hi! If you have any troubles with space.metahumansdk.io/ please contact us in Discord discord.gg/MJmAaqtdN8 or e-mail [e-mail]

[@arielshpitzer]

based on a third party plugin that still does not support 5.2, and we are already moving to 5.3.

[@metahumansdk]

Hi! We create this tutorial as example of external chatbot integration for your project to understand how our plugin get information from it. You can get this scene in our discord and update it with any other plugin for Open AI technologies or other chat-bots that you want to use in your project.

[@user-cf7ti5mc5j]

I have almost finished this project, but when tested, an error appears like the following: LogMetahumanSDKRequestsManager: Error: Request response code: 400 LogMetahumanSDKRequestsManager: Error: Request error: {"error":{"status":400,"source":"azure-tts","title":"TTS Error","detail":"Connection was closed by the remote host . Error code: 1007. Error details: Starting September 1st, 2021 standard voices will no longer be supported for new users. Please use n USP state: 3. Received audio size: 0 bytes."}} LogMetahumanSDKAPIManager: Error: Combo request error: {"error":{"status":400,"source":"azure-tts","title":"TTS Error","detail":"Connection was closed by the remote host. Error code: 1007. Error details: Starting September 1st, 2021 standard voices will no longer be supported for new users. Please use n USP state: 3. Received audio size: 0 bytes."}} Please Help.

[@metahumansdk]

Hi! We apologize for this issue, it fixed now.

[@northernspirit]

All good until “Could not be compiled. Try rebuilding from source manually.” Any ideas?

[@babasahebpinjar6290]

Even I am stuck here, any solution for this ?

[@metahumansdk]

Please make sure that you have VS 2019 installed with recommended settings docs.unrealengine.com/5.0/en-US/setting-up-visual-studio-development-environment-for-cplusplus-projects-in-unreal-engine/ VS needed for project rebuilds. Also in this video we used UE 5.0 our MetahumanSDK plugin and third party OpenAI plugin from 4th frbruary commit 9ea09256e4dc82a8f62846d100f0daef5598ae98 on github

[@messengercreator]

and I'll challenge u AI CHAT DEEPAI and GPT only create whole game in unreal

[@BR-hi6yt]

Not good enough compared to others doing this type of stuff. Doesn't look real.

[@jxfala]

which ones for example?