NVIDIA Omniverse Audio to Gesture to Unreal Engine 5 Metahuman

[@TheNinonitro]

Keep up the good work man, you're a credit to the community.

[@milagro2300]

Thanks for your time J.

[@Alter3go]

damn... 2022 and industries still need to figure it out how to do this things easier and compatible. We are lucky there's you explaining how to do this hell

[@Instant_Nerf]

Nice man. Became a member. You deserve it. I’m curious why this interest you, when you have the Mocap suit? I guess for pcs walking and talking.. if that’s the kind of scene you need? Wanted to point out that when you were importing the skeleton the rotation tab was hiding because you were right in front of it at @5:11

[@Jsfilmz]

hahaha yea for some reason it didnt render the text overlay i put

[@OMEGATROPOLIS]

Keeo it up JS

[@UnrealDojo]

i cant find this in the membership tab_ the ik rig

[@윤씨-m9q]

I love your video! Thanks

[@Jsfilmz]

Thanks for watching!

[@ClydiieTwo]

" IK Retargeter unable to initialize. Missing target Skeletal Mesh asset. " i keep getting that message when I select the metahuman rig 8:33

[@karthikuk09]

can u suggest a tutorial in which i can upload an audio file from my language,lip sync to the meta human with body gestures and render it out, somwthing which i can do it in did,but with body gestures

[@FAITHBASEDGAMES]

WOW! That is way cool! We really need this kind of technology implemented into Unreal Engine 5, just think of how much quicker we would be able to make great cinematic movies! 👍

[@Jsfilmz]

agreed

[@FAITHBASEDGAMES]

@@Jsfilmz Also while I'm thinking about it. Do you know how to get the VR4D clothing into a project? I have tried to migrate the clothes, the pack of clothes, just about everything. I ended up just copying the files over and redoing the materials so that they would show up.

[@Jsfilmz]

@@FAITHBASEDGAMES so far ive been just using the original vr4d project as my base

[@FAITHBASEDGAMES]

@@Jsfilmz Oh OK, no problem. I didn't know if you had tried it or not. I'll just do what I'm doing, just make new simple material for the clothes, takes only a few minutes.

[@OMEGATROPOLIS]

Hey i emailed you Faith Based, We are looking for people of faith to add to our group. Wish JS would join us so he actually had a story to work on. LOL jk I am a long time fan.

[@caradabikeazul]

I can't find the Audio2gesture one omniverse.

[@Jsfilmz]

check out beginner tutorial i made couple days ago

[@caradabikeazul]

@@Jsfilmz I found it. We have to download machinima, audio2gesture is inside. Kind of not intuitive. 😅

[@matsche6099]

Just if anyone had a problem finding the IK_metahuman. I needed to update my Unreal version from 5.01 to 5.03 and then reimport a metahuman.

[@aneinanein5636]

i know im late… Can Anyone tell me how to combined Audio2Gesture and Audio2Face Anim in unreal? I have both animation sequences but the head separates from the body…

[@47humza]

how to apply animation on metahuman ?

[@mattivilho1493]

Thanks for the useful tutorial! I joined your channel to get access to the download file of the IK Retargeter but I can't find it anywhere. How could I get the file?

[@dweversole9036]

would it not have been easier to import your metahuman in omniverse first. do the skeleton binding where u have tools to match the rig to you final mesh. the export that rig back to ue5?

[@vi4dofficial]

All give donation to this guy

[@Nathan-ib2rs]

Great stuff man.good tech how to walkthrough. I'll have to check out the audio to gesture to see how useable it is in production. What are your thoughts on the quality of output?

[@Jsfilmz]

man for quick animation its legit very first of its kind its mind blowing man lol. for conversations i think it can work

[@rob_rauber]

Great Video! Is there any way to combine NVIDIAs A2F and A2G together and stream both data into the engine using live link?

[@madrasmicher]

great tutorial. just in time for my project. bless you. could you please make the ikrig mapping available for members ?

[@Jsfilmz]

i did check out the community post thats where i posted the project with the rig

[@madrasmicher]

@@Jsfilmz I recently became a member and new to all this. I looked in to the community Tab but dont see anything related to rig. The last post was from 19 hrs ago "Even @nba players getting into the @Unreal Engine #metahuman #mhc #unrealengine5". Sorry to make this painful for you.

[@Jsfilmz]

@@madrasmicher i think your a subscriber not a member

[@MrSunderr]

@@Jsfilmz great catch! I am now able to see it from the other account! Thank you. Please continue to do an outstanding job!

[@Jsfilmz]

@@MrSunderr cool thanks! ill drop more goodies in there eventually

[@rodszera]

Excellent video! I could make my animation work on my metahuman but the animation in the metahuman is overlaping some meshes and distorting fingers. Do you have a video how to fix it?

[@Radwalls]

JS… could we upload our own audio? And what about for performance- like singing?

[@Jsfilmz]

heck yes u can

[@jerrtinlsc]

Awesome!

[@DamielBE]

Does it work for lipsync on songs as well?

[@Jsfilmz]

yea omniverse has audio to face too

[@JhanicSatori]

I did joined as a member on your channel with membership, where do i go to download the rig?

[@Jsfilmz]

sorry mate they changed the rig so the template dont work anymore ill make new vid when machinima come out soon

[@JhanicSatori]

@@Jsfilmz That`s a bummer.... :(

[@chriscooper4910]

@@Jsfilmz any update on this?

[@윤씨-m9q]

Great! Thanks! I have one question, how could I combine animated face with body?

[@VladGohn]

I specifically subscribed for the retargeter asset, but it’s not included in the bonuses. well somehow :\ Was it so difficult to post the finished file without having to do it all over again?

[@Jsfilmz]

try this their skeleton changed iirc kzbin.info/www/bejne/pWfLmXaIobmUb9Efeature=shared

[@LoMozart-cr7mx]

sorry, where is the link i don,t see it even i subscribed for it

[@GENIUSGT]

Did you see RTX remix? Wild.

[@Jsfilmz]

of course yea its nuts cant wait to try

[@GENIUSGT]

@@Jsfilmz Super curious how well the auto-PBR texturing would work.

[@unveil7762]

I menage to have the pose tracker outputing osc adding a udp server in the extension it self. I guess that you can do the same for audio to gesture. And than use OSC plug in in unreal to read the bones data and send it via blu print in the the metahumans. Of course baking animation is nice but having a realtime workflow os always better!!! At least in my case!! 😂

[@ade4200]

Hello, 1 month ago you replied below that this does not work with your subscriber download any more? is that right? I joined the channel and spent the day not getting very far. Waste of hours if so. Maybe mention in description.

[@Jsfilmz]

Made a new video for you last month my dude kzbin.info/www/bejne/qZCbp4iObat9hrM also uploaded new file 3 weeks ago in the member page

[@ade4200]

@@Jsfilmz nice!

[@ade4200]

OK that new tutorial is a lot easier to follow for a beginner! I did it for Metahuman, fingers are weird, body a bit weird, I think I saw a tut from you about fine tuning. Shame your download is a bit off for us lazy folks haha, but for a couple of bucks to join its amazing. Thanks for the download.

[@ade4200]

The hands are all messed up on Metahuman and IK_Manequin. Also you don't seem to have mentioned in either video how to use our own animations from Machinima, when it's our own fxb import, the 'fixed' skeleton or rather the un-fixed skeleton from the new fxb import is linked to the other imported items. It seems to require a comprehensive knowledge of Unreal to make this work? Such that would not require this tutorial by the looks of it. Or am i missing something.

[@Instant_Nerf]

All this ai stuff and we still have to manually retarget SMH!🤣 where will you upload the retargeting for members?

[@Jsfilmz]

i already did mate check the community posts

[@damianwinter3045]

Amazing tutorial but it took a turn for the worse at around 7 mins lol

[@Netsurai]

It is strange that the M.H. cannot be imported into OmniVerse from the beginning. Looks like the OmniVerse developers have no interest or time to implement at least a UE4/5 standard skeleton. That's a shame, now 2000+ people can play around with this “completely unstandardized” OmniVerse skeleton, with OP joints :P. Unfortunately it's always the same, every animation app makes its own human skeleton, unfortunately in 2022 there is no minimum base standard in 3D apps, the only standard I know was BioVision Motion Capture (.BVH). We even messed weeks around in UE4/5 to retraget UE to M.H. :P Let's hope that in 2024 an AI tool can solve this mess or all 3D animation tools implement at least something along the lines of the old BVH. Sometimes I think that these developers regrettably have no idea what the end user really needs! They code the greatest things in, but they don't use the app in practice ….

[@Netsurai]

I like to call this OmniVerse Skeleton “Snake Human”

[@47humza]

This retargeting not working with Unreal engine 5.4

[@k.krocky3889]

Who else has been making soft for a wNice tutorialle before but with a different DAW and wants to learn how to us soft?

[@charlyhi]

Omg you buy that sht

[@Jsfilmz]

yeaaaa buddy

[@charlyhi]

@@Jsfilmz the clothes? Yisusuuu

[@Jsfilmz]

@@charlyhi yes worth it i think

[@sinanimacionnn-q3z]

Too many steps and the legs don't have good AI animation.

[@kunliao6224]

Bro you got an email? I have like a proposal I would like to send to you man