Control AI Actors with this Awesome Workflow (LivePortrait Tutorial)

[@fraterseamus]

Best tutorial on Live Portrait I've seen yet, thanks for posting.

[@nikgrid]

@@DrZaious Agreed. You can do this for free using comfyUI and Live portrait workflow.

[@curiousrefuge]

Wow, thanks!

[@IAmVo]

THANK YOU!!! When you opened Comfy I almost clicked away. Its so so complex and I just wanna hire people to collab with at that point. But you broke it down SO well, I understood and it made so much more accessible thanks

[@curiousrefuge]

Glad I could help!

[@animateclay]

Thanks for the tutorial. it's the most simple solution out there for animating faces that I've seen!

[@heartshinemusic]

Thanks for the tutorial. These lip-sync features are so important to advance the control over consistent characters. I hope more A.I. video platforms will implement this under the hood, without all the incredible complex Comfy parameters.

[@curiousrefuge]

Glad it was helpful!

[@ZedMagnet]

I respect the flexibility of ComfyUI, but only super geeks are going to even attempt to grok it. Higher level tools (even if Comfy is under the hood) will be needed for wide spread usage, IMO. Awesome presentation. You are top notch, sir.

[@curiousrefuge]

Thanks for watching!

[@SuperDigitalAI]

For better results, I think it's better that you don't put in your prompt any "talking" "Saying" (mouth/lips movements) anything for the base video, so the mouth and face do not be doing any expression as base face video, so then will be modified from a "blank default/idle expression" of the face by the face controlled video, so the modification will happen to the same face and not a face that it's already doing/changing over time.

[@Aurelius511]

Did you have a stroke while writing this comment?

[@curiousrefuge]

True!

[@Hadar360]

Glad that I found your channel and tutorials, great stuff

[@curiousrefuge]

Glad you enjoyed this!

[@WattleburyWoods]

That sweater on that background is ❤‍🔥

[@EvilEyeWeddings]

word.

[@curiousrefuge]

Glad you liked it!

[@ART-ificial]

You were the best sweaters lol

[@DigitalAI_Francky]

With the local version and adding an upscale node, you should be able to have it in a bigger resolution 😊

[@curiousrefuge]

Great tip! Thanks friend!

[@malusjoon]

i would love to see your studio...how you setup and how you get the ring animation in the background! 🙂

[@curiousrefuge]

Maybe we'll do a desk breakdown someday :)

[@Realidyne]

So if you wanted to sync up with audio, I'd imagine you'd have to slow the final output by 25%? If that's the case you'd have to increase the fps by 25% (around 30fps) if your final final output is 24fps with synced video?

[@phu320]

I just went through a nightmare like this yesterday.

[@EmilyNilsen]

I found that as long as I had the same frame rate on each video, I didn't need to increase the speed of the driving video by 125% initially. Also, I got better results, prompting for subtle head movements only and without prompting the character to speak into the camera.

[@curiousrefuge]

Would def take some finesse in the editor!

[@logan_smith]

@@EmilyNilsen Yup. Grab the fps from the driving video via the Video Info VHS node and plug that into the Video Combine VHS node. You shouldnt have to do any speed changing.

Thanks for the info, some useful tips. Wondering how do you get around videos that need lip syncing? Also, where can I buy a sweater like yours? :)

[@mitchdavis5159]

Thank you, Curious Refuge!

[@curiousrefuge]

Our pleasure!

[@jzwadlo]

would it not make more sense to also record the reference footage in front of a white wall or plain background vs clutter in the background? Surely that makes a difference?

[@curiousrefuge]

It would likely help! But not totally necessary

[@jzwadlo]

@@curiousrefuge fair enough! 🤝

[@justinlloyd3]

Great tutorial! Thanks you!

[@curiousrefuge]

You are welcome!

[@MaHaw-d1c]

Please explain about live portrait installation and usage, thanks

[@curiousrefuge]

You can do it via huggingface or even in comfyui

[@abbasmedina786]

Awesome

[@FilmSpook]

Many Thanks for all your help, Bro.

[@curiousrefuge]

Always welcome

[@gumvue.studio]

thank you, this is good to try . I upload some AI films, 8 min, but will try this way also

[@curiousrefuge]

Glad it was useful!

[@photogeneze]

why there is no workflow downloadable?

[@curiousrefuge]

It's not something to download!

[@photogeneze]

​@@curiousrefuge i mean the comfyui's workflow, simple json file, of course the models and everything else is separate download, i understand that. But why not workflow file?

[@Tojek_VFX]

Agreed, can you please share the comfyUI workflow?

[@TheJereld]

I like the new background!!!

[@LaurenceBricker]

Thank you! Can you add how to deal with the actual audio in the performance?

[@curiousrefuge]

You'll have to bring it into your editor :)

[@IlluminousThoughts]

this is really cool

[@curiousrefuge]

Glad you enjoyed this!

[@chariots8x230]

Can you also do a video about ‘Crew AI’? I think ‘Crew AI’ seems like a useful AI tool because it gives users a lot of control over their AI character’s facial expressions, just by moving some sliders around. It doesn’t really create animations, but it can change the facial expressions of characters in still images. I need to work with both still images and animations, so I like that there’s a tool that can modify characters in still images like this. Also, in terms of animation, I can see a tool like this being useful for creating a ‘start frame’ and ‘end frame’ for an AI animation.

[@curiousrefuge]

We'll check it out!

[@Sean43322]

Thanks for tutorial. You used close-up image and the result was almost good and acceptable. My question is, can the same result be obtained with medium or wide shot images? Have you ever tried it?

[@curiousrefuge]

We will have to do some testing! But close up is always best results

[@julbombning4204]

Hey I want to buy your sweater asap, where can I buy it?

[@festivitycat]

Love that sweater!

[@curiousrefuge]

Thanks!

[@okgo5152]

Thanks, great video!

[@curiousrefuge]

Glad you liked it!

[@joakimmogren1727]

I'm a horrible actor and I don't want to record myself. I know this is going to sound incredibly lazy but I hope the option to prompt different emotions becomes available. I don't need Oscar worthy performances for the projects that I'm working on. It's great that's there's a lot option for controlling faces. Now only if we could have 3D poseable consistent characters we'll finally be able to have AI actors?

[@SykoActiveStudios]

Metahuman with unreal.

[@JulioMacarena]

Hedra AI - check it out.

[@JulioMacarena]

​@SykoActiveStudios thanks. Tell me more. :)

[@curiousrefuge]

You can certainly try other things like Hedra!

[@SmartVideoBE]

can we run it locally on mac or windows without paying? how is this done?

[@curiousrefuge]

Yes but we prefer running it online as it takes quite a bit of cpu power

[@mitchdavis5159]

How do I get the output video to be longer than 8s?

[@curiousrefuge]

Editing software and splice clips together :)

[@StéphaneLévy-i7g]

Can't get this to work. I've followed the tutorial step by step, and used the specified assets but the video combine node always appears EMPTY and I cannot render. How to fix this? Thanks in advance.

[@curiousrefuge]

Feel free to jump in our discord and we can try and help you troubleshoot!

[@StéphaneLévy-i7g]

@@curiousrefuge Thanks.

[@TheCrackingSpark]

Hey! Great video. I have an unrelated question, kind of an odd one. What's your background? Are you using a greenscreen? If so why can't I see any green in your glasses? Is it not a green screen and just a straight up tv screen? Your shot is lit very nicely, and it matches the background very well! I just had that question lol

[@curiousrefuge]

It's a projector :) Glad you enjoy the look!

[@Escelce]

Great video thank you

[@curiousrefuge]

Glad you enjoyed it

[@High-Tech-Geek]

Just to clarify if I've got this correct: 1. If you're uploading a control video to animate an image, you can move your head all over the place and the resulting video will mimic those movements. 2. But if you're uploading a control video to animate a video, you need to keep your head still as it will only apply your face to the video character that is already moving their head. Is it correct to say the control video head movements will not affect the existing video character's head movements?

[@adarwinterdror7245]

Curiois about this as well

[@curiousrefuge]

Jump in our discord and we can help clarify!

[@0A01amir]

Any site with face portrait videos that we can download ?

[@curiousrefuge]

You can use Midjourney to make thme!

[@omer-books]

Can you put the link of the workflow?

[@curiousrefuge]

We'll see what we can do!

[@stevey19861]

What background are you using?

[@curiousrefuge]

A projector :)

[@khalil2099]

can we only do lipsync ? meaning, the app will ignore the eye.

[@curiousrefuge]

It's not as refined for other facial features

[@elgodric]

What's the GPU requirement to run Live-portrait locally?

[@curiousrefuge]

Good question...we'll have to check!

[@DanielSierra-l1u]

Why does my output come out so damn blurry? Its so blurry i can barely see lip movement

[@curiousrefuge]

Let us know in our discord and we'll try to help!

[@Gamingtamilworld-o4x]

Why we have to use comfy ai nodes? When we already have the liveportrait easy way of uploading videos and photos to make this process easier? Whats the difference bro?

[@curiousrefuge]

You can use either!

[@nickjames5602]

It's not letting me pay $1. It says I have to pay at least $16. Is this normal???

[@curiousrefuge]

Hmm....it may ahve changed over the thxgiving/black friday sale.

[@ai.ai.captain]

❤❤❤

[@faisaledits1]

Gen 3 Alpha is not free 😔

[@curiousrefuge]

True! Not free for Runway!

[@JoJoAIventures]

Thank you, ComfyUI is pretty much a game changer these days. Every serious AI video creator should consider learning it.

[@curiousrefuge]

True! It's a big deal!

[@NoFaithNoPain]

What we need is faces at an angle and people who are moving

[@curiousrefuge]

True!

[@DaveandhisDeathbeanie]

Why are people so into the sweater?!

[@curiousrefuge]

Haha we have no idea!

[@planetofshireen]

Wowwwwwwwwwwwww

[@curiousrefuge]

thanks for watching!

[@JulioMacarena]

Tell the actor that you're going to use their single performance many times in different projects before you record them. :)

[@curiousrefuge]

Hah, it's a MJ image!

[@deepdiver849]

Its not working for me when i upload an image of an ai created cat. It doesn’t recognize it.... so ai is still behind

[@curiousrefuge]

Hmm, jump in our discord and we can try to help!

[@julx97]

i dont like ComfyUI, that's why I like to use LivePortrait in "LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control".

[@adarwinterdror7245]

What? What are you talking about? Is there an alternative to do what was shown in this video without comfy UI and doesn't involve installing Live portrait on the PC? (Which I had issues with)

[@julx97]

@@adarwinterdror7245 aitrepreneur made a tutorial about this topic. It’s his second most recent video

[@julx97]

@@adarwinterdror7245 a it re pr en eu r made a tutorial about this topic. It’s his second most recent video. I had to write spaces in between. As my comment disappears when writing his channel name

[@findthatvid-pro]

@@adarwinterdror7245 Find the link in my post (as KZbin comments get deleted all the time) The post will expire in 24 hours

[@findthatvid-pro]

@@adarwinterdror7245find the name in my post

[@GaryMillyz]

AI gfs are gonna be f'in wild

[@curiousrefuge]

Hahha, wow!

[@jonathankrimer]

Im not sure if my scrept needs that. Lol

[@curiousrefuge]

Maybe one day it will :)

[@Steger13]

Still not perfect though need another 6 months to a year i think

[@curiousrefuge]

Indeed! We agree!

[@kamizama9177]

Not easy at all lol

[@curiousrefuge]

We appreciate you trying!

[@ryan18462]

Every AI tool is the best!

[@jeremyleonbarlow]

If you want easy, Runway has Act-One for talking heads. This however looks like it might be beneficial if you were doing a walk and talk video from an image to video and you needed a better facial performance on the character. Is it easy? No. Is it less expensive and easier than hiring a location, hiring actors, and doing a full fledged on location shoot to get essentially a similar shot? Yeah, and what's more you can create the perfect version of your character if you iterate to get what you want in the original image generation and take the time to train proper LORAs.

[@woltergeist9175]

AI is really good at generating non-ugly women

[@curiousrefuge]

True! It's trained on a bunch of models it seems for all genders!

[@fabianoperes2155]

Looks too artificial, maybe in w few next versions they will fix it.

[@curiousrefuge]

We def get better!

[@ToneVirtue]

I think it is over. All music and film was about relating to fellow human beings. This AI is making everything oversaturated and quite frankly pointless. Doesn't make me feel like watching anything. It is just a pile of plastic straws at this point.

[@themightyflog]

Nah. That was done away with when it costs so much to create a vision. Money made us lose the connection.

[@curiousrefuge]

Before AI, how did you feel about online content?

[@ToneVirtue]

@@curiousrefuge Great. It opened a market place with loads of opportunities and jobs for people. If you want people to use AI to replace photographers, filmmakers, writers musicians then It will basically kill the whole collaborative market. AI content was not made to improve anything but rather to destroy content creation economy and alienate people form each other. Soon AI won't even need people to write a prompt. I personally think they should ban AI generated content monetization to save content creator economy.

[@robertdouble559]

Very rubbery. No regard for underlying bone or muscle structure. But hey, its early days, its only gonna get better. Good enough for memes and amateur work, I guess.

[@curiousrefuge]

It's certainly still developing!

[@lefourbe5596]

Lora dataset augmentation. Great for that

[@雪鷹魚英語培訓的領航]

Still not good enough for my taste, but one or two more papers down the line, or if the EMO paper gets put to use I'm game.

[@curiousrefuge]

True, still needs a little work!

[@Pure_Science_and_Technology]

This is bad.

[@curiousrefuge]

It's come a long way!

[@anthonymark2516]

Not bad! But my technique is better. As I don't need any input videos, just my imagination. And I can get multiple performances, so I can actually direct the actors by choosing the performance that works best. /watch?v=iMXbSXSJqqs We're at the point now where the tech does not matter so much as how we use it.

[@curiousrefuge]

Thanks for sharing!

[@seencapone]

It might have been better if you would've laid your original vocals over the final clip so we could see how well the lip sync holds up through the processing. No one wants to go through this whole workflow just to make a silent film of people flappin' their lips. This reminds me of all the face capture demos -- and I've watched 'em all -- no one demonstrating the tech ever records any dialogue or normal scenes of simple talking, they just pull crazy funny faces like that's supposed to be useful.

[@curiousrefuge]

You've watched them all? We've seen quite a few of people using their real voice (and typically adding some flavor with elevenalbs with v2v)