Making Viggle Motion Capture Movies

While Viggle is mainly used for dance videos and memes, it can be used for so much more. Using generative AI images mixed with acting performances on video can be used to create some cool animated films.
Inspired by this video
• Ìràwọ̀ | Episode 1 - A...
Original how-to video by John Taiwo on LinkedIn
/ john-taiwo-51b720244_t...

[@nitromusik9275]

The first one to incorporate this into a streamlined tool together with storyboarding is going to be the king of AAA preproduction and used to level up pitching scripts by a lot

[@hyperbolicfilms]

Yeah, I think it's a nice way to do previz, and if you see my KZbin short where I integrated JetSet's virtual production app into this workflow, there's some big possibilities. But it is too many steps. I think it can already be a single workflow in ComfyUI (there's a Mick Mumpitz KZbin video that has relighting and background swaps, the only necessary addition would be Animate Anyone Evolved as a replacement for Viggle).

[@seanmaynard7417]

The acting is the key.

[@wholeness]

Always, a nice guitar doesn't make good player.

[@TheObaby18]

You did a great job 👍🏿

[@Steger13]

It looks like a ps4 gme. Pretty good but i say in about 2 years it's going to be crazy easy to make a realistic ai film at home. Cant wait!

[@hyperbolicfilms]

Yeah, I have found some tricks since that really improve the quality, but it doesn't ever quite look 100% real. It's fun watching this technology get better and better.

[@TPCDAZ]

This is really great work man

[@hyperbolicfilms]

Thanks!

[@Mrim86]

Love the lower third

[@FrankEBailey]

I really need to see General Spade having his face eaten off by one of the creatures, and our main character looking over with an "I told you so" look on his face.

[@hyperbolicfilms]

Yeah, seems like it should be the last enemy Spade faces.

[@jacekb4057]

This is freaking amazing!!!

[@VegterMedia]

Gonna try this with blender

[@brawlmaster7]

very cool!! :D excellent tutorial too! hahah, great work!

[@hyperbolicfilms]

Thanks a lot!

[@TheJan]

insane thanks!

[@Young_Zealous]

Yeah I’m getting this

[@gabeaiartist]

Wow, amazing film!

[@FilmSpook]

Very cool, thanks Bro! Just subscribed. Cheers

[@hyperbolicfilms]

Thanks for the sub!

[@Broadydudester]

Gonna be my next music video

[@markdkberry]

this is great. eager for AI open source movie making. I'm guessing 1.5 hour vids arent possible yet.

[@hyperbolicfilms]

@@markdkberry The rendering would take a while. I’m sure as the models get more efficient and video cards get better, we’ll start to approach realtime with all this, but I think that’s still a few years away.

[@Herve_art]

Crazy

[@marcusmanningtv]

Awesome

[@DrHansBrewery]

Cool

[@gintokigojo]

Wow

[@tunbimideoluyede8464]

Hey this video is amazing, I was just wondering what prompt you used to get the specific style of character you got?

[@hyperbolicfilms]

Thanks! The Midjourney prompt was "Photorealistic, full body, latino soldier with stubble in dirty t-shirt and black pants, 40 years old, white background, f4, 35mm" He ended up looking more concept art style than photorealistic. I later ran the image through Krea to get it to be more realistic looking, but that was in a video after this.

[@danielschweig3177]

I really like the workflow. Have you tried further editing the finished scene with Runway vid2vid? Maybe there is a way to create consistent runway vid2vid. It should actually work.

[@hyperbolicfilms]

I don’t think there’s a way to be consistent, but I’ll give it a try. I suppose it might work using the same prompt and a seed. As I was finishing this, I started playing with FaceFusion 3.0, and the results were very very good. Doing one more test and then I’ll put a video together about that fix. I think it removes the Krea step, and it can be run locally, so it’s free.

[@danielschweig3177]

@@hyperbolicfilms As far as “motion capture” is concerned, I know of 3 other AI platforms that have it: - Chromox -Domoai -GoEnhance I test the platforms myself and see who delivers the better result.

[@danielschweig3177]

sorry for the spamming, haha, but there is also the website software called “Wonder dynamics”. Do you know her? The results are a lot better than AI motion capture. The results can also be downloaded as an MP4 file.

[@hyperbolicfilms]

@@danielschweig3177 Yes, Wonder Dynamics seems very good, but the limited amount of time you are given (2.5 minutes in the $16 plan, 10 minutes in the $84 plan) has kept me from using it.

[@hyperbolicfilms]

@@danielschweig3177 Sorry, I thought I responded to this. I have tried it and it’s very good but they don’t have many character options to work with. I keep waiting to find a project to use it on.

[@IkeWaghSF]

I'm definitely on the Viggle train! I made this a few months back using Viggle for live performances. I tried my best to include it within a coherent story (character, violence, action, plot, etc.). Viggle wrote a blog about it. Hope you like it: kzbin.info/www/bejne/mqDQoZKdZ7KUr8k

[@hyperbolicfilms]

Incredible work! That is easily one of the best films made with AI that I've seen. Are you working on part 3? Really inspiring.

[@IkeWaghSF]

@@hyperbolicfilms Thanks! I typically can only make one big project every few months. I definitely have a draft script, but I'm working on my storytelling skills a bit before jumping into the lab. But it'll definitely be a project for early 2025! Glad you enjoyed it

[@LeonGustin]

Have you had the chance to check out Runway Gen3 video-to-video yet? Be cool to get ayour thoughts on it.

[@hyperbolicfilms]

I like it a lot, but in my tests I couldn't get a second clip to match the first clip, even with the same prompt and seed. The background would change, and so would details on the character. So while it's good for short shots and definitely easier than this method, it's not as consistent across multiple shots.

[@FSK2]

You can do it runway ml also

[@hyperbolicfilms]

@@FSK2 Video to video in Runway is not repeatable. Even with the same seed between shots, it will change a lot of the details. I hope they get there eventually, but it still has that limit now.

[@bhsmith5496]

What do you do about the generated image taking on the physical characteristics of the human model. I like Viggle but I have noticed that the image will take on the facial characteristics of the human model.

[@hyperbolicfilms]

I'm not sure there's a way around that, as it is taking the expression and posture from the input video. I walk with a bit of a hunch, and I noticed that it transferred over in some characters. Also a broader person will result in a broader character, at least sometimes.

[@bhsmith5496]

@@hyperbolicfilms Thank you!

[@offishow]

Is that old topaz vision free

[@hyperbolicfilms]

No, I just have an old license and can't justify the upgrade price. I don't think you can get the previous versions.

[@EllisJonesDeath]

What site did you use for your character, I have tried Kling and Bing but it always creates characters with shadows on the face, I have tried prompting no shadows etc, but it always adds them.

[@hyperbolicfilms]

It's hard to not get shadows. You can try asking for even lighting, flat lighting, or diffused lighting and see if that works.

[@sundayreview8152]

can you do a review of runway 3 video to video?

[@hyperbolicfilms]

@@sundayreview8152 I may make a short film with it this weekend, if my camera man doesn’t have Covid

[@wrillywonka1320]

Can you use a 3d mesh in place of the 2d image?

[@hyperbolicfilms]

Yeah, should work. This week I plan to test using a 3D environment and using a motion captured 3D character to do the action scenes for the same character that a human plays in live action.

[@knicement]

How did you slice the 2 minutes into 10 seconds each?

[@hyperbolicfilms]

In my editing app (Davinci Resolve), I rendered out 10 seconds of the performance at a time. It's very slow and tedious.

[@hyperbolicfilms]

In Resolve, you can also set the Output to Individual Clips, and then break up your video into 10 second fragments. That works well.

[@knicement]

​@@hyperbolicfilmsthank you

[@kleber1983]

Is wiggle free? Or is there a open source alternative?

[@hyperbolicfilms]

As far as I know, Animate Anyone is similar, but I don’t think it’s as advanced. There may be another one for Comfy with Motion in the title, but I don’t remember its exact name.

[@gwart41]

What program did you use to alter the voices?

[@hyperbolicfilms]

@@gwart41 Eleven Labs. They seem to have broadened which voices can be used for voice to voice in the past few weeks.

[@markdkberry]

look at RVC

[@pdwhalen]

Up your gain going forward please😊

[@giuliomarunti2093]

It all looks like shit... But considering the ease and time necessary to do everything it's pretty impressive!

[@FilmSpook]

Typical comment from someone depressed who has nothing on their channel. Get your eyes checked, son. *It looks excellent.* You are unrealistically hoping for Hollywood level production here, and that's not happening at this point. The result that he created looks very clean. You have no creative content on your channel, and nothing good to offer.

[@FilmSpook]

Says the Karen with nothing on her channel. 😂 The result that our bro created looks very good and clean. Where is YOUR original content on your channel?? That's what I thought, hehe...

[@izzmotion9917]

If you know how hard to motion capture or animation, you would never comment anything like this. This video isn't for people like you, don't be a cyber bully

[@giuliomarunti2093]

@izzmotion9917 ok guys, I wasn't cyber bullying nobody... My comment was just an exaggerated way to say that this ai model is still unripe, and impressing yet. Wasn't criticizing the work of the KZbinr in any way... but I think he understood what I meant, considering he liked the comment.

[@LFPAnimations]

On the one hand it's impressive how far this tech has gone. On the other hand I don't like anything about your final result. It is sloppy, inconsistent, uninspired, and inhuman. I'd rather watch the raw webcam performance of you delivering the lines than the AI version.

[@hyperbolicfilms]

That's fair, but in the past 3 months since I posted this, it's come much further along. Characters are much more realistic and I've incorporated moving cameras, so the filming experience can be much closer to traditional filmmaking, just with an AI layer over it. I see the evolution of this letting a small group of actors play dozens of characters in whatever setting they want. I also think this style of AI generated stuff at least incorporates the performance of an actor. So many of the AI video bros are desperate to move to the point where they press a button and get a shitty TV show. I am looking for ways to make sure humans remain a key part of the process.

[@LFPAnimations]

@@hyperbolicfilms I could see how this would be good to try out an idea, but I'd never trust it for a final. If anything it is probably good reference for a VFX artist to take it the rest of the way. Glad to hear you aren't on the AI bro train, but personally I remain extremely skeptical. Traditional 3D tools are so accessible and so good now that AI honestly has steep competition. Have you tried metahumans and Unreal yet?

[@hyperbolicfilms]

@@LFPAnimations Professionally, I am using AI stuff for previz. Essentially when clients need to show what the final project can look like, it's a better communication tool than just a storyboard or a moodboard. And that's where I think all of this tech is best at, I have used Metahumans quite a bit, even live on an LED wall as a background character, but I'm more of an iClone guy. More flexibility even if the characters aren't as realistic. I'm in the midst of a big iClone project now, and nothing can beat animation for real control of what the characters are doing.

[@realhamza2001]

I know this is unrelated but give the Quran a read

[@zherusalemvideos]

Hi there! Just shot you an email, but in case you missed it - I lead Partnerships at Viggle, and we would love to connect and chat!