Kling is very overpronounced with its lipsync generally. It's good for people speaking loud. Runway is better for people speaking quietly.

Yes, just working with it right now. Much better on characters with realistic shading, but not getting great results at the moment with anime style characters.

Thanks! The Midjourney prompt was "Photorealistic, full body, latino soldier with stubble in dirty t-shirt and black pants, 40 years old, white background, f4, 35mm" He ended up looking more concept art style than photorealistic. I later ran the image through Krea to get it to be more realistic looking, but that was in a video after this.

A Whovian with enough disposable income could rebuild so many lost episodes from Doctor Who!

Yeah, definitely has that rotoscoped feel!

Glad you enjoyed it

It's hard to not get shadows. You can try asking for even lighting, flat lighting, or diffused lighting and see if that works.

This started as Midjourney images that I animated with Viggle, and then used Krea to clean up the video quality.

1. In theory. I think they slow you down after a certain number of credits. 2. You have to upload a video of someone acting. It's essentially like a motion capture for the face/head. 3. I don't think Runway has any translation functions. 4. If you want to take a photo and an audio clip and make a talking head, there are other tools that do that. Kling does it indirectly. Hedra is probably the easiest way to do this.

ElevenLabs voice to voice

In my editing app (Davinci Resolve), I rendered out 10 seconds of the performance at a time. It's very slow and tedious.

In Resolve, you can also set the Output to Individual Clips, and then break up your video into 10 second fragments. That works well.

​@@hyperbolicfilmsthank you

@@Mrim86 Thanks! I’m trying to think up ways to break the dialogue shots up as well, so there can be some change in pose to fit the dialogue. Might not be feasible with Act One as it is

These backgrounds were generated in Midjourney itself, along with the character.

Hedra uses audio to automatically animate a photograph, but you don't get control over how it moves the face. Live Portrait is similar, but the results of Act One are much better. With Live Portrait, some face movements add jitter to the face. Act One also seems to work well on stylized faces, which I don't think is the case for Live Portrait. At least I can't remember seeing any results of stop-motion style characters.

@@hyperbolicfilms Cheers for the reply, be interesting to see the coming updates from Viggle, Hedra and poss Live Portrait - lots of compation with AI tools now :D

Yeah, I have found some tricks since that really improve the quality, but it doesn't ever quite look 100% real. It's fun watching this technology get better and better.

I have used Krea for this in the past too, and it does work well. The 10 second limit is a pain, but the results are great. I hadn't heard of Flowframes. Thanks, I'll check it out!

I used to make stuff in unreal. I joined sum intensive courses where they had worked on movies. They ALWAYS said right at the beginning to work in 24fps when making movies . Maybe to many frames gives the look a strange artificial vibe.

@@armondtanz You're right, but the reason it's a good idea to make Viggle footage into 60 fps is that it smooths out some of the jittering that Viggle causes when it outputs a video. Then you can turn use that 60 fps video in a 24 fps timeline to get more consistent motion.

@@hyperbolicfilms oh ok. I'll have to check out viggle. Did u ever see the workflow of the guy who made the joker walk on stage viral. He put it thru comfy. I'm a complete noob when it comes to comfyui. Looks insane. His end result was so polished. You could see textures and shadows in clothes. They were not there in original viggle version.

@@armondtanz I haven't seen the Joker one. But Eric Solorio did a Deadpool video with Viggle and Comfy that came out really good.

It's a combination of images generated in Midjourney and some Photoshop. There's a very long tutorial about it called Making Dialogue scenes for AI films with Runway and Kling

@@hyperbolicfilms thank you

The resolution is 1280x768. There is no option to do any camera movement, so you would have to add that in post production. The background is limited to whatever is in your input image, so no alphas. It's a step in the right direction, but not a magic bullet.

I did that a few weeks ago, and it did give amazing results. The 10 second limit in Krea is a bit of a bottleneck, but it definitely gives great and consistent results.

I don't think this would work because Act One only takes images as an input. LivePortrait is the only tool that I know of that works on video like that. You could use the Runway lipsync on a Viggle clip, but I don't know if it would fix the resolution of the Viggle clip. That to me is where Viggle is falling apart. It doesn't use the full resolution of your input image, or doesn't upscale the results.

@@hyperbolicfilms I meant: 1) animate the head in Act One (picture + video with facial expressions). 2)animation in Viggle (the same picture, only on a smaller scale, with arms and legs + video with body movements). 3) Superimposition of the head from point No. 1 on the body from point No.2. P.S. A lot of hassle, and you can't twist your body much, but your head is in adequate quality.

​@@С.Н-ш2уI tried this a while back with hedra and my body. I tried to composite it. It's never going to work. The human body is so so complex every tiny movement has a 100 off shoots which all need perfect sync otherwise it looks like crap. Funny thing is my test was the simplest , I was talking into the camera and it still looked like a bad bad animation. I've wasted over $1000 and 100s of hours trying to crack this. Its 100% not worth it.

@@armondtanz Thanks for your experience

@@С.Н-ш2у i learned the hard way, stubborn fool who didnt check out the competition.. lol, thats probably the easiest way to work something out, if no one else is doing it, its probably not going to work... The only way it can come off in the slightest is if you are a super advanced animator and can match the movement with advanced motion tracking and stabilizers, then sit the head on using tracking markers... But even then its still gonna look unnatural and everyone will be focusing on this head that's not quite sitting right on the body. Other factors will come into play, like lighting, 3d rotation, your neck muscles not reacting to your shoulder muscles. If you look at all great animators (tex avery). That part of the body is so so crucial. Theres so much expression in head-nick-shoulders. Thats why these new AIs look a bit a flat, that area is really behind.

The ultimate would be a gaussian splat type scenario where it's almost like 3d software. U can pan out of your scene and see yer 3d world. I think midjourney are launching it or looking to. They said there video gen is ready but they wanna better it. They also talk of 3d environment, so maybe that's the future.

Amazing work, love the perseverance on getting your idea to reality.

I first generage the images in Midjourney because it's the easiest way to get consistent characters. So I prepare my shots there. Then I use the image to video in Kling to get something like "the man talks calmly". When I am happy with the way a clip looks, I then use Kling's lipsync to get the mouth to match the dialogue. There's a long tutorial on my channel that shows my whole process (I show both Kling and Runway)

@@hyperbolicfilms Thank you very much. Yes I have already seen your very interesting tutorial, but I had this doubt because maybe at that time Kling did not have lip sync and the videos it generated were not very good. Anyway you have seen now Runway has announced act-one which can be very useful in this. Greetings!

Glad it was helpful! Aximmetry is a great option if you want to see your final pixels immediately, and Jetset is great if you are willing to do some post production.