Luma vs Runway Gen 2 Comparison

So while we were all waiting for Sora to be available, Luma dropped. And it's really mind blowing 🤯
To put it through its paces, I made a quick comparison of Luma vs Runway Gen 2. No Sora comparison since I don't have access yet. But Luma is clearly way ahead of Runway.
That is to say Runway Gen 2. Because Runway just announced that their Gen 3 model, which is supposed to be a general world model, is coming in the next few days.
Let's get back to the comparison.
To keep things equal, I did not provide any text prompt.
I simply uploaded the same Midjourney image into both tools and let them convert that to video in the best way they thought.
This approach also reveals the image assessment qualities of both video generation tools, as the animation gets decided based on their assessment of different elements and the inferred motion within the source image.
Looking at these comparisons, Runway's animations tend to be very subtle. That is probably to limit morphing. Gen 2 is not a general world model, so it's animations are based on known movements of things like clouds, water, etc. This is also why it tends to produce a lot of morphing when human movement is requested like walking, running, etc.
Luma on the other hand favours BIG motions. Head turns and sweeping camera motions that really show off its general world model.
It's impressive that Luma kept object and landscape geometry intact during these large motions. The reflections on the muddy water in the jungle scene for instance is incredibly impressive.
Having said that, I haven't had much luck with Luma when it comes to controlling the motion with text prompts when an image is used in input. It tends to favour its own assessment over the text prompt when it comes to motion. If you have figured out how to control Luma better, drop a comment below! Would love to learn.
As soon as Runway Gen 3 is released, I will do another comparison, but until then, check out Luma. It's free for up to 10 videos per day.
See you next time!
#GenAI #Luma

[@robbiekohen]

lumaaaaaaa

[@denieladam2046]

Is it me or does runway do nothing in the 3rd one

[@YalinSolmaz]

Runway just adds a minor eye blink and some movement in the hair. But you're right, it's so minor that it feels like there is no movement at all!

[@ghost-user559]

I’ve had Luma do literally nothing at all multiple times. At least in this case Runway makes the character and background do something. It’s not much but at least it does something.

[@YalinSolmaz]

@@ghost-user559 That's interesting. I've had the opposite experience with Luma. It usually adds TOO MUCH movement more often than less. But it might also be related to the images I'm feeding it...

[@ghost-user559]

@@YalinSolmaz Yeah I do think the original image has a lot to do with how it interprets the camera and character movements. I’ve had the best experience with Kling now actually. Kling has much more “dynamic” and subtle movements, but at the moment it is lower resolution than Luma, so it takes me forever to upscale the scenes one frame at a time. Luma has given me some phenomenal scenes, but for every good one, it either gave me exorcist level of motion with jittery contorted horror scenes, or no motion at all lol. So Luma needs a lot of work for consistency. Kling is amazing thus far, but it’s new, and basic features like higher resolution are still being added.

[@YalinSolmaz]

@@ghost-user559 "exorcist level of motion" my fave way to explain Luma's tendencies now 😆