Generate 3D Sets for your Short Films!

[@PrairieFawkes]

For anybody having issues with your steps being visible and creating jagged lines and you can't figure out what it means to save in higher bit depth, check to see what kind of file you're saving from Auto1111. If you're saving JPG files, then those files have a lower bit depth than PNG files. Switch the save file type to PNG in Auto1111 settings and download the depth map as a PNG. This solved the issue for me.

[@SoulTuner]

I'm so grateful to you. I've been looking for a quick way to create virtual worlds for so long. And in this video you have revealed everything in such detail. I'm shocked, thank you!

[@relaxmax6808]

Are you open to creating a video clip for a musical remix without it costing me too much? I'm just a musician and I don't have a producer behind. thank you

[@SoulTuner]

@@relaxmax6808 It's possible) Please send me your remixes. I want to hear them) And write, how you imagine the clip. My email is in description of channel. Thanks.

[@relaxmax6808]

@@SoulTuner ok , thank you , i will answer on your email .

[@brandonjacksoon]

Thanks mate! Always awesome tutorials!

[@TheAllthegoodstuff]

Thanks for the training Mr Muppetz 😊 much love 🙏 ❤

[@jonathankr]

I've watched 4 of your videos now. You make everything seem possible. I know it's complicated and you are a genius. Do you do commissions?

[@nichodgkinson7163]

As a VFX artist with 22+ years experience I loved the line "if you shot on a green screen you can just simply key them" !! Very funny.. It seems vfx is finally having (or about to have) it's punk moment. Thanks for helping bring it forward!

[@theonm.5736]

many people still dont get it that they ll never get rid of the artists with the help of ai because of its own limitations, artists that are producing art, are never going to die

[@yvesdalbiez1395]

What's so funny about it

[@zo.mp4]

@@theonm.5736 those limitations will lessen over time. eventually you'll just need an idea and ai will do it for you

[@harem137]

its come along way...after that is the Glam metal phase watchout !

[@videojames199]

@@theonm.5736 I can’t emphasize this clearly enough, we used to think that AI couldn’t create “art”. Now that is highly suspect. The same thing will happen with the idea that we need artists to create high quality art. We are not special. We are bioorganic computers that are good at inference. That is all.

[@Dopamite]

Wow! What a brilliant idea to use ai generated normal maps to relight live action footage.

[@Thierrystheories]

Dude this is mind blowing

[@mauimrc]

Awesome video! I'm gonna try these on my giant L.E.D. production studio tomorrow.

[@Exilleron]

Wonderful work. I as a German Hobby-Artist, interested in Tracking and Matchmoving, was pretty amazed by your creativity and deeply explanations. I let an abo here. Good luck my friend.

[@Truth621]

I Appreciate your work , thanks

[@camiladorin4153]

Hi, can you please provide some explanation on how to shift the depth image values in photoshop?

[@jonckyy]

Well done ! nicely explained.

[@aiakym]

Hello Brother, please consider covering the software options available for this process, the steps involved in converting 3D animation videos to 2D animation videos.

[@kulewa1]

Thank you for the video, and I did hit the Like button. Yes, I am also experimenting with virtual sets to work in a feature length screenplay that I have finished. There's not enough money to do the movie live on locations, so virtual sets is the only other option. So far in the set building with Blender, my conclusion is...well...it might work...

[@relaxmax6808]

Are you open to creating a video clip for a musical remix without it costing me too much? I'm just a musician and I don't have a producer behind. Thank you

[@idanbs]

Amazing! Thank you very much!

[@coltish]

Love the content. I just watched something the other day about scanning someone into the Unreal Engine. I wonder if that process is simpler?

[@coltish]

It’s a Matt Wolfe video on using AI to create 3D games. I want to get my hands on Unreal’s metahuman. But your method allows for real human actors.

[@oscarjimenezgarrido7591]

_The future of filmaking_ Kerry Conran more than 20 years ago: _Here, hold my DeLorean..._

[@nightmisterio]

Can we use a 3D model background on unreal engine with all those tools also?

[@junttivision7743]

Awesome 🤯

[@Some1uNo]

Nice work

[@erichylland4809]

For Stable diffusion 1.5 now there is a 360 LORA

[@jxmentor]

ufff really nice 👌🏽

[@iamvulgar8188]

I have 16 gb of ram, an rtx 3050, and a ryzen 5 2600. I create my 3d setting in Blender and add a displace and subdivision and it just tanks my system and Blender becomes unresponsive, if I try to add a 2nd subdivision it crashes, if I try to render it crashes. 100% ram and before I turned off GPU acceleration it was using 100% of that too. This doesn't seem like a very demanding thing that's being done. My setup seems to be plenty enough to do something like this so why is it crashing blender?

[@cinematricks]

cool

[@nostalgiaprobe]

This needs a high GPU/RAM computer right?

[@ayliahandvicky7386]

It's good

[@foufedu74]

bonjour tout le monde j'ai cette erreur qui s'affiche Error code: 1 stdout: stderr: Traceback (most recent call last): File "", line 1, in AssertionError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check quelqu'un a une idée comment la régler

[@JantzenProduktions]

I've been using the "Ian Hubert" workflow (Put a video on a plane) for a long time, but this is awesome! My problem is: When I select my resolution in the img2img tab and set the same resolution in the sliders of the extension section, I get a "out of memory" error (I have a RTX 3070 and 32 GB Ram). If I change the resolution in the Extension section to 512x512, I don't get an error message, but I obviously get a 512x512 output. Does anyone know how to fix this to just generate a normal map in HD resolution without the out of memory error?

[@anonnymous7009]

You need video card memory not normal memory. Meaning the memory on your gpu. A 3070 has 8GB Vram. So you would need a gpu with more vram.

[@svgsprite4942]

👍👍

[@ChichinoS30]

Right, the issue here is that I can’t find a good tutorial for installing Stable Diffusion on my AMD PC. 🫠

[@TheFuzzypuddle]

It's insane that a person with creativity, persistence, and a willingness to learn can make a high quality short film with almost no resources. A phone with some apps, a computer with some video processing power, and some free and/or inexpensive ai tools will give you realistic composited visuals. The same equipment will give you a soundtrack and overall sound design. The tools are at our fingertips. The skill is knowing which tool to use for which task and having the imagination to combine their power to create something great. It reminds me of something Deadmou5 said several years ago when referring to creating music electronically: "This can all be done with a minimal amount of software which is why a kid can make a dance hit on a laptop."

[@arkadiuszm.j.wernicki7345]

4:50 MID-LEVEL Bro.. in the Displacement modifier you have this blue slider. Change it to 1, that will get rid of the "problem".😉 Great video! 👀🍿❤️

[@mickmumpitz]

Haha yeah, I should have mentioned that! The problem is that at a midlevel of 1, the strength of the displacement effect could no longer be changed (it just scaled the sphere). The difference between maximum and minimum displacement was too weak for me, the effect was hardly noticeable from the inside of the sphere. Do you have an idea how to fix this?

[@arkadiuszm.j.wernicki7345]

@@mickmumpitz What you mean by "the Displacement effect could no longer be changed"?

[@LFPAnimations]

@@mickmumpitz you could also use a color ramp node on the displacement map in the material to adjust the displacement map's range. Just make the white values grey.

[@genshian]

Mind blowing as usual! Blown away on the research and implementation you do. Exceptional!

[@RobertoBermeo_]

0:04 how did you put yourself in an animated version?

[@nrdkraft]

Here’s a thought. What if one took an HDRI of the real filming location, whether in front of the green screen or outdoors, et cetera. Then depending on how feasible it is, make a depth map of the original video, and smooth it out, and then displace the keyed video plane in front of the camera so that we effectively have a 3D version of the subject instead of just normal mapped. And then take the HDRI of the original film location, but give its brightness a power of minus one or something negative, and make that light only interact with the displaced video plane in hopes that depending on how accurate the the depth map is it could subtract the lighting of the original filming location (like in the case it wasn’t feasible to film outdoors on an overcast day, and have more distance lighting indoors but it’s also in practice to light it any other helpful way) and then add other lights and the environment otherwise to re-light the video plane as if the subject were another object in the scene. Probably way too overthought and impractical itself anyway but it’s a thought anyway😆 (I just wondered if there’s a way to record LiDAR on one’s phone in the video to use as the footage displacement map instead of generating it with AI)

[@manecolooper]

Excellent, always pushing the limits of imagination!

[@epessoarocha]

Loved the way you simplified all of this creation. Well done!!

[@stephenjohnson3307]

If you don’t mind my asking, how long did it take you to make a single virtual set? I’m wondering because I thinking of making some content with this method, but I wonder if the workload is actually manageable with what I have in mind.

[@toapyandfriends]

Subscribed

[@bUildYT]

and for those who r using adroid??? :(

[@Alex-rr7qc]

Heck yeah! He's back 🎉🎉🎉

[@LucienHughes]

Outstanding content as always. Pioneering stuff.

[@Caged_Monuments-x6p]

I think the AI revolution is to do everything without Adobe, since they've became a "you will own nothing and be happy" type of company. Thats why I use Davinci Resolve.

[@5timesm]

Sometimes I would like to see the name of the software used on screen……. Was it Blender, was it Davinci where you did the color correction? It would be helpful to see the names of the used plugins especially where you‘re a newbie to this whole process….😇

[@Kareem_Essam]

Incredible dude , I'm kinda disappointed because of the relight on the actor normal map trick didn't go as i expected , If i have to do so , I use photoshop and Ebysynth but that's a pain workflow.

[@Bluemograph]

A.I never replace filmmaking the reason is very complex, A.I is not standart of filmmaking (except plugins on some software maybe will help and solve filmmaking problems) A.i is not profesional for filmmaking, you're not done by just clicking 1video to generate and compete with profesional company

[@StrongzGame]

Does anyone know if you can export the scenes made into unreal engine?

[@TheFilthLA]

Dude... these videos are incredible. You're finding ways of doing things I thought it wouldn't be truly possible to accomplish for another 5 to 10 years. Very cool! I have an ambitious short film I shot most of in 2019 that I've been wanting to finish but knew it would take a significant amount of money to create the fantastical world in the film's conclusion. This has given me hope that completing it might be feasible on a much smaller scale if I sit down and really focus on banging out in AI that would be very hard to do practically or with more traditional VFX techniques. Definitely spreading the word and keeping a close eye on your channel going forward. It's VERY EXCITING!! Thanks man, appreciate what you're doing here!

[@TREXYT]

Midjourney have also --tile option, nice video btw 😃

[@Strawberry_ZA]

Hi Mick! Amazing tutorial! I'm trying to reproduce the workflow but I'm stuck on the depth map white value alteration. I don't know how to do this (i have photoshop, but no skills) - also, I can't overcome the blocky appearance of the depth map. I save the images as 32 bit depth but the same blockiness is still there.

[@ChristophTallerico]

I'm having issues with the "image plane from visible ref" step. When I click on the "image plane..." button, set the shader as Principled and distance to 1 nothing happens. Am I missing a step to get the image plane in my scene?

[@ValentinesProducts]

Great stuff! Do you know how I could find someone to teach me how to use these tools to make cool short films myself?

[@epelfeld]

It was great to see the full process without cuts also. Even as paid mini course. It's hard to get a lot of things without deeper explanation

[@zhehang1986]

the new feature of lighting in DaVinci resolve18.5 based on Depth may be useful in this workflow. maybe you dont need to build normal maps???

[@maya-akim]

Awesome job man! You always have great ideas

[@EricLefebvrePhotography]

You should revisit this tutorial using the new relight functionality of DaVinci Resolve Studio BETA.

[@quantitativemedia182]

My thoughts greatly aligned with yours...I have been fantasising on using AI to produce a short film for a while now. Just yesterday I found blockade...and then boom 💥 today I found your page...please can we connect and interact one on one ?

[@photobackflip]

Lol you think the future of filmmaking is fucking SKYBOXES.

[@DivineMisterAdVentures]

I never thought the day would come. Well, yeah, I did. I guess a couple of my stories from long ago cover it. But it seemed far away when I was trying to break into the Hollywood writer scene. It looks like even a single actor can play many roles.

[@azielarts]

You are a wizard! Thanks so much for sharing all of these explorations. Definitely taking notes!

[@AmitYt_108]

Thanks for this great video...... Is there any easy way to make cartoon short movie taking scene from an actual movie using AI....

[@bailahie4235]

That smile of "I enjoy this landscape" (while in fact you see the most boring green in existence)! 😂 😜

[@janjansen6443]

I am kind of scratching my head here. What the point of this all ? it seem to be subpar for loads of work. It seems like it would just be lots quicker and yield better results just making this all in blender or UE5, render a world in it or even pre load one into it. other than person's and or other objects that need a more realistisc direct interaction on screen. You don't all those other tools.

[@patfish3291]

Not so far from a classic workflow we do since 20 years only cheaper :D. The only thing who is really different is the possibility of AI image generation (start)..

[@Ollacigi]

im a 3d animator, the most hardest part of my skill is creating a good environment..and this become my new favorite channel!!

[@Kenb3d1]

Awesome work as always. Have you tried Tokyo_Jab's technique yet for generating more consistent frames in SD?

[@MODEST500]

i knew we will come to this stage, this is the teaching age where those who were the early adopters of the technology will start educating the people who are just trying to work things out . AMAZING !!! thanks for the workflow and tying things toegther

[@yashmehta6054]

Hey great tutorial! but can you make it beginner friendly after sometime i got lost :( Thank You!

[@titchc3657]

So glad you have this channel, great stuff.

[@RGBJD]

Very nice and great infomration. Thanks!. Pls do share more of this..

[@knottage]

There is so much in this video I have no idea about... still fascinating to watch though

[@thestorybank]

can you create something like leiapix is doing with blender if yes than can you please make a tutorial about it please

[@M4rt1nX]

You can record using an iPad pro as well if you don't have the right iPhone.

[@lanceanthony198]

Future of soulless artless creative works flooding and over-saturating every genre

[@mooh1245]

Can you please do one on creating 3d short film but using unity instead of blender

[@jagsdesign]

it has been an amazing learning curve and it certainly made me watch a few times to really grasp the technique and workflow. Amazing work and super cool explanation with a mixture of cross working tool sets which again makes it fun int he learning process. Thanks

[@mohaneedabdo3366]

love yourrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr videos you are amaaaaaaaaaaaaaaaazing thank you very much you are unique

[@nutzeeer]

I bet the hi res but seamed depth map can be layered over the better but low res variant. To remove the seam.

[@rajendrabiswas]

how to create 3d avatars who you can act with ?

[@rallyworld3417]

Oh my god this is 1000 times better then corridore crew

[@futuretechgeekgal]

super and will try these techniques myself

[@VaibhavShewale]

soon we will get more awesome yt short films and more backroom videos!

[@jordanco]

You’re skill and dedication is insane. Thanks for sharing this

[@grae_n]

Thank you for the kind words! The de-flickering normal maps sequence is amazing! I wouldn't of expected that to work so well.

[@josealberto-rj1si]

new node relight in fusion, generate normal map with ai

[@LFPAnimations]

Some really cool tools highlighted here. I am really interested to see where this workflow goes as the AI tools improve. I am a professional compositor and I have to say the "re-lighting" method made me cringe a bit. The normals are way too inaccurate for that ever to work. One idea for a fix though would be an image to 3d model AI so you can project the video back onto a generate human 3d model. You can then use that Geo to catch the lighting of the scene. This will also solve the flickering problem from the normals

[@nightmisterio]

Can we upload a image pre painted?

[@pixel325]

What an excellent idea and it does look ok. I'll have to try this out! Thanks for the insperation!

[@choptop81]

Waaaaah it's too hard cant i just put in prompts

[@denizcarta6805]

- Reference to image plan not working on blender 4.1 ?

[@denizcarta6805]

i alwas get a python error, frustrating

[@DanielPartzsch]

Cool idea. You should use the mid-level offset slider in the displace modifier to shift the displacement from center to outwards, this way you can keep the original depth map values and do not have to compress them. Also, instead of an uv-sphere using an ico-sphere helps with getting more evenly distributed subdivisions and this more predictable displacements on the sphere mesh.

[@readmore3490]

Do you slide the mid level slider to 1? I don’t know how to compress the depth map values and save in high bit value so your method might be easier 😅

[@DRiMe69]

I’m also curious about this since I don’t know how to reduce the black and white values in PS or change this file’s bit depth.

[@digital_magic]

awesome tutorial, very interesting and well explained

[@Way_Of_The_Light]

You’re soo underrated man 😭🙏

[@DRiMe69]

Great video. Can you talk a bit more about preparing the 3D environment, particularly those last steps once you have the depth map from controlnet? I’m not familiar with the flow for how to reduce the black and white values in PS or change the file’s bit depth.

[@EricLefebvrePhotography]

The volume has another trick, a stage that can rotate since the screens dont; go full 360 degrees. I'd be happy to shoot with a really good, large, short throw laser projector with UE5 ... you could do that for less than 6k (USD). Ryan Conolly did a few REALLY impressive tests with short throw projectors and Unreal. Was watching another video and a company sells a small LED wall setup for Virtual Production for as low as 10k (USD) ... not EXACTLY available to us indies but still a move in the right direction.

[@nightmisterio]

Wow 360^3

[@coloryvr]

WOW! What a great video! I've been experimenting with stable diffusion for months on my VR generated worlds. This video takes me a big step further.... BIG FAT FANX! So far I have used SD to generate endless textures for creating VR brushes or to integrate Deforum clips into 360 degree clips with Davinci Resolve. But: My main interest in connecting VR and AI is the possibility of transforming a 360 degree clip (based on a custom VR art world) into something else using SD Deforum...unfortunately I haven't managed to get a decent workflow done yet...mainly because of the large amounts of data ...(?) Happy colored greetinx!

[@reelwurld-studios]

Thank you for this video. I was looking for this exact suite of tools and workflow 🙏🙏🙏

[@michail_777]

Thank you for the video tutorial, as always very informative. There is a question. If anyone knows, explain briefly. In ControlNet there is a Preprocessor and Models. In the models, for example, there is "control_canny-fp16" and there is "control_sd15_canny.pth". What are the differences between these models. The sd15 just weighs about 1.5 GB. But the fp16 is 720Mb. "control_sd15_canny.pth" an older model? Or what is the difference?