Getting Started with ControlNet & Deforum

[@techniquebeats_dennis]

It's great to see ControlNet applied in your video. The information in Discord can sometimes be overwhelming, especially due to language barriers. Your tutorial provides clear explanations for everything, and I appreciate that. Thank you for that.

[@Andy_XR]

You are most welcome. I'm working on some others, to add more detail.

[@SeraphsWitness]

This is EXACTLY what I was looking for. Days of research, and you distilled it into the exact problems I was needing to solve.

[@functionrenderdream]

Ah man, that's great to hear. As you delve deeper, you'll soon discover what works for one source video, doesn't work for another. It can be frustrating, especially when you think, 'I've cracked it!' And then you realise you have to start all over again with a different video. But that's the bit I enjoy; learning what works next. Good luck and thank you for words.

[@SeraphsWitness]

@@functionrenderdream Good to know! Thanks again.

[@stevietee3878]

Thank you very much for posting this, I wish more people posted such valuable information with as much detailed clarity.

[@functionrenderdream]

My pleasure sir!

[@Meyasfashion_at_ibp]

Thank you for this just getting started, and this is really welcomed. My husband was developing linux mpeg back in the 90s

[@functionrenderdream]

Glad it helped!

[@reallybigname]

Right on, man! Just came up on my feed.

[@leebz]

Thank you, really helpful info.

[@functionrenderdream]

Glad it was helpful!

[@honganh-t9e]

Oh my god, it looks so beautiful, just like warp diffusion. However, creating a warp clip requires a lot of colab units, that's hurt. Thank you very much.

[@functionrenderdream]

You're welcome and thanks for the feedback!

[@EllenVaman]

LOVELY :)

[@JustinBalaguer]

ive been finding everywhere on why my stable-diffusion-webui has the controlnet not found error in the deforum tab. but i already installed it. help

[@functionrenderdream]

I’d start by checking you’ve got the latest version of Deforum. Also is this Automatic1111 or Forge?

[@TomStone-bs4hd]

Love it

[@functionrenderdream]

Thank you, Tom

[@cmdelavega]

Thank you for the tutorial!! I'm having issues with ControlNet (using ver 1.1) - I specify the video path but the generated video stays always on the first frame of video, it's not following the video. I've tried Canny, Depth, Pose, Balanced, Prompt is more important, Controlnet is more important. Any suggestions I would super appreciate it. Its frustrating haha. Using Input Video but want to start using the method you used in the tutorial. Thanks amigo!

[@functionrenderdream]

Post your question on the Deforum discord… more people will see it for sure. Thanks!

[@functionrenderdream]

…and I’ve seen this happen before. I fixed it by re-downloading the yaml and pth model files from the official huggingface site

[@cmdelavega]

@functionrenderdream cool thanks for the heads up! Quick question (I'll also ask on Discord) but do you see difference when using better quality videos (higher rez) vs lower resolution?

[@functionrenderdream]

If you’re passing in an upscaled video then you’re more likely to see retained features.. but at the cost of processing time.

[@rubb3rmind]

Thanks man, this is super cool and super useful (3oxis)

[@functionrenderdream]

Hey 3oxis! Thanks!

[@oskarreschke9673]

what am I doing wrong? I followed your tutorial and it looks like diffused video follows only first frame from the video I put as a control net template. There is not no human body movement, it stands still as in the first frame

[@functionrenderdream]

Hey! This can sometimes happen if you have other extensions installed in A1111. I keep a separate install of A1111 with just ControlNet and Deforum, to be on the safe side.

[@TomStone-bs4hd]

Thank u

[@JayBingoOnYT]

You sir are a god thanks dog

[@Andy_XR]

😇 I wouldn't go that far! But thanks for the compliment! More to come.

[@JolieKim2795]

"If you don't use hybrid video function, is there any difference between Deforum video and Img2Img?"

[@AB-wf8ek]

The only difference I can think of, is Deforum allows you to animate certain parameters. So you can animate the strength, noise amount, prompts, checkpoints, cfg, steps, etc.

[@JolieKim2795]

@@AB-wf8ek oh. Img2img batch rendering could also do that from 10 years ago. I think the difference could be, hm .. it does the whole process from extracting framing to forming video, while img2img you need to do each steps

[@AB-wf8ek]

@@JolieKim2795 Ok, well that's news to me. I've never seen anywhere you can animate things like cfg scale, steps, and prompt keyframes with img2img batch before. But if it's possible to do that, then there's not much difference as far as I understand.

[@functionrenderdream]

No, in Deforum we can take each frame of a video and use that as the init source, thus allowing all sorts of extra fun including, but not limited to, hybrid motion, flow factoring, etc.

[@jaggedppsrg6777]

💪💪💪

[@thegrizzlyofzemountain1350]

negative prompt: --neg NSFW, Cleavage, Pubic hair, Nudity, Naked, Au naturel, Watermark, Text, Censored, deformed, bad anatomy, disfigured, poorly drawn face, mutated, extra limb, ugly, poorly drawn hands, missing limb, floating limbs, disconnected limbs, disconnected head, malformed hands, long neck, mutated hands and fingers, bad hands, missing fingers, cropped, worst quality, low quality, mutation, poorly drawn, huge calf, bad hands, fused hand, missing hand, disappearing arms, disappearing thigh, disappearing calf, disappearing legs, missing fingers, fused fingers, abnormal eye proportion, abnormal hands, abnormal legs, abnormal feet, abnormal fingers, nsfw, nude

[@honganh-t9e]

would you please sharing your setting by txt file

[@Divineshot]

Is this similar to warpfusion?

[@functionrenderdream]

Deforum can achieve most of what WarpFusion is capable of, and more. Deforum allows keyframing of an animation driven by prompts (which themselves are keyframeable). WarpFusion is amazing at processing videos, whereas Deforum is able to work with different sources including videos. Oh and Deforum is 100% free! 😃 I love WarpFusion though but it does have a steeper learning curve as it’s notebook-based.

[@Blooplerful]

Where is the discord thread you talk about :) I have a lot of questions

[@functionrenderdream]

Discord: discord.gg/deforum

[@livclark9871]

Why is my controlnet not working, the image is very different from my video

[@functionrenderdream]

Join the Deforum discord and ask a Q in their help channel.... Lots of help there.

[@livclark9871]

@@functionrenderdream Thank you for your answer, where can I join

[@Andy_XR]

@@livclark9871 Link to the server in the vid description

[@mahesahoesain5587]

same here mines also look very different

[@victorwijayakusuma]

I tried using controlnet tile and follow all your step here, but deforum dont give the result like your, its just creating random video

[@functionrenderdream]

Join the Deforum Discord server and raise the question there. Best way of discussing the issue. And it’s free! Over 20k members so far.

[@victorwijayakusuma]

@@functionrenderdream yes thank you will do! thank you very much

[@mariomaffiol]

Que grafica usas y de cuantas GB vRam es ?

[@functionrenderdream]

Nvidia GeForce RTX3090Ti Founders Edition with 24Gb VRAM

[@functionrenderdream]

I used this as the source kzbin.info/www/bejne/aYGahnWkbqx9apI

[@serkanaydinli]

prompts ?

[@functionrenderdream]

You want prompts? Wanna learn how to prompt? Plenty of resources on YT. Deforum is built upon SD, so if you know how to prompt in SD, then you're good to go. Oh, except that if you're using an instructional model, you can make your prompts more directional ("giver her red hair" etc)