Stable Video Diffusion Tutorial: Mastering SVD in Forge UI

oin us in this engaging tutorial where we explore the capabilities of Stable Video Diffusion (SVD) using Forge UI. This tool opens up a world of creative possibilities and we're here to guide you through mastering it.
📌 What You Will Learn:
- How to set up Stable Diffusion in Forge UI for video creation.
- Steps to download and integrate the right model from Civit AI for optimal results.
- Understanding the system requirements necessary for running SVD smoothly.
- A walkthrough of the SVD interface for uploading images and creating videos.
- Tips on configuring settings like video frames, motion bucket ID, FPS for custom effects.
- Techniques for generating unique images and experimenting with various seeds and samplers.
- Methods for converting images into videos and enhancing them with advanced tools.
- Practical advice for achieving high-quality results and overcoming common challenges.
00:00 SVD Checkpoint and Settings Forge UI
02:20 Generating Videos With SVD
03:45 Video Upscale with Topaz Video AI
04:29 More Examples and Tips

[@pixaroma]

I got the checkpoint model for SVD from here civitai.com/models/207992/stable-video-diffusion-svd Remember it can generate at 1024x576px or 576x1024px, you can use bigger images when you upload but try to keep them same size. It can generate 4 seconds video, you probably can get the last frame and get a continuation of it for another 4 seconds and so on. You need like 6-8 GB of VRAM. I used Stable Diffusion Forge UI If you want to learn more about AI or have questions join my facebook group facebook.com/groups/pixaromacommunity

[@nietzchan]

Still can't get it to work in Forge. I don't know what I'm doing wrong, does it have to use SD 1.5 checkpoints and VAE first? Tried using Animagine XL 3.1 with XL VAE for the initial image feed and running SVD just sent me to BSOD

[@pixaroma]

@@nietzchan it need a lot of video ram so many that why is crashing. It just need a photo on that that size 1024*576px and work with that svd model to generate, if it crash probably your video card can not handle it

[@nietzchan]

@@pixaroma I think my forge installation have memory management issues, or probably the something wrong with the unet setting. I managed to run it once when I'm just running SVD, but the second time it just crashes. I'm currently using 12gb 3060 and 16gb ram, and I think the bottleneck is actually the ram when Forge is automatically load models to ram on start. I want to try offload from vram options and see if it helps.

[@nietzchan]

Confirmed, I need more vram. Tried to use the offload models from vram args so SVD have plenty of room in GPU. I'm using RTX 3060 12gb despite SVD only uses around 8gb of vram the Forge backend still have the image diffuser model in vram, resulting in OOM on my GPU. the offload args works, but instead it didn't offload the SVD models once you generate video. So I'm back to square one after each generation. Oh well.

[@pixaroma]

@@nietzchan sorry it didnt work, usually they make it less vram consuming in time, so in a few months maybe we have better models and systems

[@SumoBundle]

Thank you for this tutorial

[@CornPMV]

Nice tutorial; I enjoy the animatediff extension I get really good and consistent results using it!

[@iangillan1296]

where do you use it? In comfy? I installed AD inside Forge UI, and there is now any changes, AD didn't appear

[@baheth3elmy16]

Thanks! Another good tutorial video!

[@pixaroma]

Thanks :) glad you like it

[@UmarandSaqib]

Nice one!

[@Robertinosro]

cool stuff. thank you

[@cruz2480]

Great video, subscribed. Keep making great content.

[@richctv]

Awesome tutorial. Keep up the great work

[@pixaroma]

thank you :)

[@sb6934]

Thanks!

[@DrDaab]

Great, thanks a lot.

[@fishpickles1377]

Very cool! Wish i had the hardware to run it!

[@PredictAnythingSoftware]

Thank you for the video using forge. Please make more video using forge since this is the only gui I can run SDLX model on my low end RTX 2060 6VRam PC.

[@pixaroma]

Sure, i also have an older computer with same video card and with forge i managed to get it to work, but even on my new rtx4090 seems to work better, so for a while i will do only forge, unless automatic 1111 ad something that forge can't do :)

[@robroufla]

Thanks ! Yes shame about the lack of settings for the SVD output path. It'd be great to have camera movement and prompt guidance like on Deforum but with consistence of SVD. Soon I'm sure ;)

[@bekosh248]

Great video! Do you know if forge or any other UI like this has the capability of inpainting a certain section of your image, so that only that inpainted portion gets animated?

[@pixaroma]

I don't know any, only some online platforms saw it has some motion brush, but didn't saw any to have in stable diffusion yet

[@rendercist]

Excellent video you got a new sub. Question, can you explain to me how the seed # system works on the SVD tab? It appears that incrementing the seed # changes the output each time, I get that, but why would you want to retain a particular seed #?

[@pixaroma]

You will get same result on same seed if you are using the same image, but if you change the image the animation will be different, you just use seed to get different animations but you dont actually have control on the animation, at least this is my understanding of it

[@lorenzodecarlo9125]

thank you for the video! I've not svd folder on webui > models. Why?

[@pixaroma]

it should be there since you installed forge ui, maybe you have other UI or something like A1111? not sure what to say

[@makadi86]

is this the best SVD or there are other recommended models we can try

[@pixaroma]

for stable video diffusion I didnt find a better one, stability ai released just one model for video, compared to the image model that released more then one

[@FantasyArtworkAI]

Mine creates a video in the folder: \Stable Diffusion Forge\webui\output\svd which is the same output folder where you have img2img and txt2img at.

[@pixaroma]

I didn't use it for a while but i think i put in settings all the paths to lead to the same folder

[@WizzardofOdds]

I seem to get a bit closer to animation using this. I have tried the animatediff but all I get is a still image. When I click generate with the SVD module I can see a progression bar but then I get Error. Is this because I did the one click download of Forge as that may be the issue with animatediff, or is it possible that I just don't have the right amount of vram. I have NVIDIA GeForce GTX 980 Ti

[@pixaroma]

I think you need more then 6gb of vram, usually rtx cards with 8gb or more work better. I saw another comment saying that 6bb gave an error

[@WizzardofOdds]

@@pixaroma Thanks, I guess I need an upgrade. Your videos are very helpful.

[@manolomaru]

✨👌😎🙂😎👍✨

[@MisterWealth]

How do websites like leonardo make it so it looks like the wings on a fly are flapping for example? I'm having a hard time generating a high quality video like that from svd its super grainy

[@pixaroma]

not sure what kind of models they are using, probably if you generate a lot of them some of them would have more interesting movements, other AI I saw it have brushes control that paint and tell what to move in the image, so have more control. Or like with SORA when will be reclassed with prompt that can tell what to do

[@anon3253]

I'm trying to utilize SVD on my GTX 1660 Ti, but it doesn't seem to be working. I'm encountering error messages.

[@pixaroma]

Maybe I don't have enough vram your video card , not sure, for me it worked with those settings

[@k_y_l_3]

Is anyone else having an issue with "RuntimeError: Conv3D is not supported on MPS"? Some people on github said it might be something to do with the pytorch version, but I think mine is the right version.

[@pixaroma]

I didnt got that error, but from what I found online that seems to be related to the macOS and apple processors. So what seems to be the problem with the error you're encountering is due to PyTorch's Metal Performance Shaders (MPS) backend not supporting the Conv3D operation on Apple Silicon (M1, M2, etc.). I am on windows so not sure what that does means, but maybe it has more sense to you, so probably pytorch doesnt support the apple proccesor how it should, yeah that suggest updating the pytorch but that will work only if they included that support for processor.

[@RenoRivsan]

does the checkpoitn matter??

[@pixaroma]

I think so, this one works with this settings but others might have other recommended settings

[@sircasino614]

So you can't have a prompt for "how" you want it to animate or move?

[@pixaroma]

no, is all based on the image, maybe they fix that in the future

[@wayneout]

I get the error message "attribute error; none type object has not attribute set manual cast" I upload an image from my computer. I don't know how to correct this error. Thank you

[@pixaroma]

Did you use exact same settings? Also make sure the image size is the same like in the video, if not resize it. There are some bugs when you are using a width and height that is not divisible with 64 so maybe that can fix it

[@caucho6.6.86]

how can add a prompt to video, if i want make specific videos?

[@pixaroma]

this one only works with images, so you can generate a text to image first then use that image to make the video, doesnt know text to directly video

[@AshishBista-tr7du]

Brother i got this error any help please: raise FFExecutableNotFoundError( ffmpy.FFExecutableNotFoundError: Executable 'ffprobe' not found

[@pixaroma]

Is possible to need to install ffmpeg , i didn't had that error yet

[@fixelheimer3726]

the snow overlays were not created with sd I guess?

[@pixaroma]

No, it is just a snow overlay video

[@idolgalaxy69]

can we do batch render?

[@pixaroma]

I didn't find an option for video, so I don't think it is possible or I didn't find it.

[@idolgalaxy69]

@@pixaromathanks~ you tutorial is great and clear~

[@kridadkool1319]

fam I wanna know about that A.i voice DOPE!! Vid

[@pixaroma]

I am using VoiceAir ai

[@Kevlord22]

Its good, since until i read this, i had no idea its was ai voice. pretty cool.

[@snatvb]

the worse thing that I can't really control it :( would be greate if I could add prompt, masks and etc, like in different SD tools

[@pixaroma]

Yeah I understand, hope they improve it in the future, now it is all random and needs a lot of tries to get something nice. But 2 years ago image generators were basic, so probably video get better, just needs time

[@snatvb]

@@pixaroma yep, I agree :)

[@lowserver2]

still ran out of memory with these exact settings on 8gb vram

[@pixaroma]

I don't have 8gb to test it, but online said it could work, sorry to hear it doesn't work :(

[@lowserver2]

sorry, i tried again after restarting forge and it did work. However, i cannot get good results yet. It mostly wants to do panning and the stuff outside the original pic becomes all distorted, so idk.@@pixaroma

[@pixaroma]

Try different seeds until you get one that works, unfortunately we dont have control, hope in future models they fix that

[@pixaroma]

@@lowserver2 try also using images that doesnt touch the edge, like is not cropped, so if you have a portrait make sure it has some space, then it can rotate that without distorting, if is on edge it tries to extend that and can fail

[@KevlarMike]

299 one time payment for topaz but at least it’s a onetime payment ❤

[@pixaroma]

I think I got it on black friday it was cheaper then :)

[@dziku2222]

Doesn't work for me, animated images are just being elongated or squished with some corruptions, instead of those cool animations you've showed. I use your dimensions and a model from link. Why?

[@pixaroma]

Not sure what to say maybe they changed something since i made the tutorial, if that happens with every image you use i cannot find an explanation

[@dziku2222]

@@pixaroma Sorry to bother, but it looks really interesting and I would like to get it running - maybe the cause of error is simple for someone far more experienced than me. I've discovered that it works normally when I'm using baseline realistic visions model that comes together with ForgeUI - but not when I'm using something generated with old SD1.5 models like abyssorangemix

[@retikulum]

Such a piece of crap extension. I create one Video, VRAM gehts filled, Video is finished, VRAM stays full -> OOM when trying to create the next video. So, restarting SD after every video creation. How stupid.

[@pixaroma]

It needs a lot of vram or you can't do much with it

[@retikulum]

@@pixaroma Huh? No, like I said: 1st video works, 2nd video OOM because the vram is still full from the first video. It doesn't get flushed. pytorch keeps the vram reserved.

[@pixaroma]

Yeah is possible to not work how it should with ram, but if you have more it never gets full so it still works, never crashed on 24vram, but still seems to be an old version and didn't saw a new one that work for stable diffusion so i keep using that one, i am waiting for Sora or alternatives

[@JarppaGuru]

3:08 yes seed like million variable. some complete grap. tells what AI actually do. programmed todo. not any intelligence. it will not create rabbit unless trained data has rabbit and it will be same rabbit for those prompt words. it works good with this robot bcos training data have many images from this robot. it did not work good for picture of man and face swapped myself. background move if find seed but "me" not change at all LOL got so bored first attempt worked but rest did not lol lol all that waiting to get grap! cant even choose render frame 7 without make video or render mulltip images using different seeds so can choose. seed is like motion from 1 trained clip. it will do exact that if your image match(trained todo no AI) seed 1 could me pan left seed 2 could be pan right ..etc what we learned? AI result need be checked. dont make skynet and plug it to red button(it will push red button if it programmed todo it) but if human check result and human push red button not AI. then we not have skynet. just AI (tool)(automated instructions like i say)

[@pixaroma]

Well in this case since is based on a image, the image is the variable, you can have infinite unique images for input. And yeah is not a Ai that we see in the movie is a trained model that do what is trained and knows only that for now.

[@TheMaxvin]

SVD is so boring, it`s background light motion basically.

[@pixaroma]

Yeah, definitely needs more work

[@ranjithgaddhe9818]

SVD has better orbit camera motion I have tried Other models not even making tracking well .. but SVD still need more training for better results

[@Ollegruss_Music]

Thanks!