Stable Diffusion - Better Fine Details with a new VAE! (Variational Autoencoder)

You can now fine-tune your stable diffusion local installation with an updated variational autoencoder (VAE for short). The folks from StabilityAI released this a few days after the 1.5 update was released from their business partner, so this is something you should consider adding to your installation to make things as fine as possible. This should help a bit with things like fingers, eyes, faces, and that annoying magenta drift that happens when doing loop img2img animations.
REMEMBER, ALWAYS DOWNLOAD FROM OFFICIAL SOURCES!
You can download the VAE from the official site.
huggingface.co/stabilityai/sd...
This is the line I added:
set COMMANDLINE_ARGS=--vae-path "models\Stable-diffusion\vae-ft-mse-840000-ema-pruned.pt"
If you don't have a local installation, check out this video: • Stable Diffusion 1.5 -...
Is your PC a potato? Try out Run Pod for way less than a dollar an hour (affiliate link):
tinyurl.com/ydu6rkpk
Check out Dream Studio here: beta.dreamstudio.ai/
Join my Discord server, where we create ideas together!
/ discord
New! Become a member to get exclusive access to perks!
/ @sedetweiler
Enjoy this video? Consider buying me a coffee! ko-fi.com/sedetweiler
Twitter: / sedetweiler
Instagram: / sedetweiler

[@frankenmizer828]

That was the clearest explanation of VAE and how to use it I've seen. Thanks, Scott :)

[@sedetweiler]

Glad it was helpful! Thank you for being a channel member!

[@rickwade9218]

All of your videos are informative and easy to understand. Thank you sir! You're doing God's work!

[@HadesTimer]

Thanks, I could go through all this myself. But you are saving me a ton of time. Keep up the great work!

[@sedetweiler]

Happy to help!

[@coda514]

Works like a charm. Every little bit helps. Thanks for sharing your knowledge.

[@sedetweiler]

Happy to help!

[@universe6735]

Thank you professor. Sending you love and respect from India.

[@sedetweiler]

Thank you, sir!

[@TolgaOzisik]

Thank you very much, super clean and to the point!

[@OriBengal]

Just a reminder - many Lora models remind you to disable VAE when using Lora models, or bad things happen.

[@clenzen9930]

Your example really helped! I had followed another guide, but it wasn't specific enough and I didn't get to work that time. Thanks.

[@sedetweiler]

Hope it helped you get them installed okay!

[@wsippel]

Worth noting that the VAE specified in the command line argument is only loaded for checkpoints that don't have manually assigned VAEs. WD for example has a bespoke VAE (kl-f8-anime2), so you might want to use that one instead.

[@sedetweiler]

Great point! I will mention that in my next update. Cheers!

[@KingQuetzal]

Thank you! Couldn't find info about this, thought it was a different type of hypernetwork.

[@sedetweiler]

Nope, just a fine-tuning.

[@2PeteShakur]

another well documented doc from scott! ;)

[@sedetweiler]

glad you enjoyed it!

[@MysteryFinery]

nice to see that you post after you've studied the subject.

[@sedetweiler]

I would hate to make a video from a point of ignorance. I prefer to wait and understand something, it's just easier for everyone. Cheers!

[@samko127]

Amazing tutorial, you earned my sub👍

[@impactframes]

Awesome even if the difference isn't huge, it's still a improvement

[@sedetweiler]

Yup!

[@morizanova]

thanks for video . I`m noticing your link about runpod and wonder would you someday making video about that . Especially for SD works

[@sedetweiler]

I will indeed. It is what I use when I want to do something faster that my local machine cannot handle.

[@AscendantStoic]

Been hearing about VAEs for a while but didn't bother checking them out, was more concerned about getting a good outpainting WebUI, and finally managed to setup SD infinity and get it to work flawlessly with SD1.5 Inpainting model, been wishing to do that since I saw DALL-E2 infinite canvas weeks ago. But now that this is out of the way maybe it's time to check VAEs, so far I know of two, the one you mentioned and another one fine-tuned for the Waifu Diffusion model.

[@sedetweiler]

I am sure more will pop-up, but for now those are the two I would probably focus on.

[@NapalmCandy]

Can you explain how you did the outpainting setup?

[@raid5098]

Ughh I need a new pc so bad, can't wait to use this app! Thanks a ton for all the tutorials, I just started getting into the AI art space. So much fun!

[@sedetweiler]

Just use this link and rent a machine for like .20 an hour! It's a lot cheaper than buying a machine, and some of them have like 48gb of VRAM and are like .50 an hour! runpod.io?ref=i0wb238m

[@FilmFactry]

Thanks, Scott!

[@sedetweiler]

You're welcome!

[@AIWarper]

In the latest update of Stable I now have a folder in my Models directory titled "VAE".... any idea if placing the VAEs in this folder works without this?

[@FinnieTheGhoul]

Thanks for this amazing guide of yours. After watching it, i started to search Huggingface for vaes of the models i currently use. Most of them actually offer their own vae to download. But i figured, some of them have entirely different file types, like .bin or .safetensors. After googling for a while, i was sadly unable to find out if i need to convert them somehow or if the file type doesn't really matter. Or are these vaes actually inlcuded into the model from the get go? I'm kind of stuck on this for now, so any help would be appreciated.

[@LouisGedo]

Hi Scott 👋 You are super details oriented 👏. Is there a tool / tool set within SD WebUI which does a good job of taking black & white pen illustrations and can generates them into intricately colorized images? I haven't tried the img to img tool set yet but I thought that you being an artist may have some insight into such a possibility in SD. Thank you.

[@sedetweiler]

I have not tried that, but you are probably on the right track.

[@joshpolman201]

Thanks mate :)

[@sedetweiler]

Sure!

[@cyrusbastankhah576]

Checks out, thank you!

[@lithium534]

Could you show how you did the colour difference part? Very useful when using different models. Thanks for the video

[@sedetweiler]

Sure! Just duplicate the layer and set the blending mode to "difference."

[@infographie]

Excellent

[@NADIMATION]

that's awesome , would you please explain how to add it to colab ?

[@alfonsojarago]

I love you Scott. You're my homeboy.

[@sedetweiler]

Thank ya, brother from another mother!

[@LISTEDGames]

In the video you said there's two dashes but in the description there's only one dash before VAE

[@sedetweiler]

I might have misspoke. I put the line in the video description.

[@clayman0320]

where do you get the model above the one you added?

[@3dtechnik]

Yeah, I wondered the same and could not find it under that name with that file type.

[@pshermanart]

Thanks for this. I noticed that at some point you changed the file name model.ckpt to something else, but I see you're not getting an error. Where do you set the default ckpt?

[@sedetweiler]

Initially it's just model, and after it's installed you can use any of them. this has also changed a bit already. 😂

[@pshermanart]

@@sedetweiler I'm not sure you understood my question.

[@RhetoricalTraveling]

@@pshermanart u no longer have to have a default named model, as the app can utilize multiple different ckpts, you select them in the webui, top left corner

[@pshermanart]

@@RhetoricalTraveling Thanks, yeah, I found that I no longer received the error message.

[@pepevillalobos9327]

thanks

[@sedetweiler]

yup!

[@flonixcorn]

Great video, your runpod affiliate link isnt pasted in correctly btw

[@sedetweiler]

Oh? I will check on that. Thank you!

[@sedetweiler]

Thank you!

[@darcybrown7369]

H Scott, trying to follow along here and finally got the stable diffusion local installation working A ok - but in this one I went to the SD models folder - found I didn't have the pruned learning ckpt - where was I supposed to find that? When should I have it installed - I don't have most of the other ckpts either btw - just the model one and this one for the vae .

[@sedetweiler]

You don't need that one, as any of them will work. This VAE will work on all of those models.

[@jeanhadrien]

Is it only for photos or does it also offer improvements on drawings/illustration, etc?

[@sedetweiler]

I would think it would help a lot in those areas.

[@christophermoonlightproduction]

Okay, question: When I go to edit my webui-user there's another line that loads my xformers. What do I do with that? I tried having it load both and it couldn't handle it. Thanks.

[@FilmFactry]

Do you know anything about the LSDR Upscaler? It is supposed to be very high quality but take a LONG time? I google and only read some Reddit about peoples issues.

[@ATypicalDayHere]

So are VAEs REQUIRED to run other models, just so you know, I get my models from Civitai

[@hatuey6326]

question plz : when you have already set COMMANDLINE_ARGS=--precision full --no-half --medvram, how can you do ? you put a seconde commandline args?

[@sedetweiler]

All in the same line

[@oswynfaux]

I thought I installed everything correctly for the Automatic 1111 version but when I try the GIT PULL command I just get Fatal: not a git repository

[@jamesalxl3636]

Hey is possible you can make a video on how to install [JoePenna/Dreambooth-Stable-Diffusion] Version locally? I saw that's the one Corridor Used but I can't find any tutorials on it...

[@Elcodigodebarras]

Each grain then make a big bag...Thanks!

[@hardkur]

can U please make a tutorial for local install of dreambooth-gui ? its tricky for a noob and im sure when U explain it it will get much much easier :)

[@francescameloni7859]

sorry, this works also with sd 2.0? I've some problems to install it... thanks

[@digitalflick]

i have --listen as a command line argument. How would i add this as well?

[@i_zacca_2314]

just put --listen at the end

[@MalihaSeraphim]

um.. thanks baby

[@lakislambrianides7619]

i followed your video of installation and all is working fine. but now the file i have is webui-user without the .bat so i cant go to the edit the note since i don't see it here. what's going on here?

[@sedetweiler]

Do you have the file extensions turned off?

[@lakislambrianides7619]

@@sedetweiler got it thanks a million

[@sedetweiler]

Awesome!

[@MADVILLAIN669]

Great video, I need some help though! For some reason VAEs dont seem to apply to one image I'm trying to upscale in img2img, it's still grey and desaturated, it seems to work fine in text2img, but I just cant get it to work in img2img, this is so strange. Anybody has a clue?

[@sedetweiler]

Hmm, not sure. So many variables to consider. Done models don't do as well as others, so perhaps switch to a different model and see if you get the same color issue?

[@MADVILLAIN669]

@@sedetweiler I've used 10 different models, and 5 different vaes. Same effect in im2img as if I had no vaes at all. I'm not even putting any prompts, or tweaking any settings, I just use any picture I've generated and try to apply a vae and it seems to do something in the preview, but no result in the end. I've been struggling with this issue for a few days now. Could it be something to do with models/loras/extensions I've installed? I've tried emptying out all the vae files, and putting them back, no result.

[@lovefestcentroamerica6053]

Someone else having this issue, when i try to generate in deforum only a few times works and only in short videos, please i need help to fix it x.x Error: 'A tensor with all NaNs was produced in Unet. This could be either because there's not enough precision to represent the picture, or because your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check.'. My setup: ryzen 9 6900hx, rtx3060 6g, ram 16g

[@eduacademia]

ckpt or pt ? can we rename ckpt to pt , i dont undrstand

[@EoCA777]

I got an "unrecognized arguments" error

[@LouisGedo]

👋

[@sedetweiler]

🙂

[@oxymoron02]

Just one thing; In the tutorial video you said, and demonstrated on-screen, to rename the model to just "model", but now in this video yours is named, "model v1-5". What gives?

[@sedetweiler]

That was for the initial install. It used to throw an error if it wasn't named that, but that has changed. This all changes so fast!

[@clayman0320]

the v1-5-pruned etc

[@sedetweiler]

You can use any model and this will work.

[@moaa042]

Anyone else get CUDA out of memory errors trying to use this?

[@louie8362]

what will happen to that command for --xformers 😂😂 I put rem on it, some other tutorial says it makes rendering 110% faster, but now that its off... will I still be able to doodle a poodle🤣

[@Flamonarchest.]

This cold with

[@tranquilmogambo]

Or, you can simply use easy diffusion!

[@aegisgfx]

Did this guy die??

[@sedetweiler]

😂 naa. Actually planning on a video later today. Getting back to it.

[@EmanuelLFerreira]

thanks