How does Stable Diffusion work? - Latent Diffusion Models EXPLAINED

[@AICoffeeBreak]

8:39 What I meant to say, is VQ-VAE, not VQGAN! Thanks to Luis Cunha for spotting this!

[@j2325138]

Thanks!

[@AICoffeeBreak]

Omg, thanks! 😇

[@RemiStardust]

I'm going to have to watch this twice. So much dense info!

[@BjoernBu]

Awesome explanation! Short & to the point, with super useful visualizations throughout all parts. Thank you very much for the video.

[@CK-vc3kk]

I have to comment that your Erratum is amazing. Thank you. It clearly shows the greatest challenge in ML research, which is the *smallest* seeming innocent detail can in fact be the *entire reason a model doesn't work* (or at least, not work anywhere near how it should).

[@chyldstudios]

Perfect timing, I was just reading about latent diffusion models!

[@AICoffeeBreak]

Yeasss! ⏲ 😅

[@Micetticat]

Latent space is where all the magic happens!

[@AICoffeeBreak]

🪄

[@DerPylz]

Welcome back from your deserved holiday break! 😊

[@AICoffeeBreak]

Thank you! 😁

[@CristianYones]

Best explanation so far! And I've seen a lot of videos...

[@TenzinT]

Great, best explanation of Diffusion Models so far!

[@obseraft]

You accent is great, thanks for your explanation.

[@ManuXit]

super happy I found your awesome channel

[@AICoffeeBreak]

Glad you found us!

[@niofer7247]

Always love the clarity of your explanations. Keep it up!!!

[@ShivamMamgain]

Great explanations. Ty

[@paulcurry8383]

I’m curious how inpainting works with these models. Does the removed part of the image get filled with noise and then copied to the output with only the noise prediction from the removed part used to de-noise the removed part?

[@blackrack2008]

There are options to keep the removed part of the image and just reinterpret it with a given strength which works like img2img or to just replace it by noise

[@mpjstuff]

I still feel like we are missing a bit of understanding. In your last image that you end by using img2img (however, this applies to the entire process of recognition). The baby ice cream robot has a balloon that is floating above it's head in the drawing, with a loose string. In the output image, it's echoing the daddy balloon and pulled by the wind. Now, how does the natural language search with the context of "balloon" now that blob is a balloon and should that since it floats, both would be pulled by the same wind -- is that in a prompt? Even so, the natural language engine -- how does that get the concept of "pulled by the wind" when, some google search would present results that would create a huge mess? Or is that a writing prompt as well? Even still, how does that incorporate into the image conceptually? If I'm extrapolating from the artists drawings, I suppose there is some idea of a bipedal and symmetrical structure, and "beauty" is perhaps going to result in a certain relationship in size and structure once a pattern emerges, but, I don't see how random images in a collection coalesce into being "recognized" from the noise without a lot of mutation? Is it making inferences from each step and jumping forward and then culling the branches that lead to "nonsensical or monstrous" arrangements via a learning engine? Maybe this will be explained if I spend another few hours on videos ;-) -- anyway, thank you Letitia for your wonderful work.

[@felipe_ai]

I love your videos, can't believe I didn't find you sooner lol, thank you for all the content :)

[@AICoffeeBreak]

Glad we reached you. :)

[@satpalsinghrathore2665]

Easily explained! Kudos.

[@deeplearner2634]

Man i was also misunderstanding the concept too. Its sounds just insane to even think of trying to predict the *entire* noise from a mid product

[@WhatsAI]

Amazing video and explanation as always!

[@AICoffeeBreak]

Can only recommend you video on this! :)

[@user__214]

Thank you for this! Best simple explanation of stable diffusion that I've come across.

[@alphacat4927]

I was literally mentally thinking of sitting in a hammock when you said that.

[@AICoffeeBreak]

🏖️🏕️

[@kajika135bis]

Amazing explanations, this model looks very accessible, I hope it is not filled with undocumented tricks like I experienced with Soft Agent Critic. Thank you again, this video is the last straw for me to go and run this incredible network.

[@maryguty1705]

just fabulously done! wanderful!

[@AICoffeeBreak]

[@Neptutron]

Yay, you're back! =D Also this is perfect timing for me lol - I was planning on trying out a project with this model

[@AICoffeeBreak]

Is this project idea anything you want to share? :) No worries if you would rather not talk about it, especially if you plan to publish a paper about it and do not want to be scooped. 😅

[@Neptutron]

@@AICoffeeBreak Hey:) So I'm a PhD student, and I was thinking about making a paper of it if it works, so I won't share the entire idea here because you have a good point. But the gist is that I want to make 3d models from Stable Diffusion's outputs by combining it with a differentiable rendering backend

[@AICoffeeBreak]

@@Neptutron 🤞hope that you succeed!

[@Neptutron]

@@AICoffeeBreak Thank you Letitia!!

[@Neptutron]

Hi Letitia! I sent you an email with a small update:)

[@coderaven1107]

LOVE your content!!

[@L33TNINJA51]

So are any transformers used in Stable diffusion?

[@L33TNINJA51]

Oh it uses transformers (cross attention) at every step. The paper cleared it up for me.

[@harumambaru]

If you are really using sponsor to transcribe your videos it is doing amazing job compared to standard youtube autogenerated text. Punctuation signs are so life changing! I saw exclamation mark in subtitles for the first time of 15 years of using youtube!!!

[@AICoffeeBreak]

The baked in subtitles in the sponsor spot are transcribed. The rest is not. I copy paste ma script into KZbin for the captions in general.

[@HalkerVeil]

They used to say the camera would ruin art too.

[@gunnar_langemark]

I really enjoyed your video. Keep up the good work, please. :)

[@Harduex]

Very good and easy to understand explanation! Thank you for your content and keep up with the good work 🙌

[@XuhanQian]

love ur work! great explanation!

[@HemangJoshi]

Miss coffee bean is so sweet... ❤️❤️❤️❤️

[@vikaspoddar001]

Everything is happening in great sync 🙂 After watching Andrej Karpathy's stable diffusion video now, getting an explanation by Miss Coffee bean herself 🎉🎉🎉

[@AICoffeeBreak]

Haha, I feel like an imposter just being in the same sentence with Karpathy. 😅

[@maryguty1705]

Have you ever done videos on all the weird and interesting tricks apply in DL and AI to achieve quite intriguing results?

[@AkairoAoihonoSama]

Great video! I just found your channel and is fantastic ❤ May you please explain why AI images often have a strange wavy pattern that doesn't represent anything and looks like shapes melting into other shapes? Is it correlated to the noise added?

[@AICoffeeBreak]

Hi, wonderful question! It can have something to do with noisy training data, but mostly, it is because the process of interpolating within the latent space of generative models can sometimes produce smooth transitions that appear as melting shapes. This is because in the latent space, you cramp many dimensions into few ones, but still have places in the latent space that was unsampled during training. And small changes in the latent vector inherently lead to continuous changes in the output image, but sometimes they are strange, nonsensical melting shapes.

[@Handelsbilanzdefizit]

Are skip-connections and residual-connections the same?

[@johnclapperton8211]

What constitutes a valid starting image? Is it arbitrary? If I ask for "a frog on a carrot" is the starting image as likely to be a spaceship or hamburger as either a frog or carrot? What do we mean by an "image"? It's an arrangement of pixels, but by what criteria do humans call it an image - just some spatial gradients which human brains have learned to recognise as meaningful in some way?

[@pvlr1788]

So the sentence encoder and image encoder-decoder are pre-trained and non-trainable during the training process of the LDM?

[@Phenix66]

yup - otherwise, things would be super complex again if you'd take the full image. That's what makes it elegant. And training the text encoder along with the diffusion model is also not a great idea (Imagen form google basically just leaves that out and things get better).

[@idealintelligence7009]

Stable Diffusion Kaggle notebook: www.kaggle.com/code/givkashi/stable-diffusion-model-text-to-image-generation

[@romekin]

this dog is so cute!!

[@ginebro1930]

It's funny how Dalle is made by Open IA and isn't open source.

[@robbiero368]

Apparently they've invented this new fangled thing called a "camera" all artists are now out of a job. I don't think you can really call yourself an artist if you can't imagine the creative explosion this sort of thing can promote

[@StuninRub]

when the camera became popular, a lot of painters did indeed lose their jobs.

[@robbiero368]

@@StuninRub I would argue the benefit was that it eventually allowed the vast majority of the world to create images and not a handful of elites who could afford to pay for a painted portrait. Plus you can still have your portrait painted if you have the money. Plus you can send your photo to an artist and have them paint your portrait more cheaply. So there are probably even more portrait artists now in total than before.

[@StuninRub]

@@robbiero368 A lot less portrait artist. The camera put a lot of painters out of business and AI art will do the same to digital artists.

[@robbiero368]

@@StuninRub can you supply some number's?

[@joedalton77]

Is the time T an input to the network?

[@AICoffeeBreak]

The image at diffusion step T, yes. So with a certain amount of noise.

[@TimScarfe]

Amazeballs

[@AICoffeeBreak]

😅 Thank, Tim!

[@HalkerVeil]

Plot twist. This video and the cute girl talking is AI generated...

[@matejcigale8840]

Does anybody understand what is the difference between "Disco Diffusion" and "Stable Diffusion" I cant seem to find anything useful. Both seem to steam from the code of the same person, if I understand correctly. Is it just the models?

[@maltimoto]

So how can an AI image generator two objects, for example a man holding an apple in his hand? The AI must (!) have the 3D model in order to render the scene correctly. But how does the AI get the 3D model? This is not explained in any of the videos on KZbin.

[@AICoffeeBreak]

It does not have a 3D model in the sense we have it. It learned from hundreds of millions of 2D images how apple pixels and man pixel "rhyme" with each other, which also includes a sometimes accurate, but sometimes inaccurate fit of an inferred 3D model. It's exactly how ChatGPT talks about physics and reality by just having had read text but not seen any pictures and never touched anything.

[@anonymous-vf2pr]

why are we adding noise instead of directly taking noisey image.

[@REDINKmysteries]

i like turtles!!!!!

[@ellanleicher8657]

So how does a diffusion network generate a new image? From what I understand it tries to reconstruct the noisy image

[@Epsellis]

No, Artists aren't complaining because it's easy. Just go watch any respected art youtuber

[@AICoffeeBreak]

I'm interested. Could you point me to a specific video?

[@maxicornejo9675]

​@@AICoffeeBreak This guy explains it very briefly kzbin.info/www/bejne/gqKymX2OhKajfas Basically, it's about exploitation and AI ethics, something that every tech fetishists doesn't care about. Sadly.

[@crapadopalese]

Yeah you got it wrong again in explaining what noise is being predicted. But I appreciate you not getting flustered by not getting it right and continuing to speak with complete confidence about things you don't have a grasp on.

[@AICoffeeBreak]

Thanks for sharing your opinion! If you would tell us what is wrong, we could point it out (for example by pinning your comment).

[@crapadopalese]

@@AICoffeeBreak thanks for your suggestion but I think that it's not an iterative process. Every video you post where you get it wrong disseminates misinformation. Do thorough research and be confident that you understand what you're explaining!

[@maxicornejo9675]

@@crapadopalese what a helpful comment, thank you.

[@crapadopalese]

@@maxicornejo9675 I think criticizing people that provide misinformation is important.

[@maxicornejo9675]

@@crapadopalese Constructive criticism is more valuable than just criticism. >Constructive criticism is a feedback method that offers specific, actionable recommendations.

[@81neuron]

great video! just a bit too much cynicism

[@chengong388]

I tried it, and it totally sucks, the results are often messy, with different features smeared together, very much like those earlier AI "dream" images. It's just nowhere near as natural as Dall-E, and I can't help but feel that an optimized algorithm shouldn't require so much VRAM for a 512x512 image.

[@imapimplykindapimp]

Hard to optimise a neural network, not like a regular human made algorithm

[@ДмитроПрищепа-д3я]

Have you added a negative prompt. It's quite important to tell the network what not to do. As for the vram usage, yeah, it has to fit the entire model in memory to run inference on it and the model is massive. There's some work being done to optimize it however. For one, the model you were using is likely 2 times smaller than the original already due to pruning. Stability AI is currently working on distilling the model down further to speed things up even more.

[@Schinken_]

First you make text to image and than do image to image

[@RutujaKokate-u5h]

i am so confused

[@JDechnics]

I believe you mean Imagen and not Glide.