Lesson 10: Deep Learning Foundations to Stable Diffusion, 2022

(All lesson resources are available at course.fast.ai.) This lesson creates a complete Diffusers pipeline from the underlying components: the VAE, unet, scheduler, and tokeniser. By putting them together manually, this gives you the flexibility to fully customise every aspect of the inference process.
We also discuss three important new papers that have been released in the last week, which improve inference performance by over 10x, and allow any photo to be “edited” by just describing what the new picture should show.
The second half of the lesson begins the “from the foundations” stage of the course, developing a basic matrix class and random number generator from scratch, as well as discussing the use of iterators in Python.
You can discuss this lesson, and access links to all notebooks and resources from it, at this forum topic: forums.fast.ai/t/lesson-10-pa...
Additional Links:
- Progressive Distillation for Fast Sampling of Diffusion Models - arxiv.org/abs/2202.00512
- On Distillation of Guided Diffusion Models - arxiv.org/abs/2210.03142
- Imagic: Text-Based Real Image Editing with Diffusion Models - arxiv.org/abs/2210.09276
0:00 - Introduction
0:35 - Showing student’s work over the past week.
6:04 - Recap Lesson 9
12:55 - Explaining “Progressive Distillation for Fast Sampling of Diffusion Models” & “On Distillation of Guided Diffusion Models”
26:53 - Explaining “Imagic: Text-Based Real Image Editing with Diffusion Models”
33:53 - Stable diffusion pipeline code walkthrough
41:19 - Scaling random noise to ensure variance
50:21 - Recommended homework for the week
53:42 - What are the foundations of stable diffusion? Notebook deep dive
1:06:30 - Numpy arrays and PyTorch Tensors from scratch
1:28:28 - History of tensor programming
1:37:00 - Random numbers from scratch
1:42:41 - Important tip on random numbers via process forking
Thanks to fmussari for the transcript, and to Raymond-Wu (on forums.fast.ai) for the timestamps.

[@fabriai]

Jeremy Howard, what you're doing to help people understand these models is priceless. Thank you, thank you, thank you. God bless you.

[@tokosby9700]

Nnnn

[@sunderrajan6172]

Great Lectures as always Jeremy! Its been a while since Lesson 10, I am eagerly waiting for remaining lessons.Thanks!

[@jordankuzmanovik5297]

Any news?

[@KellyNicholes]

I haven't delved into the juicy bits of python for over ten years now, and watching this was making me feel "home" again. I'm so thirsty for more! Added bonus: You learn how to create your own Stable Diffusion model from scratch. I can hardly wait.

[@sunderrajan6172]

Remaining lessons have been posted in the last two weeks. I am really excited and the lessons are awesome as always from Jeremy.

[@timandersen8030]

Thanks for the video lectures! It's really cool that you start out introducing an awesome application first (i.e. Stable Diffusion) and how it works before teaching the foundation and starting back from there.

[@howardjeremyp]

Glad you like them!

[@tsenguun]

Looking forward to the next lesson

[@ItzGanked]

jupyter also has documentation for python and other libraries inside the notebook if you go to help python reference. Keeps you inside of the notebook instead of searching the web and will provide docs for the version of python you are using.

[@jessedeng3300]

For the clip image text embeddings would it be possible to use a pretrained language model to first get the text embeddings and then use those as the targets for image embeddings in a traditional optimisation loop? Also thank you for these lessons I am deeply grateful

[@VaIhalIa]

Love the videos.

[@djcardwell]

Jeremy, where is the best place to stay updated on papers? I'd like to read all the trending papers as they come out.

[@tumadrep00]

Could a teacher/student approach be used in the imagic process to generate video?

[@CoolDude911]

It could be that the Markov way of modelling the problem stops you getting into local optimums or images which are not really part of the target distribution. The model has to "change course" between iterations. I remember generating a picture of a seal with a head coming out of each end of the seal. At some point the model was not sure about where the head of the seal was but had to commit to one decision or another.

[@vivekakaviv]

Thank you.

[@heejuneAhn]

Do we use same Random noise in each step or sampling every time in each step.

[@tsenguun]

Still waiting for the full course

[@bilalch83]

Sensei Jeremy Howard

[@satirthapaulshyam7769]

41:00

[@mattkkma]

For those looking for Jonathan Whitaker paper walkthrough kzbin.info/www/bejne/kInYfGl_h6-fpM0