Lesson 16: Deep Learning Foundations to Stable Diffusion

(All lesson resources are available at course.fast.ai.) In Lesson 16, we dive into building a flexible training framework called the learner. We start with a basic callbacks Learner, which is an intermediate step towards the flexible learner. We introduce callbacks, which are functions or classes called at specific points during the training process, and demonstrate the creation of a simple callback. We also introduce the concept of CancelFitException, CancelEpochException, and CancelBatchException.
Next, we explore metrics and create a MetricsCB callback to print out metrics during training. We introduce the torcheval library and create a DeviceCB callback to handle moving the model and data to the appropriate device. We refactor the code using a context manager to simplify the code and make it easier to maintain and add callbacks in the future.
We then focus on looking inside the models to diagnose and fix problems during training. We introduce a set_seed function and train a model with a high learning rate of 0.6 to test the stability of the training. Finally, we discuss analyzing the training process by looking at the mean and standard deviation of each layer's activations, using PyTorch hooks and creating histograms of the activations.
0:00:00 - The Learner
0:02:22 - Basic Callback Learner
0:07:57 - Exceptions
0:08:55 - Train with Callbacks
0:12:15 - Metrics class: accuracy and loss
0:14:54 - Device Callback
0:17:28 - Metrics Callback
0:24:41 - Flexible Learner: @contextmanager
0:31:03 - Flexible Learner: Train Callback
0:32:34 - Flexible Learner: Progress Callback (fastprogress)
0:37:31 - TrainingLearner subclass, adding momentum
0:43:37 - Learning Rate Finder Callback
0:49:12 - Learning Rate scheduler
0:53:56 - Notebook 10
0:54:58 - set_seed function
0:55:36 - Fashion-MNIST Baseline
0:57:37 1 - Look inside the Model
1:02:50 - PyThorch hooks
1:08:52 - Hooks class / context managers
1:12:17 - Dummy context manager, Dummy list
1:14:45 - Colorful Dimension: histogram
Timestamps thanks to fmussari

[@JohnSmith-he5xg]

First, wanted to say I really appreciate you putting all this content out. I'm incredibly relieved that at 4:30 you've broken the run_cbs() function out into multiple lines. If I had to offer a criticism of the coding so far it would be that you really emphasize being terse. This combined with using a lot of Python specific language features can make things tough to follow. Starting with simple, but verbose code might be better from a learning perspective, then subsequently re-writing. Again, really like the content!

[@grekiki]

At 1:23:20 the only reason things seem to be improving, is that you are plotting activations of validation batches.

[@ekbastu]

GOD mode activated