Geometric Intuition for Training Neural Networks

Leo Dirac (@leopd) gives a geometric intuition for what happens when you train a deep learning neural network. Starting with a physics analogy for how SGD works, and describing the shape of neural network loss surfaces.
This talk was recorded live on 12 Nov 2019 as part of the Seattle Applied Deep Learning (sea-adl.org) series.
References from the talk:
Loss Surfaces of Multilayer networks arxiv.org/pdf/...
Sharp minima papers:
-Modern take arxiv.org/abs/...
-Hochreiter, Schmidhuber 1997 www.bioinf.jku....
SGD converges to limit cycles: arxiv.org/pdf/...
Entropy-SGD: arxiv.org/abs/...
Parle: arxiv.org/abs/...
FGE: arxiv.org/abs/...
SWA: arxiv.org/pdf/...
SWA implementation in pytorch: pytorch.org/bl...

[@susmitislam1910]

For those who are wondering, yes, he's the grandson of the late great Paul Dirac.

[@miguelduqueb7065]

Such insights so easily explained denote a deep understanding of the topic and great teaching skills. I am eager to see more lectures or talks by this author. Thanks.

[@oxfordsculler8013]

Great video. Why no more? These are very insightful.

[@ramkitty]

This is a great lecture that ends at wolframs argument for quantum physics and relativity and what I think is manifest as orch or type contiousness through Penrose twistor collapse

[@matthewhuang7857]

Thanks for the speech Leo! I'm now a couple of months into ML and this level of articulation really helped a lot. I know this is probably a rookie mistake in this context but often when it's hard for my model to converge, I thought it's probably because it reaches a 'local minima'. My practice is often significantly bumping up the learning rate to hopefully let the model to kinda leap over and get to a point where it can re-converge. According to what you said, there are evidences conclusively proving there's no local minima in loss functions. I'm wondering which specific papers you were talking about. regards, Matt

[@uwe_sterr]

hi leo, thanks for this very impressing way of making somewhat complicated concepts so easy to understand with simple but well structured visualisations.

[@linminhtoo]

very nice (and certainly mindblowing) video, but according to kzbin.info/www/bejne/bWnZommhnNiHl5o, that complicated loss landscope at 13:51 is not actually a ResNet but a VGG. The ResNet one looks a lot smoother due to the residual skip connections

[@LeoDirac]

Thanks for the kind words. The creators of that diagram called it a "ResNet" - see the first page of the referenced paper arxiv.org/pdf/1712.09913.pdf . Skip connections make the loss surface smoothER, but remember that these surfaces have millions of dimensions. There are zillions of ways to visualize them in 2 or 3 dimensions, and every view discards tons of information. It's totally reasonable to expect that one view would look smooth and another very lumpy, for the same surface. TBH I don't know exactly what the authors of this paper did - they refer to "skip connections" a lot, and talk about resnets with and without them. I'm not sure if they mean "residuals" when they say "skip connections" but I'm not sure I'd call a resnet without RESiduals a RESnet myself. If you remove the residuals it's architecturally a lot closer to a traditional CNN like VGG / AlexNet / LeNet and not what I would call a ResNet at all.

[@katiefaery]

He’s a great speaker. Really well explained. Thanks for sharing.

[@PD-vt9fe]

Thank you so much for this excellent talk.

[@MrArihar]

Really useful resource with intuitively understandable explanations! Thanks a lot!

[@RobertElliotPahel-Short]

This is such a great talk! Keep it up my dude!!

[@matthewtang1489]

This is so coooooollll!!!!!!!

[@elclay]

please the slides sir

[@hanyanglee9018]

17:00 is all you need.

[@berargumen2390]

This video lead me to my "aha" moment, thanks

[@bluemamba5317]

Was it the pink shirt, or the green belt?

[@abhijeetvyas7365]

Dude, awesome!

[@srijeetful]

nice one