Lecture 7 - Deep Learning Foundations: Neural Tangent Kernels

[@TheAIEpiphany]

Cool video thanks! 00:00:00 Intro: linear regression 00:23:55 NTKs start here 01:01:33 link between NNs and ODEs (ordinary differential equations)

[@debadeepta]

Really nice lecture! I was looking to quickly learn NTKs before diving deep into the original papers and this really helped.

[@zl7460]

+1. Most well-explained DL lecture I've seen for a long time

[@StratosFair]

Incredibly clear lecture, allowed me to fill the gaps in my understanding of NTK. Thank you professor !

[@dv019]

Great video, thank you! To the student asking about Kernels: the word is overloaded. It is used in linear algebra to mean the set of all vectors mapped to 0 by a linear transformation. Sometimes Green's functions in PDEs are called integral kernels. In general a kernel is "the central or most important part of something". I don't like how overloaded the word is either, but c'est la vie.

[@weisenjiang9179]

great intro to NTK, benefit me a lot

[@mstislavmaslennikov326]

The lecturer is imho doing a great job explaining difficult material!

[@AyushSharma-ie7tj]

Really nice lecture with a very even pace. Thank you for sharing.

[@MetaOptimizer]

41:07 Do we consider the large width of parameter (m) in empirical observation as an extremely large network such as GPT3? In other words, could I interpret the meaning of "the width of parameters" as "the number of trainable parameters"? Thank for your valuable lecture :)

[@sikun7894]

Thank you so much for sharing these lectures! Really useful

[@DarkNinja-24]

Beautiful explanation!

[@itachi7243456]

These are fantastic, thanks!

[@nhl8586]

Super useful for understanding NTK in 15 mins!

[@joonho0]

Thanks a lot for sharing this lecture!

[@meghbhalerao5208]

If I understand right, the NTK is derived when we only consider quadratic mse loss, right? can it be generalized to other loss functions?

[@yuwu7547]

Very useful and easy-catching lecture. Thanks a lot!

[@AlexanderGoncharenko-e7o]

Awesome lesson! Straight and clear!

[@chongyizheng7758]

Question about the first-order Taylor approximation of neural network: Why the first term f(w_0, x) is not included in the kernel function since it is nonlinear w.r.t. x?

[@ramanasubramanyam1110]

The first derivative is included (and called NTK) because it resembles the operation of a kernel on an input, i.e a transformation function mapping to a higher dimension

[@chongyizheng7758]

@@ramanasubramanyam1110 Thanks for your reply, but I don't think I am asking for that. Let me clarify: My question is about the constant (the first) term f(w_0, x) at 41:16 instead of the derivative (the second) term in the equation. f(w_0, x) seems also nonlinearly depend on x, why it was excluded in the definition of NTK?

[@hw1451]

I think since it's a constant, we can always subtract it from y.

[@yuzhema2506]

Thanks for the nice lecture! One question: the bias term in the Taylor approximation seems dependent on x, which means for different input x, the bias term varies. This is different from the traditional kernel view where the bias term is the same for different transformed input phi(x). In other words, for NTK, the inputs in the transformed space do not strictly follow the same linear model. How do we interpret such deviation? Thanks

[@sayeedchowdhury11]

thanks for the nice lecture, I have a query, we're evaluating the gradient at w0, does it mean the kernel is evaluated based on gradients obtained from an untrained NN which has just been initialized? i mean is the f(w,x) a trained NN or just an initialized one?

[@sinaasadiyan]

great explanation, just Subscribed!

[@tanchienhao]

Thanks for the awesome lectures!!

[@da_lime]

Awesome, thanks!

[@chenamora1653]

So amazing

[@vi5hnupradeep]

Thankyou so much!

[@ihany9061]

lifesaver!

[@freerockneverdrop1236]

The formula for the neural network in this video should be a 2 level summation instead of one level.