Kolmogorov-Arnold Networks: MLP vs KAN, Math, B-Splines, Universal Approximation Theorem

In this video, I will be explaining Kolmogorov-Arnold Networks, a new type of network that was presented in the paper "KAN: Kolmogorov-Arnold Networks" by Liu et al.
I will start the video by reviewing Multilayer Perceptrons, to show how the typical Linear layer works in a neural network. I will then introduce the concept of data fitting, which is necessary to understand Bézier Curves and then B-Splines.
Before introducing Kolmogorov-Arnold Networks, I will also explain what is the Universal Approximation Theorem for Neural Networks and its equivalent for Kolmogorov-Arnold Networks called Kolmogorov-Arnold Representation Theorem.
In the final part of the video, I will explain the structure of this new type of network, by deriving its structure step by step from the formula of the Kolmogorov-Arnold Representation Theorem, while comparing it with Multilayer Perceptrons at the same time.
We will also explore some properties of this type of network, for example the easy interpretability and the possibility to perform continual learning.
Paper: arxiv.org/abs/2404.19756
Slides PDF: github.com/hkproj/kan-notes
Chapters
00:00:00 - Introduction
00:01:10 - Multilayer Perceptron
00:11:08 - Introduction to data fitting
00:15:36 - Bézier Curves
00:28:12 - B-Splines
00:40:42 - Universal Approximation Theorem
00:45:10 - Kolmogorov-Arnold Representation Theorem
00:46:17 - Kolmogorov-Arnold Networks
00:51:55 - MLP vs KAN
00:55:20 - Learnable functions
00:58:06 - Parameters count
01:00:44 - Grid extension
01:03:37 - Interpretability
01:10:42 - Continual learning

[@josephamess1713]

The fact this video is free is incredible

[@umarjamilai]

You're welcome 🤗

[@AdmMusicc]

You're on a mission to make the best and friendliest content to consume deep learning algorithms and I am all in for it.

[@edsonjr6972]

Your videos are literally the only ones with 1hr+ I would ever watch on KZbin. Keep going mate, extremely high quality content 👏🏽👏🏽

[@bensimonjoules4402]

Amazing content, thanks! I'm very excited about the continual learning properties of these networks.

[@mohamedalansary2542]

Clearly explained and very valuable content as always Umar. Thank you!

[@nokts3823]

Thanks a lot for making this accessible for people outside the field, for which reading and understanding these papers is quite tough. Thanks to you I'm able to stay slightly more up to date with the crazy quick developments in ML!

[@xl0xl0xl0]

Wow this was a super clear an on-point explanation. Thank you, Umar.

[@MrNathanShow]

The intro of a basic linked up linear layers was so well done and really makes this introduction friendly!

[@franciscote-lortie8680]

Incredibly clear explanations, the flow of the video is also really smooth. It’s almost like you’re telling a story. Please keep making content!!

[@goldentime11]

Thanks Umar for such a wonderful tutorial! I've been eyeing this paper for a while!

[@andreanegreanu8750]

Very clear, well explained, top notch!

[@manumaminta6131]

Your videos help me (a grad student) really understand difficult, often abstract concepts. Thank you so much... I'll always support your stuff!

[@odysy5179]

Fantastic explanation!

[@MuhammadrizoMarufjonov-os5fv]

Thanks for including prerequisites

[@AlpcanAras]

This is life changing, in my opinion. Thank you for the efforts on the videos!

[@ozgunsungar9370]

awesome, easy to follow even person dont know anything :)

[@anirudh514]

Thanks for the crystal clear explaination!!

[@stacks_7060]

One of the best math videos I’ve watched on KZbin

[@anmolmittal9]

This is really great! Power to you!!🚀

[@luigigiordanoorsini5980]

Ho appena letto la piccola bio del tuo canale, spero di non essere offensivo dicendo che adesso capisco perché il tuo ottimo inglese mi sembrasse comunque molto familiare. Ad ogni modo ti ringrazio enormemente per il tuo contributo hai spiegato tutta la teoria in un modo, a mio avviso, estremamente chiaro e soprattutto coinvolgente. Ti prego continua così, di nuovo un enorme grazie e complimenti per il tuo contributo alla scienza

[@umarjamilai]

Grazie a te per aver visitato il mio canale! Spero di pubblicare più spesso, anche se per fare contenuti di qualità ci vogliono settimane di studio e preparazione. In ogni caso, spero di rivederti presto! Buon weekend

[@luigigiordanoorsini5980]

@@umarjamilai Avevi già guadagnato un iscritto adesso hai guadagnato un fan. Ahahahahah

[@MuhammadMuzzamil-ki4he]

Thank you for such great and detailed explanation.

[@jeunjetta]

I think KAN will be the catalist of a significant tipping point in science. I want to apply this to power system grids and replace existing dynamic models with ones made from PMU data using KAN

[@artaasadi9497]

that is very useful, informative and interesting! Thanks a lot!

[@user-pu4oc9ek9u]

Hello Umar, this video is my best birthday gift I have ever received, thanks a lot :)

[@JONK4635]

Extremely clear explanation and content here! Very helpful. I am happy that you came from PoliMI as well :) keep it up!

[@kmalhotra3096]

Hats off, what an awesome video!!!

[@ansonlau7040]

Thankyou Jamil, what a cool video

[@ScottzPlaylists]

High quality explanations.. Thanks.

[@arupsankarroy8722]

Sir, you are great..💙💙

[@johanvandermerwe7687]

I saw this paper on papers with code, and thought to myself I wonder if Umar Jamil will cover this. Thanks for your effort and videos!

[@enricovompa1876]

Thank you for making this video!

[@zaevi6855]

crazy that it took me an hr video to understand that its the (control points) being trained on the spline graph vs weights with MLPs and CNNs, thank you!

[@bankayxy00]

Thank you so so much for this amazing content.

[@user-il1hu5xp2x]

What funny, is that i predicted your next video will be on KAN, after i see you in github. I WILL WATCH THIS VIDEO, AS I FEEL THIS WILL BE THE FUTURE OF NEUR NETWORK, THANK YOU FOR YOUR WORK AND CONTENT ❤

[@lethnis9307]

Your explanations are the best, thank you so much😘🤗

[@howardmeng256]

Amazing video! Thanks a lot !

[@prathamshah2058]

Thank-you so much for explaining the paper, it is so easy to understand now, btw can you also make a hands on video with the kan package developed by mit which is based off pytorch.

[@wolfie6175]

Good video, quality content.

[@sergiorego6321]

Phenomenal! Thank you :)

[@hajaani6417]

You’re fantastic, mate.

[@coolkaran1234]

You are savior, without you mortals like me would be lost in the darkness!!!

[@GUANGYUANPIAO]

awesome explanation

[@user-wy1xm4gl1c]

This is awesome!

[@samadeepsengupta]

Great Content !!

[@RiteshBhalerao-wn9eo]

Amazingg explanation !

[@danielegiunchi9741]

brilliant video!

[@dhackmt]

i loved it sir .

[@vaadewoyin]

Cant wait to watch this, saved! Will comment again when i actually watch it..😅

[@JuliusSmith]

Excellent video, thanks! At the end, I _really_ wanted to see an illustration of the relatively "non-local" adaptation of MLP weights. Can that be found somewhere?

[@p4ros960]

bruh so good. Keep it up!

[@seelowst]

Having a such good teacher is so adorable, i wish i could be your students.

[@umarjamilai]

哪里哪里啊，谢谢你的赞成！

[@seelowst]

@@umarjamilai 太棒了，您还会中文👍

[@umarjamilai]

@@seelowst 我就是刚刚从中国来的，在中国主了4年了，现在回欧洲了。

[@seelowst]

@@umarjamilai 我从没离开过我的城市，我希望像您一样👍

[@pabloe1802]

An implementation video will be awesome

[@ezl100]

thanks Umar. Very nice explanation. Just 2 questions : 1 - Does it mean we can specify different knots per edge? 2 - I am not understanding how the backpropagation will work. Let's say we calculate the gradient from h1. It will update phi 1,1 and phi 1,2 but how the learning process will impact the knots to the desired value?

[@akramsalim9706]

awesome bro.

[@ai__76]

amazing

[@fatemeshams9758]

awesome👍

[@faiqkhan7545]

Umar bhai you the great

[@subhamkundu5043]

Hey @Umar, great content as always. Looking forward to a KAN implementation video from scratch. Also I think in 31:01 there is a minor language mistake. I think it will be for using a quadratic Bspline curve rather than quadratic Bezier curve

[@satviknaren9681]

Please do post more ! please do more videos !

[@user-jb3ht1wq5l]

THANK YOU

[@daleanfer7449]

刚好期盼这个!

[@umarjamilai]

期待你的评价😇

[@daleanfer7449]

❤很好的内容，有考虑做inverse rl的内容吗❤

[@shubhamrandive7684]

Great explanation. What app do you use to create slides ?

[@umarjamilai]

PowerPoint + a lot a lot a lot a lot a lot of patience.

[@plutophy1242]

this video is so amazing！！！！！！！

[@bzzzzz1736]

thank you

[@fouziaanjums6475]

Hi, can you please make a video on multimodal LLMs, fine tuning it for custom dataset...

[@emiyake]

Thanks!

[@user-hd7xp1qg3j]

Could you please next explain multi modal llms, techniques like Llava, llava plus, llava next?

[@Patrick-wn6uj]

I waiting for that day too

[@baba42kachari]

Thanks

[@user-sy6xn7nq7s]

There are continuous but indiferable points in the spline, right? What are you going to do?

[@Kishan31468]

Thanks man. Next xLSTM please.

[@DiegoSilva-dv9uf]

Valeu!

[@willpattie581]

One thing I didn’t catch: how are the functions tuned? If each function consists of points in space and we move around the points to move the B spline, how do we decide to move the points? Doesn’t seem like backprop would work in the same way.

[@umarjamilai]

The same way we move weights for MLPs: we calculate the gradient of the loss function w.r.t the parameters of these learnable functions and change them in the opposite direction of the gradient. This is how you reduce the loss. We are still doing backpropagation, so nothing changed on that front compared to MLPs.

[@routerfordium]

Thank you for the great video! Can you (or anyone) help understand why you need to introduce the basis functions b(x) in the residual activation functions?

[@MrAloha]

Wow! 🙏

[@Engrbilal143]

Time to implement it

[@rohitjindal124]

Sir I have been a huge fan of your videos and have watched all of them . I am currently in my second year BTech and really passionate about learning ml sir if possible can work under you I don’t want any certificate or anything just want to see observe and learn

[@jeremykothe2847]

fwiw I took a MLP solution for MNIST, substituted KAN for the MLP layers and no matter what I did (adding dimensions etc) it couldn't solve it. My intuition is that KANs only work well for approximating linear-ish functions, not irregular, highly discontinuous ones like image classification would need. But perhaps I just screwed it up :D

[@ChukwuemekaAmblessedchinenye]

can you make tutorial video on model like Perplexity that use website live search

[@ScottzPlaylists]

Please explain DSPy

[@suman14san]

Please add a payment option

[@umarjamilai]

Your love and support is enough! Have a great weekend!

[@Patrick-wn6uj]

@@umarjamilaiJust woow

[@pratishdewangan132]

In search of gold i found a diamond

[@einsteinsapples2909]

Your explenations are great. I think though, you should take breaks to blow your nose maybe, because you were sniffing a lot. It will make you videos more enjoyable.

[@ln_exp1]

Interesting

[@kiffeeify]

Thanks!

[@alfredmanto5487]

Thanks