Gradient Descent, Step-by-Step

Gradient Descent is the workhorse behind most of Machine Learning. When you fit a machine learning method to a training dataset, you're probably using Gradient Descent. It can optimize parameters in a wide variety of settings. Since it's so fundamental to Machine Learning, I decided to make a "step-by-step" video that shows you exactly how it works.
NOTE: This video assumes you are already familiar with Least Squares and Linear Regression. If not, here's the link to the Quest: • The Main Ideas of Fitt...
For a complete index of all the StatQuest videos, check out:
statquest.org/video-index/
Sources:
There are a ton of websites that describe the math behind Gradient Descent. One of my favorite is the wikipedia article: en.wikipedia.org/wiki/Gradien...
If you'd like to support StatQuest, please consider...
Buying The StatQuest Illustrated Guide to Machine Learning!!!
PDF - statquest.gumroad.com/l/wvtmc
Paperback - www.amazon.com/dp/B09ZCKR4H6
Kindle eBook - www.amazon.com/dp/B09ZG79HXC
Patreon: / statquest
...or...
KZbin Membership: / @statquest
...a cool StatQuest t-shirt or sweatshirt:
shop.spreadshirt.com/statques...
...buying one or two of my songs (or go large and get a whole album!)
joshuastarmer.bandcamp.com/
...or just donating to StatQuest!
www.paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
0:00 Awesome song and introduction
1:25 Main ideas behind Gradient Descent
5:38 Gradient Descent optimization of a single variable, part 1
9:08 An important note about why we use Gradient Descent
9:40 Gradient Descent optimization of a single variable, part 2
14:48 Review of concepts covered so far
15:48 Gradient Descent optimization of two (or more) variables
21:55 A note about Loss Functions
22:13 Gradient Descent algorithm
23:06 Stochastic Gradient Descent
#statquest #gradient #descent #ML

[@statquest]

NOTE 0: If want to learn more about The Chain Rule, see: kzbin.info/www/bejne/rZ2Unqyup9mEfrM NOTE 1: The StatQuest Gradient Descent Study Guide is available! statquest.org/studyguides/ NOTE 2: A lot of people ask why we are using Gradient Descent to estimate the parameters in this video when we could just use least squares. We use least squares to produce a "gold standard estimate". This is the best possible estimate. We then attempt to derive the same estimate using Gradient Descent. This shows 1) how gradient descent works and 2) that the estimate is pretty good compared to the "gold standard". NOTE 3: A lot of people ask how I found the slope value, 0.64. In the example in this video, we can compare the estimates from Gradient Descent to those that come from another method called "least squares". For specific problems, we can plug the data into the least squares formula and the output is the optimal parameters. To learn more about the specific least squares formula see: en.wikipedia.org/wiki/Least_squares If you're wondering why, when we have least squares, would we want to use gradient descent... the answer is that least squares only works in specific situations and gradient descent can work in many more. Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

[@emperorcyber509]

StatQuest with Josh Starmer you are the best

[@statquest]

@@emperorcyber509 Thanks! :)

[@lordoftranceaa]

it was driving me crazy, but thanks! it was an error

[@BeksKazama]

Thank you so much! I have a qn. It has to be SSE instead of SSR, right? SSR is (y predicted - y bar)^2, where SSE is (y pred - yi)^2

[@jojotheyorkie3157]

Thanks. Was looking for this. This should be pinned.

[@Android-cm5gn]

I cannot believe how many teachers get paid in schools and in universities to make students feel stupid just because they cannot explain a very important topic in a similar way like you! So much respect!

[@statquest]

Thank you! :)

[@wirito]

M AK that’s not the problem man. Problem is we have to pay them thousands of dollars when in reality, many people in KZbin deserves that money instead. My education wouldn’t be complete without KZbin. Thanks Josh for your videos :)

[@srtghfnbfg]

Everyone upvote this comment please. And 100% respect to Josh !

[@Android-cm5gn]

@@tilkesh I don't understand why it needs decades of hard work to explain topics in a decent way. I had teaching assistants who were totally fresh to a topic and yet they explained it way better than my professor.

[@jaliu]

and you end up studying for years without learning anything!!

[@RandomGuy-hi2jm]

there are real institutes which takes 4000-5000$ to teach you this and then comes statquest which saves your tons of money and provide u better explanation...... SUPER BAMMMMMMMM!!!!!!!!!!!!!!!!!!!!!!!!!!!

[@statquest]

BAM! :)

[@shreejanshrestha1931]

@@statquest it's not just BAM, The BAM is really big. Bigger than the bam that killed the dinosaurs. Cause u r best bam bam bam bam

[@taraprasadmishra3839]

This man can teach ML to a 5 years baby 😅

[@statquest]

Thanks! :)

[@sparkartsdistinctions1257]

👶🏻 🍼True.

[@krshah2008]

im 5 and i can confirm this.

[@sparkartsdistinctions1257]

@@krshah2008 No you are too young to confirm. 🐶

[@kunthoopi]

exactly 😂

[@greatmanokonkwo656]

I love how this guy cares to explain every single detail, not assuming any prior knowledge whatsoever. I was genuinely shocked when he started calculating the derivatives in the video. Most resources will skip over such minute details and calculations, only focusing on the concepts. These videos are the most beginner-friendly resources on the web for ML, simply amazing.

[@greatmanokonkwo656]

And also the fact that he doesn't scare is viewers with conventional mathematical notation. He really just gets to the core of the topics.

[@statquest]

Thank you so much! I'm really glad you like my teaching style. :)

[@EagleYin]

I really enjoyed this video but it's lack of code. I got a great video here implement sgd using Python!!! Feel free to check it out!! kzbin.info/www/bejne/q4nYc4ihjMRjqqM

[@kevindave277]

@@statquest You are HIM.

[@izzatulummah1594]

@@statquestI really like your teaching style. I will adopt this to my class. Thank you very much.

[@knightganesh]

" If you can't explain it simply, you don't understand it well enough" - Albert Einstein Can't get much simpler than this explanation. Thank you so much awesome work keep it going 👍👍👍👍👍👍👍👍👍👍👍👍

[@statquest]

Thank you very much! :)

[@clementt3940]

La vraie phrase est : "Ce qui se conçoit bien s'énonce clairement, et les mots pour le dire arrivent aisément." de Nicolas Boileau. Einstein coupable de plagiat ? ;)

[@arvinmohammadi290]

i think that's from Richard Feynman. anyways I agree

[@thongnguyen1292]

Clearest explanation in this universe, as always! Thanks a lot!

[@statquest]

Thanks so much! Glad you like it. :)

[@anujsaboo7081]

So let me see, you taught this like teaching a kid by repeating the same thing over and over again and covered such detailed calculation in 20 minutes. It is just amazing how you make short videos and even revise prior concepts at the beginning and still manage to keep everything short. The songs, the BAMS! everything unique about this channel. Amazing learning experience. THANK YOU!

[@statquest]

Thank you very much!!! :)

[@kawishabbas7254]

Oh man you are a great. How clearly you explained everything. Thanks

[@pradeepmallampalli4541]

Give this man the highest medal in teaching. Thanks a lot for all the effort you made to explain this.

[@suwarnachoudhary8591]

When ever I feel I am not able to understand any concept after reading any book or any literature available on internet, I open Statequest. This channel should be awarded as the best ever tutorials on Machine Learning concept. You are unbelievable Josh Starmer!!! Thank you

[@statquest]

Thank you so much!!!! :)

[@whyareugae5528]

I wish I saw this video before I attended a 3 hour lecture on GD. Professors have no idea how to explain the basics. Thank you so much, now it all makes sense.

[@statquest]

Glad it was helpful!

[@aryana506]

Your videos are so well put together. Thanks for all the time you put into preparing them!

[@statquest]

HOLY SMOKES!!! Thank you so much for supporting StatQuest! BAM! :)

[@wei2674]

“it sounds fancy, but it’s really no big deal” Thank you for marking me feel the same way about those fancy named methodologies after watching the videos!! I do feel more confident now in learning new stuff! If I don’t get it, it’s not because Of me, it’s simply because the book/paper/course note is not as good as statquest :)

[@statquest]

Awesome! :)

[@kartickdhali4352]

This is the best video on Gradient decsent on internet. I wish I could have a teacher just like you in my college.

[@statquest]

Thank you!

[@divjyotsingh7400]

Brilliant! 2 years into ML and I still find this the best explanation of GD.

[@statquest]

bam!

[@kk10-]

You did what Andrew Ng failed to. Explaning me THE GRADIENT DECENT algo.

[@statquest]

Hooray! I'm glad the video was helpful! :)

[@DixitGokhaleEngineer]

Completely agree. In fact I just look at the Andrew's course for the topics that needs to be learnt in order. And for the detailed explanation 'BAM' does a fantastic job. Thanks a lot Josh.

[@karisc.anoruo2212]

@@DixitGokhaleEngineer what is BAM please? I'm taking Andrew Ng's course and I want to learn from as many sources a possible!

[@darksideofthemoon3185]

​@@karisc.anoruo2212 8:51

[@yassertalebi1026]

Exactly , that's why i was searching for something else to understand it since Andrew Ng failed

[@adolfomotanavarro6553]

Boop beep beep boop boop (Translation: Excellent explanation, 5 out of 5)

[@statquest]

Awesome! :)

[@samskyverareddy3135]

I remember what gradient descent does but your explanation on slopes and derivatives gave me a big 'AHA!' moment. Thank you! Instant Subscription! Looking forward to more videos.

[@nriezedichisom1676]

Beautifully explained. For the very first time, I understand this

[@janilayturatkhankyzy1321]

I will pray for your health. So much hard work put into this channel. God bless you

[@statquest]

Thank you very much! :)

[@999Stergios]

StatQuest. You're Gold, my friend. Pure Gold. 1000 Thanks and bravo. The way you explained it and the graphics was brilliant. Super simple super easy, very educational! Bravo, Bravo!

[@statquest]

Hooray!!! I'm so happy to hear that you like this video. :)

[@mohamedaitahmed2842]

Of all the explanations I've watched, this is, by far, the best explanation of gradient descent. Thank you for existing !

[@statquest]

Wow, thank you!

[@liar.2390]

Ahh thank you so much! I recently started working on (or more like preparing to work on) my MD thesis and there's a lot I need to learn by myself. Channels like yours really are lifesavers - you explain everything so well and it's actually entertaining to watch. You have a new subscriber. :)

[@statquest]

Thank you and good luck with your thesis. :)

[@j8ahmed]

One of the absolute best educational videos I have ever watched. Based on your explanations I have been able to: - walk myself through coding a gradient descent algorithm - understanding the concept of tuning the learning rate (alpha) - how to handle multivariable problems, - and more. I'm studying machine learning right now and this has helped me so much. I can't say how great this was. Thanks a million!

[@statquest]

Wow, thanks!

[@xx-jk1iq]

would you be able to share what you did with the algo

[@alexpehers3033]

The best explanation I´ve seen so far. Slow, clear and super easy to understand!

[@statquest]

Thank you! :)

[@jaelbutler7966]

So helpful!!! Thank you so much! The imagery and step-by-step walk-through were just what I needed! It makes so much more sense now!

[@statquest]

Hooray! I'm glad the video was helpful. :)

[@macunknown9173]

Love how clearly you explained, the recap in middle sure helps too, doesn't let us get lost !! More power to you 🙌🏻

[@statquest]

Thank you! :)

[@kartik_exe_]

to be honest i never ever saw a teacher like him which teaches us and after the sessions we don't have any doubts to even ask. Kudos to him and also thank you StatQuest

[@statquest]

Thank you!

[@aanya8834]

I've never come across a clearer explanation for gradient descent,this is so cooool! Love this channel! Thank you so much for making us fall in love with ML!

[@statquest]

Awesome, thank you!

[@shivakumarjadipalli6823]

I have took a data science bootcamp of $4000 but even they didn't explain it like you did great work , Please don't stop doing this

[@statquest]

Thank you!

[@justinli19901027]

can't appreciate enough how clearly the concept is explained. thank you

[@statquest]

Thank you very much! :)

[@jayghosh8898]

I can't thank you enough for making these informative and immensely useful videos. I think this is the first time I am using KZbin to learn something useful, and your videos make it so enjoyable

[@statquest]

Glad you like them!

[@FrequentC88]

Love humor you've added to what can usually be such a dry topic. The "squiggle" for logistic regression made me laugh. Keep up the great content!

[@statquest]

Thank you so much! :)

[@TheFolkRevival]

What's really golden about this video is that you don't assume the viewer knows f.e the Chain Rule (even though it's good) and also show all the steps clearly. A lot of Math "tutorials" skip some crucial steps that might not be obvious to someone new to an area/idea, which make the material inaccessible. For comparison, even if your videos lack the great animations of f.e 3Blue1Brown, I think some of your explanations are much easier to understand.

[@statquest]

Wow! Thank you very much! I really appreciate it.

[@cool20guitar]

Your videos are the best. No one could have explained it better. Love the sense of humour in between to spice things up. Thank you so much!

[@statquest]

Thank you! :)

[@gauthampughazhendhi8173]

Josh Starmer, you are really good at making videos that explain the difficult concepts in a simple, understandable and engaging fashion. THANK YOU

[@statquest]

Thank you! :)

[@Thekingslayer-ig5se]

This man is an icon! He should be celebrated much more! You have no idea of how well he teaches stuff! Way to go my man

[@statquest]

Thank you!

[@yugix901]

Great explanation! Really glad this video was the first one to pop up when I searched for gradient descent, thanks:)

[@statquest]

Awesome! :)

[@vipulchaudhari810]

Explanation is crystal clear and spot on. Thanks Josh for investing your time and energy to create such wonderful learning material.

[@statquest]

Glad it was helpful!

[@davidphan6499]

Mr. Starmer was born to be a teacher. Thank you.

[@statquest]

Thanks!

[@anur749]

You are a saviour! Whole University wasn't able to make this so simple..you should be given highest honorary award in Data Science

[@statquest]

Awesome! Thank you so much! :)

[@merebhayl5826]

Josh, You are a beautiful creature and gift! glad you exist! your thought process, your explanation, your positivity in teaching this to us, your creativity through art, and your music and albums. Be glad you exist to us all. I am also an electrical engineer, working primarily in software engineering, with a passion for music, numerous instruments, and painting. I am glad I came across you as a person, besides the great education!!

[@statquest]

Wow, thank you!

[@yaygrace]

Thank you so much for making this video!!! I'm struggling in my stats class but after having it broken down step by step like this, I understand it so much better! You are a LIFE SAVER

[@statquest]

Glad it helped!

[@wonggran9983]

you are a blessing, you give a high level explanation and with a little background knowledge, we can _descend_ into the depths of formality that university courses go into ourselves!

[@statquest]

Thanks! :)

[@blueboystudios]

Best stats teacher on the planet!! Thank you for your videos Josh!

[@yeparthepirate]

I'm struggling so much in Machine Learning right now. After watching just two of your videos I am feeling so much more confident. Thanks so much!

[@statquest]

Hooray! Good luck with your studies. :)

[@SuperPvenegas]

Thanks Josh, I'm a first time viewer of StatQuest, and thought I'd leave a comment and say BAM! Thanks for taking the time to appropriately distill key concepts (e.g. gradient descent) and make them more consumable. Also, just purchased the paperback on Amazon, should arrive tomorrow DOUBLE BAM!

[@statquest]

Hooray!!! Thank you for supporting StatQuest!!! :)

[@emirhalilovic3362]

The best explanation of this topic I've come upon so far, helps me a lot while learning the gradient descent. Thank you so much!

[@statquest]

Thank you!

[@gix111]

You explain complex things to me better than my professors. Thanks for saving my semester. ❤️

[@statquest]

Happy to help!

[@laslodes2550]

I usually don't comment under youtube videos, but I had to it this time. Currently working on a robotics project and I have to use the gradient descent method to solve inverse kinematics. I didn't understand the method as explained during my lectures and I ended up watching this video. Seriously, I could even explained it to my prof and submit a request so that this video is shown to students during the teaching of this topic. You're amazing and your explanation is so much easy to follow and understand. Thank you so much for this masterpiece.

[@statquest]

Thank you very much and good luck with your robotics project.

[@Alan-qb9qt]

the most intuitive and detailed in every point gradient descent tutorial i've ever watched

[@statquest]

Thank you!

[@Rahul-fq9kf]

You teach so so well. Very thankful to you for these videos. I wouldn't have understood anything had it not been through you. Can't thank you enough! God bless you!

[@statquest]

Wow, thank you!

[@diencai1812]

OMG this is pure gold !!! Thank you so much for the time in compiling the video and sharing the insight

[@statquest]

Thank you! :)

[@diencai1812]

Love your work Josh; a lot of planning and sequencing have gone into the video production - you deserve a KZbin 'Oscars'. Your channel is making an impact to my learning journey👍🙏

[@ghosh5908]

This channel is awesome...It should have millions of subscribers....I have become a fan Josh...big fan...

[@SachinRaneTheSKR]

This has to be, without a doubt, the most understandable and clearly explained video on Gradient Descent on the internet! Thank you so much!

[@statquest]

Wow, thanks!

[@filizcamuz8119]

Thank you, the content is amazingly clear and straight to the point. Graphics are very helpful. Thanks for your time. Please keep going.

[@statquest]

Hooray! I'm glad my video is helpful. :)

[@nikhilpradeep336]

Hi Josh, I an a new entrant to the field of Machine Learning and was really struggling to get a hold of the topics; understood the concept of Gradient Descend by watching this one video. Thanks a lot Josh , you are the best!

[@statquest]

I'm glad I could help! :)

[@fishermanwithfishes2286]

Teacher: "what we knew about Gradient Descent?" Me:"THE CHAIN RULE!!!!!!!!!!"

[@statquest]

YES! :)

[@carlosmorenom.122]

Bup bup bip bip bup bip bup

[@StanleyTam117]

This is one hell of a super awesome explanation of gradient descent. Thanks so much so much so much, Josh!

[@statquest]

Thank you!

[@waleedalmarshidi3494]

I swear this is the most clear and FANTASTIC explanation I've ever found

[@statquest]

Wow, thanks!

[@hashiska.5358]

I will summarize what's written in the comments section: WE LOVE YOU.

[@statquest]

Hooray!!! Thank you very much! :)

[@markusvonderluehe5468]

Great explanation starting with the basics and explaining it step by step! Brilliant!!!

[@statquest]

Thanks! :)

[@MrDiegonolovich]

This was a very well explained video. Clear and to the point with easy to follow examples.

[@statquest]

Thanks! :)

[@joyhumin1]

Cannot express enough appreciations of your amazing content! Please continue your work!

[@statquest]

Thank you! :)

[@Frost571]

Thank you for explaining this in a simple way I can understand. I'm almost in tears right now 🥲

[@statquest]

Glad it was helpful!

[@ghaliahmed]

best of the best explanation in this word !!!!!!!!! Thanks a lot!

[@ivanjonathan4240]

The clearest explanation i've ever watched! thank you mastah!

[@statquest]

Awesome!

[@gesunonerabianco8016]

Straight to the root of each subject, thanks a lot for your crystal clear explanations my friend!

[@statquest]

Thank you very much!!! :)

[@hajhouj]

THE CHAIN RUUUUUULE !!! :-D ...... I Love your videos, best courses I ever seen

[@bbow4972]

Normally I don't like when tutorial vids try to be funny or cute in their vids but this video was great. It was just enough to make the lesson not feel so monotonous or droning without getting away from the lesson at hand. It was also surprising easy to understand. Thanks for the vid.

[@statquest]

Awesome! Thank you. :)

[@meow-mi333]

Dude explained things clearly. Huge thanks! Helped me review what I’ve learned. I feel much better.

[@statquest]

Thank you so much!

[@tagoreji2143]

One of the Great Lectures I have ever attended.Thank you Professor Josh stammer 🙏🙏🙏

[@statquest]

Thank you!

[@ramsunil4317]

Loved the way your teaching, u made mathematics my favorite subject

[@statquest]

Hooray! :)

[@elgabotb1]

Me encanta! Eres todo un tezo como decimos aquí en Colombia. Love it, you are such a "tezo" (brilliant) like we used to say here in Colombia

[@statquest]

Muchas gracias!!! :)

[@simetric6551]

Hola soy nuevo en esto. Me podrias decir de donde salio el 0.64 del slpoe o pendiente al principio del video. Gracias

[@viniciusdeoliveira7637]

@@simetric6551 I think that it probably came from the previous calculation upon the example data.

[@sharathcg4697]

This is the easiest possible explanation that one can ever or never give. great work in video editing as well

[@statquest]

Thanks a lot!

[@swathikakandan4749]

Wish every learner gets a teacher like you.You are awesome,Thank you so much

[@statquest]

Thank you very much! :)

[@kalanagayan5283]

I'm currently proceeding to my Final year of Bachelors degree, This is the minimalist most clear , interactive and funny session I've seen , Love your work, its been a great help for the studies. Love and respect all the way from Sri Lanka !!

[@statquest]

Thank you very much! Good luck finishing your degree! :)

[@xiangzhang7355]

I have never seen anyone explain gradient descent as clearly as you!

[@statquest]

Thank you!

[@raghus8535]

Many Thanks, Mr. Josh, cant believe it. watched several videos on gradient descent but this one is the best i have ever watched. appreciate your effort.

[@statquest]

Thank you very much! :)

[@lilyli339]

Love it so much!!! This is the best gradient descent lecture I found!!! Hope there will be mini-batch gradient descent lecture in your channel too!! Thank you so much for your videos!

[@statquest]

For mini-batch, see: kzbin.info/www/bejne/rH7LYa2GiZWngqs

[@AdityaFingerstyle]

There couldn't be any better explanation for Gradient Descent !

[@statquest]

Thank you! :)

[@123gregery]

I'm following Andrew Ng's Machine Learning Course. I thing he should give a reference to your video for an explanation. That is superb, provided that we know some elementary calculus.

[@statquest]

That would be awesome if he did that. A lot of people come to my videos via Andrew Ng's course, but they have to find it on their own, just like you do. Perhaps you could suggest it to that class. Regardless, I'm glad you like my videos. :)

[@baruchschwartz819]

@Beyond Oblivion me three

[@Leonardo-jv1ls]

You explain things so slow and step-by-step as it was being explained for chimps. Exactly what i needed. Thank you a lot.

[@statquest]

Glad it was helpful!

[@manuelplank5406]

This video is just perfect to gain an intuition of gradient descent (what math heavy lecture slides fail to deliver)

[@statquest]

Thanks! :)

[@Adventurer_Deepu]

Thanks a lot sir Clearly explained Awesome video.

[@statquest]

Thank you! :)

[@EvelinaPettersson-mk5ot]

Thank you! Very helpful video, I really liked the way you presented it, made it fun to watch and easy to follow. I usually have a problem with keeping my concentration when watching videos like this, but I had full focus throughout the whole video! Cheers!

[@statquest]

Thanks!

[@maierelias7315]

I've watched so many videos about this topic but this is the first time I understood it. Great explanation thanks for the video

[@statquest]

Glad it was helpful!

[@albertding9700]

Hi Josh this was really helpful. Can you do deep learning and neural networks next?

[@statquest]

Those are definitely on the to-do list, but right now I'm working on Gradient Boosting, so that's next.

[@samiulsaeef2076]

@@statquest Any chance to get the Gradient Boosting video withing a week? That would be specially helpful due to some personal urgency!

[@user-ek7sz3mu3i]

I envy modern young math students in colleges all over the world since they can easily choose best explanation of complex things on KZbin( Your channel seems the best I found!). It;s not like studying in the beginning of 2000's when the teacher was drawing some ugly formulas with chalk. making so great thing as maths boring to us. Who knows may be someone who will treat cancer or colonize Mars will pick up some missing brick of understanding from your videos. Pretty much thanks for your effort!

[@statquest]

Wow! Thank you very much! :)

[@ArexHopeRX]

The most useful and understandable for a bird head like me explanation i've ever heard so far! Thank you

[@statquest]

Awesome! I'm glad my video was helpful. :)

[@MarcelloNesca]

Unbelievable, I'm so PROUD to support your Patreon, your explanations leave me slack jawed with how conceptually easy it is to understand. Meanwhile I'm reading a book that over complicates things, you should get a teaching award!!

[@statquest]

Thank you very much and thank you for your support! :)

[@jaliu]

you accidentally taught me how to take derivatives in a few seconds which i never understood for years... mind is blown.

[@statquest]

That's awesome!!! :)

[@victoro8295]

really? It took you years to apply the chain rule?

[@omnesomnibus2845]

Great presentation as always! One note, at 18:53 I believe you forgot to take out the ^2 for the second and third slope-derivative terms of the residuals. But the idea is clear anyway. I'd like to see this implemented for clusters and logistic regression as well, and what loss functions are best for them and why. Maybe also a bit more detail about why the slope multiplied by the learning rate is the best way to find the step size. You mentioned it, in terms of the desire to approach a slope of zero (therefore it makes sense to have a negative multiple of the slope be the penalty, since a large slope would lead to subtracting a large amount and a small slope would lead to subtracting a small amount). As I said, you mentioned it, but perhaps a bit more visual demonstration would be good. Great though! P.S. As an additional source of revenue, you might 'sell' a zipped version of all your videos for a donation, or perhaps put it into a book that can be downloaded for a donation.

[@statquest]

Ah - there are always a few typos. Darn. I thought I'd caught all of them. But, like you say, the point is clear, so I won't lose any sleep over it, I hope! If you're curious about how gradient descent works for t-SNE, just look at the original manuscript for it (simple google search). They show the gradient that they use.

[@omnesomnibus2845]

@@statquest Okay, thanks for the pointer!

[@mariuszwilk4718]

​@@statquest Also at 11.05 you say that the step size is "negative five point seven".. I presume you meant to say -0.57, as it is written on the slide.

[@a_sun5941]

Agree that the presentation is really really great! And just one more note besides the typo at 18:53: there are two pairs of concepts slightly a bit confusing in this video -- (a) weight for linear regression vs the x variable of human weight (b) slope of the linear regression fit and slope of the sum of squared residuals vs intercept. For (a), as slope of linear regression fit is often referred as 'weight' of the function y^ = w* x+b, using a variable of human weight could be a bit confusing, it might be better if you call it 'human weight' to distinguish with function weight, or find some other variables instead of human weight. For (b), when you talk about 'slope' in gradient descent, I sometimes thought you were talking about the slope of the linear regression, and then realized it's another slope. Maybe it will be good if you use 'gradient' for the second slope?

[@Yousafkhan-gv7cs]

Teaching is an Art and you explain this dry and difficult topic in a very comprehensive way! thanks for sharing your knowledge and skill.

[@statquest]

Thank you very much! :)

[@Anonymous-fr2op]

I have no words to say how amazing you are. Not only are your videos super easy to grasp, but also, wth, even after 4 years you're replying to your comment. Wow! Amazing

[@statquest]

Bam! :)

[@kinwong6383]

Please keep coming up with videos. The possibility of getting my Master in Data Science is solely depend on you now lol

[@retrom]

Which university?

[@hunters.dicicco1410]

i gained new intuition on gradient descent that i never realized during university because of this video. thanks a MILLION josh!

[@statquest]

Hooray! I'm glad the video was helpful. :)

[@MoumitaHanra]

I think I only understand your lessons (and a handful of very few others) and it has brought about a new understanding of these algorithms and concepts, really appreciate your lessons!!!!

[@statquest]

Thanks!

[@edidiongesu4035]

First time on this channel and Big shout out to you, man. Subscribed quickly too. Thank you.🔥🔥🔥

[@statquest]

Thank you!

[@hai7263]

The beep boop really caught me by surprise. Laughed out loud in front of my computer...

[@statquest]

Bam! :)