Gradient Boost Part 1 (of 4): Regression Main Ideas

Gradient Boost is one of the most popular Machine Learning algorithms in use. And get this, it's not that complicated! This video is the first part in a series that walks through it one step at a time. This video focuses on the main ideas behind using Gradient Boost to predict a continuous value, like someone's weight. We call this, "using Gradient Boost for Regression". In the next video, we'll work through the math to prove that Gradient Boost for Regression really is this simple. In part 3, we'll walk though how Gradient Boost classifies samples into two different categories, and in part 4, we'll go through the math again, this time focusing on classification.
This StatQuest assumes that you already understand....
Decision Trees: • Decision and Classific...
Regression Trees: • Regression Trees, Clea...
AdaBoost: • AdaBoost, Clearly Expl...
...and the tradeoff between Bias and Variance that plagues Machine Learning: • Machine Learning Funda...
For a complete index of all the StatQuest videos, check out:
statquest.org/video-index/
This StatQuest is based on the following sources:
A 1999 manuscript by Jerome Friedman that introduced Stochastic Gradient Boost: statweb.stanford.edu/~jhf/ftp...
The Wikipedia article on Gradient Boosting: en.wikipedia.org/wiki/Gradien...
The scikit-learn implementation of Gradient Boosting: scikit-learn.org/stable/modul...
If you'd like to support StatQuest, please consider...
Buying The StatQuest Illustrated Guide to Machine Learning!!!
PDF - statquest.gumroad.com/l/wvtmc
Paperback - www.amazon.com/dp/B09ZCKR4H6
Kindle eBook - www.amazon.com/dp/B09ZG79HXC
Patreon: / statquest
...or...
KZbin Membership: / @statquest
...a cool StatQuest t-shirt or sweatshirt:
shop.spreadshirt.com/statques...
...buying one or two of my songs (or go large and get a whole album!)
joshuastarmer.bandcamp.com/
...or just donating to StatQuest!
www.paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
0:00 Awesome song and introduction
2:58 Gradient Boost compared to AdaBoost
5:50 Building the first tree to predict weight
10:37 Building the second tree to predict weight
13:28 Building additional trees to predict weight
13:50 Prediction with Gradient Boost
14:28 Summary of concepts and main ideas
#statquest #gradientboost

[@statquest]

NOTE: Gradient Boost usually uses regression trees. These are very similar to classification trees, but have a slightly different way to decide how to add branches and leaves. For more details, see: kzbin.info/www/bejne/nWrGZ2mKit6fkJY Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

[@nataliabarros4765]

Hi Josh. Is there a video similar to this but about Lightgbm?

[@statquest]

@@nataliabarros4765 Not yet, but I have a video about XGBoost: kzbin.info/www/bejne/haWnaaqMlqugbKc

[@parijatkumar6866]

@@statquest Keenly awaiting Sir.

[@statquest]

@Kristofer Yes, that is correct. That tree is a Regression Tree because it was fit to continuous values (which we see in the leaves).

[@statquest]

@Kristofer No, Gradient Boost Classification is still done with Regression Trees because we are fitting the trees to residuals. For details on Regression Trees, see: kzbin.info/www/bejne/nWrGZ2mKit6fkJY

[@user-dq6di3fg8c]

Thanks for your immense contribution to humanity

[@statquest]

Thank you! :)

[@Privacy-LOST]

This is not an overstatement

[@statquest]

@@Privacy-LOST :)

[@dahailiu2067]

@@Privacy-LOST 😂😂😂😂😂

[@lpcruz5661]

I agree, a step in the right direction and likely save the world

[@xiao2634]

This channel should be boosted. Better than my professor.

[@user-wl4ck1vb9q]

I also think soI

[@trantoan6985]

Sure

[@parijatkumar6866]

Yes

[@beautyisinmind2163]

@@user-wl4ck1vb9q 中国教授这样教不了的

[@suprobhosantra]

Use xgboost to boost the chnl😂

[@ahanadrall5661]

i still remember the first StatQuest i watched. "you can fit a line, you can fit a squiggle, you can make me laugh, you can make me giggle. StatQueeeest" keep up the good work. this is one of the best channels of all. thanks josh.

[@statquest]

Thank you so much! (and that's one of my favorite intro songs...!)

[@emrecaglayan1329]

You were BAMless for ten min. I was kind of concerned. But then you made a step in the right direction.

[@statquest]

Ha! :)

[@AyushMandowara_xx7]

I am assuming that "BAM" is a famous keyword in these videos. Is my mental model making the correct prediction?

[@panchamdesai7090]

Hey your channel is a saviour for me because you make all complicated concepts easy to understand . Thanks for contributing the datascience community with such great content

[@statquest]

Thank you so much! :)

[@chongxj9263]

Thanks for your hard work producing all these incredible tutorials Josh. I just can't imagine how'd I have learned these concepts otherwise without your videos!

[@statquest]

I'm happy to hear my videos are helpful. BAM! :)

[@anjithnair3082]

One of the best explanations for GB regressor. Excited for the next one.

[@jaysoncena8539]

Holy shit, I was able to finish my assignment in 1 night while watching this. For the past 1-2 weeks I'm doing an all-nighter and pulling my hair to finish my assignment using the notes I have from my class.

[@statquest]

Hooray!!!! I'm glad the video was helpful! :)

[@yilinxie2457]

I’m preparing for an interview and this definitely helps!

[@shaz-z506]

After waiting for a while, finally, the series is here. Thanks :)

[@SamirSheriff]

This video came out at exactly the right time!! Triple BAMMM!!!! Thanks, Josh! :)

[@uniwander]

Your channel has the best videos I found for explaining gradient boosting (and many other data science and stats topics). Really appreciate it!

[@statquest]

Thank you very much! :)

[@yulinliu850]

Excellent! Looking forward to future parts of the series!

[@mau345]

great material, thank you :) sometimes, we need to relax, take a step back and look at seemingly complicated things with a simple perspective. I hope more teachers are exposed to this kind of freedom and agility to creatively present their lessons.

[@statquest]

Glad it was helpful!

[@quant-trader-010]

I really like the contrast between different models that are seemingly similar. It helps a lot to clarify the ideas.

[@statquest]

Thank you! :)

[@weemengaitan9106]

Hi Josh, this is by far, the clearest explanation for a concept I was try to figure out for 3 months. Thank you very much! really appreciate your effort ! Keep up the good work !

[@statquest]

I'm glad you finally got the information you needed to understand how Gradient Boost worked.

[@marvellousmoonvalley7085]

Same here. For more than a year, I have been using Gradient Boosting heavily because of its accuracy, without understanding how it works. Now all is clear to me. Thank you, StatQuest!!!!!!!!!!!!!!!!!

[@astaconteazamaiputin]

This is a great example of designing good instructions ! The methods you use for putting the learner in the right mindset for taking in difficult information is just inspirational

[@statquest]

Thank you very much! :)

[@xudongzhang9234]

The Moment I clicked in and realized it is from StatQuest, I just knew I'm gonna have a really great grip on this topic.

[@Josh-di2ig]

thanks for another amazing vid. watching this video alone improves my understanding not only about the model, but also about hyperparameters which makes me better at ML modelling. lucky to have you in the community.

[@statquest]

Thank you!

[@Maya_s1999]

Wow Josh - this is just what I needed right now. Bring on part2!!

[@kaunghtetmyat1704]

I was scratching my head for 2 days to understand the GB algorithm and all the sites and resources on the internet are quite confusing. You just explained it so clearly and BAM...you got a new subscriber!

[@statquest]

Awesome! Thank you very much! :)

[@mingyuli4865]

Every time I try to learn new machine learning algorithm, I will firstly check your channel. You really helped me a lot!

[@statquest]

Awesome, thank you!

[@joerich10]

Josh - another amazing video, well done! Been waiting for this one, looking forward to parts 2 & 3. Thank you so much for your crystal clear explanations - its so refreshing in a world full of confusing, jargon filled stats tutorials. Your videos really stand out!

[@gabrielpradyumnaalencarcos5129]

Loved your videos, man! It's actually makes it's easy to understand the concepts behind the models instead of trying to read their papers, which can take time and some times can be really hard .

[@statquest]

Thank you!

[@ritendradhakciya9935]

Greatest Video I saw on the Internet to understand with this much clarity

[@statquest]

Thank you very much! :)

[@aop2182]

Finally! I have been waiting for you to talk about this since last year. GBM for classification would be more exciting, can't wait.

[@statquest]

Two more weeks to go!

[@adiamirudin430]

This is some godlike storytelling, I know myself sometimes skipped/half-understand the mathematical reasoning behind every algorithm, because sometimes its unbearable, too mathematical. You get the essence and storytelling it perfectly! Thanks man! Tha'ts a subs and bell for you!

[@statquest]

Bam! Thank you very much! :)

[@jasonshen4404]

You are amazing! I have been reading papers and ppts, none of them helped me to visualize the mechanism as what you have done! BAM!

[@statquest]

Thank you! :)

[@Demonithese]

Thank you for all the hard work you do to produce these videos. They are simply fantastic and every one rekindles my appreciation for statistics.

[@statquest]

Awesome! :)

[@MrLilipili]

You are just awesome!!! It’s 2:10 am London time! I have to submit my coursework tomorrow morning! I had to code it from scratch! Thank you so much 🙏

[@statquest]

I hope you get a good grade! Let me know how it goes. :)

[@KishanKumar-cr8hs]

Man !! you are such a gem. Even kids can learn these topic if they refer your videos

[@statquest]

Thanks!

[@arnavgrover8301]

This is brilliant. Simple yet detailed

[@statquest]

Thank you! :)

[@vpee]

Salut Josh et merci! (Just a french person trying to show his gratitude). Man, your method is amazing and you have a good voice. I saved your videos and locked them in a safe for when my 5 y old daughters are ready to hear about stats and ML... Precious gift you've got. Au revoir!

[@statquest]

Merci beaucoup!!!

[@cziffras9114]

Je fais la même chose! I'm doing the same @@statquest , Merci à toi!

[@statquest]

@@cziffras9114 DOUBLE BAM! :)

[@tymothylim6550]

Thank you Josh for this very fun video! I really enjoy watching these videos and I learnt a lot on how this is done! The use of specific step-by-step examples makes a big difference in terms of my understanding!

[@statquest]

Glad it was helpful!

[@moushumisikdar198]

Finding StatQuest channel "results in a small(but hugely significant) step in the right direction'" for ML. Thank you so much Josh!!

[@statquest]

Hooray! :)

[@yurobert3007]

I found boosting conceptually harder to grasp than bagging and random forest. This video explains each step with nice clear graphics with just about right pace! Thanks for the great efforts you have put into making this brilliant work.

[@statquest]

Glad it helped!

[@LucjanKucharski]

You got my heart at the intro.

[@MrKashyani]

You are beyond words Josh. Just love the work you are doing.

[@statquest]

Thank you! :)

[@jhfoleiss]

Awesome explanation! Thank you very much!!!

[@okioking2]

this is incredible! very clear. thank you so much!

[@statquest]

Thanks! :)

[@xuhao7208721]

thank you so much, you are basically not wasting any of your words explaining this concept.

[@statquest]

Thanks! :)

[@NIHIT555]

what a great video.I watched multiple lectures on this from various colleges.it is by far the best

[@statquest]

Awesome!

[@sandeepm625]

very good explanation. First time, I got a a detailed understanding of the boosting algorithm and how trees are built.

[@statquest]

Thanks! :)

[@TheSachinbob]

Hi Josh, I am a Data Scientist and a music enthusiast (drummer). This video has cleared a long time confusion of mine. Thanks a lot buddy. Your songs are also good. :) Keep Rocking.

[@statquest]

Thank you very much! I'm glad you like my stuff! :)

[@pushkarparanjpe]

Having summary at the end is so thoguhtful but extra effort on your part but hugely helpful for the learner! Thanks.

[@statquest]

Thank you! Very few people watch the summaries (youtube gives me statistics that tell me when people stop watching) so I've been debating whether or not they are worth the effort. It's good to know that you appreciate them.

[@ashishjadhav7752]

This was the first video which I've seen from your channel and believe me now i can't get rid of following u.. thankyou for such a nice explanation..

[@statquest]

Thank you!

[@3rl0y]

I was working on this all day, because most examples on the internet only touch the classification side of tree-based models. And now my Messiah, my saviour. StatQuest is the only true way of living and all infidels shall be doomed to misinterpret data for the rest of their mortal existence.

[@Erosis]

BAM

[@statquest]

Thanks!!! :)

[@ailiani]

TRIPLE BAM!!!

[@calvin5371]

So basically while constructing a tree we can take features in any combination? ... like .. and do we have to take all the features?

[@marvellousmoonvalley7085]

@@calvin5371 Decision Tree is a whole subject in itself. The algorithms for efficiently constructing an efficient decision tree are quite established. Sometimes we can ignore certain features when they are irrelevant.

[@DeepakSingh-fo2wm]

I am amazed .. such a complicated topic you have explained so simply.

[@statquest]

Thank you so much! :)

[@sanglierquipue]

Probably one of the best explanation of GB I have ever seen

[@statquest]

Thank you! :)

[@__krossell__]

This is the best explanation of the gradient boost. Period.

[@statquest]

Thank you! :)

[@jmarkinman]

This is probably the best CS/Math channel I've ever seen. Right up there with 3Blue1Brown, Kudos!

[@statquest]

Thank you!

[@Zombie-wx9cq]

statquest is the best ML video out of all on the internet

[@meysamamini9473]

I wonder how can someone dislike a great content like this :/// these videos are amazing!!! even a 5 year old kid can learn complicated ML concepts from StatQuest! THANK U Josh ❤️!

[@statquest]

Thank you very much! :)

[@HKwak]

This man saves my life. I first tried to understand the one by reading the book: Elements of Statistical Learning, the book is too complicated and too much information for beginners. I think it's better approach to learn the cores with the visualization first, just like this video, and mathematical properties then next. Looking forward to watching the next video! Right away!

[@vamsikrishnaj4429]

Well Said!

[@anaswahid8520]

You are LOGICALY CONSISTENT You have depth knowledge of your subject

[@boglu4382]

Watched many machine learning videos from your channel, very nice and clear and easy to follow , thank you ! good luck to my final tomorrow too!

[@statquest]

How was your final?

[@kohei4828]

Another great video! It would have saved me a ton of time to understand how GBM works if I had watched this video rather than going through lots of random online materials.

[@ahmedabbas2595]

You're AWESOME man! your videos are easy to understand and you use your BAMS wisely.

[@statquest]

Thank you so much! :)

[@rreinec]

the clearest explanation for gradient boosting on the internet

[@statquest]

Thank you! :)

[@asabereowusu5854]

This is similar to Random Forest. Great explanation. Great job!

[@pratik3476]

Thanks for painting such a clear picture. It really helped!

[@statquest]

Thanks!

[@deepwebtube]

Too good sir! Nothing can get better than this...namaste!!!

[@statquest]

Thank you! :)

[@rrrprogram8667]

This is becoming more n more awesome josh.... MEGAAAAA BAMMMMMM

[@calvin5371]

So basically while constructing a tree we can take features in any combination? ... like .. and do we have to take all the features?

[@shashankkapoor2828]

I cant thank you enough for this amazing tutorial!

[@statquest]

BAM!!!

[@bernardmontgomery3859]

It is the first vedio that makes clear for me to understand Gbm

[@statquest]

Hooray! :)

[@adsax1903]

this is pure gold, Josh!

[@statquest]

Thanks!

[@sammitiyadav6914]

What would I do without your videos? Thanks a ton!

[@statquest]

Wow, thanks!

[@user-jt7kw4jf9o]

look forward to see the next 3 series

[@prathvikgs4406]

That was a really great explanation ! thank you

[@statquest]

Thanks!

[@maheshgudisa378]

Your videos are awesome..you deserve much more attention from ML community. Hope you reach more heights and explain us many more concepts!

[@statquest]

Thank you very much! :)

[@shubhamborghare873]

oh wow! amazing explaination, amazing, thank you so much.

[@statquest]

Thanks! :)

[@michelpereira3194]

Excellent methodology! You've just gotten a new subscriber.

[@statquest]

Awesome! Thank you very much! :)

[@lalithahemanth4357]

amazing videos. thanks for the playlist. I learnt a lot!

[@statquest]

Thank you!

[@hyejinlee913]

Best video I've ever watched. Thanks a lot :)

[@statquest]

Wow, thanks!

[@sebastian3291]

Wow, you are amazing, thanks for sharing your knowledge in such a didactic way! Greetings from Chile :)

[@statquest]

Hola Chile! I'm glad you like my videos. :)

[@girlthatcooks4079]

Im in love with this chanel, idk how i didnt know about it

[@statquest]

Bam! :)

[@EmilyBoInvests]

i love your channel! Thanks a lot for providing such good resources to the curious learners :)

[@statquest]

Glad you enjoy it!

[@parveenparveen9384]

Thank you. Just Amazing. Made it simple and clear

[@statquest]

Thanks!

[@charlesd4572]

Excellent presentation - BAM!

[@statquest]

Thank you!

[@hongkyulee9724]

I was going to give up to study machine learning, but you save me. I will have to contribute to the world like you. Really thank you for nice educational resource...>

[@statquest]

Happy to help!

[@krocodilnaohote1412]

Thank you good sir! Your videos are amazing, very helpful!

[@statquest]

Thanks!

[@rodolfoviegas8504]

Simple and excelent explanation, thanks for made this class; greeting from Brasil!

[@statquest]

Muito obrigado! :)

[@binbochen1885]

This channel saves my machine learning!

[@statquest]

Glad to hear it!

[@koderr100]

I like these BAMs. Improves understanding a lot!

[@statquest]

bam!

[@41abhishek]

Best Tutorial I have ever seen

[@statquest]

Thank you! :)

[@kennethleung4487]

Great video as always!

[@statquest]

Thank you! :)

[@asimazehra8597]

Thank you for this video...

[@subhz1]

Excellent!!!!! BAM!!!!! great explanation👍

[@statquest]

Glad you liked it!

[@jpeel36]

You are a hero. Thanks for such a clear lecture

[@statquest]

Thank you! :)

[@aly7401]

Thank you very much, I've got the intuition of gradient boosting. MEGA BAM🥰

[@statquest]

Awesome!

[@algokiddo]

Great video dude! Thanks! Keep Bamming!

[@statquest]

Thanks! Will do!

[@marjunodtojan3912]

Great job sir. I am hoping that you will make a video on explaining Frequency Ratio. Thank you.

[@himanikhurana6656]

Wow !!! Amazing Explanation

[@statquest]

Thank you! 🙂

[@hhhhh-pb2ep]

Bro! You deserve the greatest like of all time!

[@statquest]

Thank you!

[@LITHIUMINWATER]

"Then it scales the tree".-Awesome animation and great video as always!

[@statquest]

Thank you so much 😀!

[@edvinsjudins1109]

Thanks, your videos helped me start my professional career! (initially small but gradually increasing in significance BAM!)

[@statquest]

That is awesome!!! TRIPLE BAM!!! :)

[@ghexz7]

You are making my dream of becoming a data scientist comes true. Thank you so much fron the bottom of my heart

[@statquest]

Thanks!

[@erfanmoosavi9428]

Thanks for your interesting videos!

[@statquest]

Glad you like them!

[@harkus8831]

This video was helpful! Thank you !

[@statquest]

Thanks! :)