Gradient Boosting Complete Maths Indepth Intuiton Explained| Machine Learning- Part2

Gradient boosting is typically used with decision trees (especially CART trees) of a fixed size as base learners. For this special case, Friedman proposes a modification to gradient boosting method which improves the quality of fit of each base learner.
wiki link :en.wikipedia.org/wiki/Gradien...
Please join as a member in my channel to get additional benefits like materials in Data Science, live streaming for Members and many more
/ @krishnaik06
#GRADIENTBOOSTING
Please do subscribe my other channel too
/ @krishnaikhindi
Connect with me here:
Twitter: / krishnaik06
Facebook: / krishnaik06
instagram: / krishnaik06

[@vipindube5439]

IF you teach in this way people will become passionate about data science, thanks for your effort.

[@Arjit_IITH]

I am enrolled in an ML online course, but I was unable to understand Gradient Boosting there, but everything is cleared after watching this video. Thank you Krish Naik

[@alliewicklund6192]

You are brilliant, Krish! I've had trouble with the theory and intuition of data science before, but these videos make things so clear.

[@someshjaiswal545]

Thanks for the explanation Krish. If someone wonders why Gamma_m in step 4 changed to alpha at 15:32. It is because alpha is a hyperparameter (something whose value you set), set it between 0-1 and it remains fixed through all iteration from m = {1...M}. In this case however you dont need step 3. If you dont want to set alpha by yourself, want to learn it from data itself, and that is adjusted automatically in each iteration for m={1..M}, use step 3, but from wikipedia link given in description. Modification at 15:32 is done to make things look simple I believe. Awesome explanation. Thanks Again.

[@sandipansarkar9211]

watched it again today.Very important for interviews in product based companies

[@tengliyuan1988]

Thanks Krish, I cant tell how much I appreciate your sharing of the knowledge.

[@karunamayiholisticinc]

Thanks to Professor Leonard's Calculus classes on here that I could understand this. Great explanation. I don't think it would be as easy to get this so fast from Wikipedia. Thanks for taking out time to explain the concepts. Keep up the good work!

[@urmishachatterjee5127]

Thank you for making this video on gradient boosting. I am getting a better understanding of ML from your videos. Thanks a lot

[@pramodkumargupta1824]

wow Krish, you made math equation so easy to understand that it really motives to me to look at equation in different angle. Great Job.

[@anon44492]

amazing bro! i have been trying since months to get my head around this... thank you so much!

[@SK-ww5zf]

Krish -- Fantastic teaching! Thank you! You mention we first fit y to the independent variables, then fit the residual to the independent variables, and repeat that second step. When do we stop iterating? Will there be an iteration after which y-hat will start to deviate away from true y values, and how do we identify that?

[@abhinav02111987]

Thank you Krish for helping us understand many complex algorithms.

[@somalkant6452]

Hey Krish, Thanks a lot for the awesome explaination, just love watching your videos like m watching a TV series, continuously i can watch for 2-3 hours :) One request if time permits can we have video on LGBM and Catboost. There are no good explaination available.

[@vijaymukkala27]

You are doing a great job .. This is bulletproof explanation of the whole algo .Actually you made me inspired in recording my own videos of my understanding which might help me in future ..

[@pushpitkumar99]

Sir you make things look so simple. Really learnt a lot from you.

[@mainhashimh5017]

Krish man I'm so thankful for your work! Passionate and intelligent.

[@ankitgupta1808]

awesome krish you have an amazing ability to describe complex things with ease

[@priyabratamohanty3472]

Nice to see the gradinent boosting series

[@rajeshrajie1237]

Thanks Krish ...Good explanation ..Much Appreciated..

[@akhandshahi3337]

If we are euclidean distance then you are the standardization. You make our calculations very easy.

[@SuperRia33]

Wikipedia scares me(with formulas) but Krish saves me ,Thank you for all your hardwork ,for simplifying complex things and keeping me motivated to learn!!

[@ManishKumar-qs1fm]

U r doing well Sir Awesome 👍

[@saichaitanya9613]

Hi Krish, thanks for your explanation. So the column r11 is the residual we got when we subtracted y^ with actual target value,but in your explanation you said it is the output of decision tree trained with r11 as target. I am bit off here, may be I might have understood in a wrong way. Anyone can correct me :)

[@utkarshsalaria3952]

Thanks a lot sIr for such clear explaination.!!!

[@sandipansarkar9211]

Superb explanation Krish,Thanks

[@Vignesh0206]

Awesome sir 👌✌️..also please do indepth videos for PCA too.. personally heard many people find it little difficult to understand. Please consider this as an humble request on behalf of all

[@himanshubhusanrath212]

Very beautifully explained Krish

[@sushanbastola947]

15:32 The moment when your teacher caught you dozing!

[@amardeepsingh9001]

It's a good explanation. Just one thing: at last (at step 4), what you are referring to as alpha (learning rate), is a gamma(m) actually. It's an obtained coefficient for minimum loss at step 3. However, we can add alpha in a multiple of gamma there, to perform regularization. Just tried to understand from wiki ;)

[@kalppanwala6439]

Wonderful !!! explained like an arrow ie. on point

[@SreeramPeela]

should the learning rate be fixed ahead or change over interations?

[@hiteshmalhotra183]

Thankyou sir for sharing your knowledge with us..

[@satishbanka]

Very good explaination fo Gradient Boosting Complete Maths!

[@thepresistence5935]

nice explanation about this thankyou so much !

[@rkaveti]

I am taking this class cs109a at Harvard and I tell you what- you beat the professor any day. So clear!

[@jewaliddinshaik8255]

Hi krish sir i am following all your videos ,easy explanations .....keep doing same ..thanks a lot sir

[@sagarmunde3088]

Hi Krish. All your videos are really well explained but, can you please upload how to implement the algorithms using code also. so, it will be helpful for everyone

[@abhishekmaharia4837]

Thanks for the great explanation....my question is how do you select a loss function pertaining to a problem or is it like try different loss functions according to different ML models

[@davidzhang4825]

Nice video. What's the connection in Step2 between (2) Fit a base learner and (3) Calculate the gamma using the argmin sum function ?

[@sohailhosseini2266]

Great work!!!

[@anirudhagrawal5044]

hello krish , i have doubt regarding this video only as we use gradient descent technique and find the first order derivative of y^ we equate the equation with zero to find the local minima value for y^ but as we know gradient descent technique is a greedy technique we will never be able to reach best solution or global minima how can we implement gradient descent also and have global minima at the same time?

[@TheR4Z0R996]

Hey krish, I have a doubt, when we update the model shouldn't we multiply the base learner with gamma_m instead of the learning rate alpha? There is this little mismatch from your video and the wikipedia page. That being said, keep up the good work. You're such an amazing guy, 10x a lot.

[@krishnaik06]

Oh yes, well I missed that part..thank u for pointing it out...it helps everyone :)

[@gardeninglessons3949]

@@krishnaik06 sir can u point out the step and rectify it in the comments , thanku

[@mranaljadhav8259]

hey did you got the point means how to update the model? Here our gamma is y^ right? it's like 60+60(-10) ?

[@MrAbhiraj123]

@@mranaljadhav8259 no bro he missed that part kindly check the wiki page

[@shashankbajpai5659]

The explanation is strikingly similar to StatQuest's explanation on gradient boosting.

[@spicytuna08]

thanks. 11:30 - confusion between r and gamma.

[@keerthi5006]

Awesome explanation. I want to know what is the better course to learn Python for data Science.

[@tanmoybhowmick8230]

Sir can you please show a full video on model deployment....

[@srinathtripathy6664]

Thanks man . you have made my day 😊

[@kmnm9463]

Hi Krish, Excellent math discussion on Gradient Descent. I have one clarification and an observation. Clarification : at the start the loss function is defined as 1/2 summation y-y^. Want to know where the 1/2 came from.? Also in calculating the y(cap) in the first base model - it is also the direct average value of the initial dependent variables ( salary). This gives 60 ( the same as derivative route). Why to use derivative in the first step? Regards KM

[@RoamingHeera]

shouldn't it be Gamma multiplied by h(x) in equation 3 (equation 3 on bottom right)?

[@9604786070]

In step 4, h(x) is simply r_m i.e. residual calculated for that DT. Then why use different notation h(x)? And there should be summation over i in last term of eq.4, right?

[@RajeevRanjan-u7z]

Basically, for getting minima of f(x) => we need to find x such that d(f(x))/dx = 0 or f'(x) = 0

[@ppsheth91]

Hey Krish, Can u please upload the remaining videos for Gradient boosting.. Thanks..

[@1pmcoffee]

Hello Krish, I have a doubt: at 14:00, you mentioned the previous value of the model as 60. But as calculated earlier in the video, the latest error was r11 that is -10. So, shouldn't we put -10 instead of 60?. A side note, I am enrolled in Applied AI course but couldn't understand this concept. You made it so much easier. Thank you so much.

[@radhakrishnapenugonda734]

If you have observed the equation clearly, Fm-1(x) is the value that obtained in the previous model. We are trying to find the gamma that minimizes the loss obtained in the present model.

[@srinagabtechabs]

excellent teaching..thank u ..

[@DharmendraKumar-DS]

Great explanation...but is it necessary to remember all these formulas from interview point of view?...or having understanding of concepts is enough?

[@shashirajak9997]

Hi krish. Just a request that whenever u make a video which is continuation of a video (part 2, part3 ) then plz put link of part 1 or last video related to it. This will really help . Thanks

[@lijindurairaj2982]

thank you, was very helpful

[@ThePKTutorial]

Nice video please keep it up

[@willwoodward4150]

How is gamma_m calculated in step 3 used in subsequent steps?

[@rupeshsingh4012]

Hats off to you sir ji

[@niladribiswas1211]

what is the use of gamma(m) in 3 rd step because later you changed the forth step to F(x)=Fm-1(x)+alpha*h(x),but in wiki it is gamma (multiplier) instead of alpha which makes quiet sense

[@rafsunahmad4855]

Is knowing the math behind algorithm must or just knowing that how algorithms works is enough? please please please give a reply.

[@abhijeetjain8228]

thank you sir !

[@yashasvibhatt1951]

In the third sub-step, according to the formula doesn't that always makes things to 0. Since yi will be your original values, y_hat will be the residual, Fm-1(xi) will be the base estimator's value. These values makes it, 1/2(50 - (60 + (-10))) which is apparently equals to 0 and it is not for a single sample, it is for all the sample. Correct me if I am wrong.

[@subarnasubedi7938]

I exactly got the same problem if you minimize you will get y(hat)=y(bar)-60 which is 60-60

[@chirodiplodhchoudhury7222]

Sir please makr the part 3 and part 4 of the Gradient Boosting series

[@avishgoswami2141]

Fantastic !!!

[@parthsingh3473]

Hello I am first year btech student. How much maths is needed for ai. As I am average in mathematics should I choose Ai as my career option Please tell sir

[@dheerendrasinghbhadauria9798]

Is Data Structure and Algorithms same for data science field and software developer field ?? Are OOPs & DSA of software developer field important for data science field as well ??

[@clivefernandes5435]

Well I would say maths is more important . becz most of the algorithms are already implemented in framework like sklearn , tensorflow , but if u have a good math foundation specially in statistics and linear algebra , probability that will take you a long way when reading research papers

[@fatmamansour8606]

excellent video

[@MrKishor3]

hi krish, i've a doubt, you said d/dx(x^n ) is nx^n-1. so it will be d/dx(1/2(y-y^)^2)=2/2(y-y^)^2-1, but you are taking it to be 2/2(y-y^)*-1.please resolve my doubt.

[@abdulbasith7665]

Where did the 1/2 came from? If considering the loss function as MSE then it should be 1/n sum((y-y_hat)**2)

[@danishwais2701]

why is the loss function starts with 1/2. n is the number of samples. are threre only 2 samples ?

[@subhadipchakraborty8997]

Could you please explain the same with a classification problem

[@tapasbiswal6693]

How you gonna implement this equation into python.. Kindly explain

[@ganeshkharad]

that was a good explanation....

[@punithraj5478]

Sir videos on NLP??

[@rashidquamar]

We need to step 2 for m --> 1 to M, what minimum M we should consider ?

[@Abhishekpandey-dl7me]

wonderful explanation. please upload a video on xgboost

[@talkswithRishabh]

thanks sir so much 😀

[@aminearbouch4764]

thank you my friend

[@pranabjena4438]

Could you please make a video on xgboost algorithm.

[@ShashankMR]

will you start deep learning and neural network also

[@dheerendrasinghbhadauria9798]

In India , no research happens during masters or PhD degrees . Masters or PhD degree in india is not of much use . In such a case what should indian students do to become data scientist ??

[@JalalUddin-xy7lf]

Excellent explansion

[@hemantdas9546]

Great video

[@Vishal-rj6bn]

What i think is, learning rate is not the one in update model equation that is our multiplier gama(m). Learning rate is the one that we need to use while computing the multiplier. Since it is used to decide the rate at which we minimize the loss function.

[@mranaljadhav8259]

Can you explain how to update the model ? step 4) with that example

[@nareshjadhav4962]

Very nice explained krish!...can we expect Xgboost after this or when?

[@krishnaik06]

Yes

[@bhavyaasharma9920]

I am not getting the sequence which is to be followed. Is it repeat(1-2-3) and 4 or repeat(1-2-3-4).

[@priyeshdave3799]

Hi, Can anyone please explain me why we took 10 in the step 2.4 for model updation? That is 60 - 0.1(10). As per my understanding 10 was the residual value.

[@roshankumargupta46]

3:25 Why 1/2 sir? Shouldn't it be 1/n?

[@harshavardhan3282]

Should be 1/n

[@tejasvigupta07]

It should be 1/(2n). Usually it's fine to have it as 1/n too but as you can see the loss function ,it is having power of 2 so when we will differentiate it the 2 will come forward and we can cancel it with 2 .In short we use 1/2n to make calculations simple.

[@khushboovyas5932]

Very informative.. thanks sir.. but i have one query here that how do we find optimal number of trees??

[@meetshah7989]

That you have to find using hyper parameter tuning

[@shivadumnawar7741]

Thanks krish

[@jaysoni7812]

you said that there's part 2 will be come as gradient boosting classification, please make it bcz the classification of gradient boost is different compare to ada boost in ada boost it's easy but i found difficulty in gradient boost.

[@Prem-xj7zh]

Could you please make a video on XGBoost

[@akashanande6725]

In step no. 4 it is gamma m instead of alpha as per Wikipedia

[@suvarnadeore8810]

Thank you sir

[@16876]

thanks a lot

[@dkm865]

Best lecture on the mathematics of gradient boosting regression. Thank you so much Krish Sir!

[@anantvaid7606]

Sir, could you make a video to explain which boosting algo is suitable to appropriate scenario?

[@manishsharma2211]

Hello

[@anantvaid7606]

@@manishsharma2211 Hello bhai

[@ArunKumar-sg6jf]

can u make video for light gbm maths intutiton please