Machine Learning Lecture 31 "Random Forests / Bagging" -Cornell CS4780 SP17

[@sameerkhnl1]

Thank you Dear Professor for making these available to us. Not only do you make it interesting, but you have a way of explaining at a deep level, making the concepts so much clearer for us to grasp.

[@smolboii1183]

very true

[@vatsan16]

It goes without saying that you are a great teacher. I also like how you always tell the name of the people who invent these algorithms! :) Makes the class a lot more engaging for me

[@blainemalachi1403]

You prolly dont care at all but does anybody know of a way to log back into an instagram account..? I somehow lost the password. I would appreciate any help you can give me

[@nicolasachilles524]

@Blaine Malachi instablaster ;)

[@blainemalachi1403]

@Nicolas Achilles i really appreciate your reply. I got to the site through google and I'm trying it out atm. Takes quite some time so I will get back to you later with my results.

[@blainemalachi1403]

@Nicolas Achilles It worked and I now got access to my account again. I'm so happy! Thanks so much you saved my account!

[@nicolasachilles524]

@Blaine Malachi No problem =)

[@rezasadeghi4475]

Wow! That was the most astonishing thing I could ever think I would find on the internet about machine learning. Thank you professor for sharing your deep insight.

[@jasemhazbavi]

خایمالو سگ گا**د

[@shrishtrivedi2652]

This is the best lecture series of all ML lectures.

[@puneetjain5625]

You are such an awesome teacher. I laughed and learned simultaneously. Thanks.

[@AnoNymous-wn3fz]

+1 some extra laugh in this particular module :D

[@abdelmoniemdarwish4773]

I don't think I have ever wrote any comments on youtube. Thats my first. Actually just wanted to thank you for sharing these amazing lectures, and for your wonderful teaching mythology and explanations

[@thirstyfrenchie3872]

“Boosting brings me to tears sometimes.” “You gotta eat a lot of fruit before the next lecture.” I love you.

[@newbie8051]

17:54 volunteers 🤣 Thanks prof for the fun and interesting lecture, got to revise these fundamentals quickly 🙏

[@sandipsamaddar875]

Hats off Sir, You are truly a great Teacher.

[@baohoquoc5982]

Your lecture is awesome, Sir. It also brings me to tears 38:02

[@BrunoSouza-wy2et]

beautiful lecture , greetings from Brazil professor

[@JoaoVitorBRgomes]

34:46, you say bias is not a function of H (your hypothesis), it is a function of the average classifier, that's why your bias is low. Could you also explain it is because when you sum uncorrelated errors, to find the mean classifier, they sum up to zero?

[@TheCrmagic]

Prof. Weinberger, Thank you for posting your course online, it has been an extremely helpful and an extremely enjoyable learning experience. Will you post those(or future) recitations online in future? As they would add a lot of value by supplementing the lectures, thereby helping online learners like myself get a better understanding. Thank You.

[@aragasparyan8295]

In the definition of out-of-bag error, what we take usually as a loss function while implementing classification via random forest?

[@kilianweinberger698]

For out of bag error people typically use the squared loss (for regression) or 0/1 loss (for classification).

[@aragasparyan8295]

@@kilianweinberger698 Thanks, that makes sense. One more question concerning the out-of-bag error, you have mentioned it is an unbiased estimate of the test error, I am not sure how to prove that or get some intuition behind that. Could you suggest a reference where I can read about that in more detail?

Dear prof. Weinberger, first of all, thank you for publishing your lectures. They are awesome! I would like to ask you in which way random forests can be used to perform feature selection since each tree composing the forest does not consider all the features; can you explain in which way the feature are evaluated by looking at the trees? Thank you in advance. Best regards, NG

[@abhisheksingla2260]

Prof. Weinberger, When we use bootstrapping, it will duplicate records. Wouldn't that be a problem while training our model? It's like giving more weights to some records. It might lead to biasness. Also, need your opinion in general on whether we should remove duplicate records in preprocessing or not because of IID assumption which I believe holds for all machine learning algorithm?

[@kilianweinberger698]

In general, that’s not really a problem, because bootstrapping treats all samples identically- so in expectation they are all over counted about the same. If you are introducing biases, you are simply not using enough bootstraps. There may of course be issues with some algorithm implementations if they assume that all samples are unique...

[@JoaoVitorBRgomes]

Obrigado! Suas aulas são um presente!

[@doloressanchez3891]

Excellent lecture, thank you very much for uploading and sharing your knowledge.

[@connorfrankston5548]

Very curious as to why sqrt(d) is the standard for the number of sampled features as opposed to something like log(d)

[@maddoo23]

How is the variance calculated for the graph shown in 32:32? Is it just the squared error calculated for some test set (or as shown earlier by the formula at around 27:00)?

[@moumniable]

Thank you for your great lessons prof ! ( From Morocco)

[@Lfmpereira1]

Amazing lecture.

[@rakeshkumarmallik1545]

Hi Professor Killian, You said the estimator in random forest is an unbiased estimator[around 27:00 minute], I am not able to understand why its unbiased , can you explain a bit about the unbaisness. Thanks in advance

[@kilianweinberger698]

It is unbiased because it never sees any of the leave out data (for each point you only take those trees that were trained without this point in the bootstrapped data set). Because of this, the estimate you are getting is the same that you would obtain if you left the point out completely (as validation) and trained that many trees without it. Hope this helps.

[@rakeshkumarmallik1545]

@@kilianweinberger698 very happy to see u replying me personally Professor. Your reply is definitely helpful.

[@AmitKumar-vy3so]

thanks sir!wonderful lectures!

[@bolinsun9565]

Really excellent explanation.

[@frankysama100]

Thanks for the great lecture on random forests!! I have a question regarding training and test errors for this particular algorithm for classification: By design, it seems that the training error (if you'd just refit the trained model on the training data directly) is very very low (~0) - Would it thus be appropriate to use either your OOB error or CV error (if you have the time to do CV) as your training error instead (for which to compare to your test error)? In this regard, if one were to use the OOB error as the model's representative training error to be compared against the test error, would it then be infeasible to compare the training and test errors on most metrics (i.e., precision, recall, f1-score), where only accuracy can be used (because that's how we derived the OOB error)?

[@kilianweinberger698]

Yes, you can use the OOB error as the validation error (e.g. you can stop adding trees if the OOB error stops declining). So essentially you get a validation error without holding out any data from the training process, which is a nice side effect of bagging.

[@omerfarukyasar4681]

Thanks for this great lecture, very helpful

[@Stormdaklak]

Thank for share, great lecture

[@marialuizacantanhedewuilla7309]

Do we have access to the projects that you talk about in class? This is the best machine learning course that I found on the internet and I would love to work out on some implementations.

[@Vishakh_Patel]

Prof. Weinberger, great lecture, couple of questions: 1) can you link material that explain the choice of k = sqrt(d) ? 2) Also in Breiman(2001) he conjectures Adaboost is a random Forest, has there been any advance in that directions? 3) is there a implementation of random forests that are made of ball trees? (I am having a hard time thinking of intuitive substitution for the "Random Feature selection" for this version?

[@kilianweinberger698]

1) You can find the details in Elements of Statistical Learning Theory ( web.stanford.edu/~hastie/ElemStatLearn/ ) 2) Hmm, sorry not sure 3) Probably not. Ball-trees are really a way to speed up nearest neighbor search, which in the end you don't have to for RF. The analogy is more that they are similar tree structures, but they optimize different things.

[@Vishakh_Patel]

@@kilianweinberger698 Thank you very much and is this the only course you upload material for?

[@mohammadshahadathhossain981]

You are a good professor but you could be a great James Bond villain too!

[@vijayshankar9529]

Is alpha a hyper parameter and as far as I understand the boosting is for reducing the bias , so will it lead to overfitting .

[@kilianweinberger698]

Yes, it is. Boosting does overfit eventually, but in practice is surprisingly resilient against it.

[@mimmakutu]

I found some of the texts tends to tune tree depth and k or both for random forrest using grid search. Is it a normal practice?

[@nguyenhuyanh9424]

Thank Prof, very helpful lecture.

[@mrcoolpiano]

Hi Kilian, I have a question on bagging. When sampling our approximate i.i.d. sets D_i why do we not sample with replacement n datapoints and then add to these the datapoints in D. In some sense my question is: would it be sensible to more strongly encode the distribution of D into our datasets D_i much like we did in parameter smoothing? Thanks

[@kilianweinberger698]

If you sub-sample from the training set with replacement your samples are still from the same distribution as the original training set - just the samples are no longer independent. The intuition really comes from bootstrapping, which is a good way to estimate the variance. If you were to make the samples even closer to the training set you would likely reduce the variance of the various data sets and diminish the variance reduction effect of bagging. Hope this helps ...

[@mrcoolpiano]

@@kilianweinberger698 I think I understand that variance produced cancels when averaged over many datasets D_i. Thanks for your answer!

[@globalSentry]

Thanks Professor 😊

[@rockretroing]

Similar question to Uday's question- The recitation by the PhD student expert in Gaussian Processes, that you mentioned in this lecture, would be great to watch

[@kilianweinberger698]

sorry, I don't think it was recorded. :-(

[@mohamedbalabel4370]

well explained, thanks a lot!

[@mia__p]

This guy rules

[@Chevignay]

really you are excellent! big thank you :-)

[@vocabularybytesbypriyankgo1558]

Thanks a lot !!

[@juliocardenas4485]

Beautiful!!

[@tommgn2664]

Hi ! About the bias / variance demo for RF kzbin.info/www/bejne/anaydISAnNZ0hbs ? I understand that the bias is constant cause we just do average of more averages when increasing the ensemble size. Thus, does the variance term converges to the bias term when the ensemble size goes to infinity ? I would have said no since the bias is calculated by averaging over all possible training datasets while the variance term is computed on one particular dataset. Is it correct ? Merci beaucoup ! ;)

[@kilianweinberger698]

You are right about the bias.The variance term does not become the bias term, because h does not converge to the expected label \bar{y}.

[@JoaoVitorBRgomes]

17:30 Omg lol

[@varunjindal1520]

I have a shitty classifier ..... hahahaha

[@born2sly116]

So uhh what is this about…

[@linxingyao9311]

Dear Prof. Kilian, you will be ranked with Homer, Virgil, Dante, and Shakespeare in terms of machine learning lectures.