7.2 Majority Voting (L07: Ensemble Methods)

Sebastian's books: sebastianrasch...
This video discusses one of the most basic case of model ensembles, majority voting. Using a toy example (making certain assumptions), we see why majority voting can be better than using a single classifier alone.
-------
This video is part of my Introduction of Machine Learning course.
Next video: • 7.3 Bagging (L07: Ense...
The complete playlist: • Intro to Machine Learn...
A handy overview page with links to the materials: sebastianrasch...
-------
If you want to be notified about future videos, please consider subscribing to my channel: / sebastianraschka

[@sassk73]

So grateful that I came across this channel through this video! I love how you explain the concepts so well and make them easy to understand by being so calm and composed and showcasing easy to understand examples! Please continue the great work :)

[@SebastianRaschka]

Wow, thanks so much for saying this. It feels really great to hear this!

[@cheukchitse8692]

As a beginner in this field, I think your videos make the whole process quite clear and easy to understand. Thank you so much Professor Raschaka. Keep creating excellent content plz :)

[@nak6608]

omg you have a youtube channel! I've been working through your Python ML book for months now. I've been struggling on page 228 so I went to youtube. 5 minutes into this video I was like, "this guy is using all the same examples from the book" lol to my surprise it was you! This is amazing! I'll have to go back and watch some of your other lectures!

[@SebastianRaschka]

haha small world! The other lectures are a bit different, but yeah, Lecture 7 is closely based on my book :)

[@talsveta]

Thank you so much for sharing the lectures. They are really helpful!

[@SebastianRaschka]

Thanks a lot for saying this! It is very motivating to hear and for producing more videos in the future. (Spoiler: I am planning to make some on Bayesian methods in ML, i.e., Bayes optimal classifiers and naive Bayes, later this year :) )

[@talsveta]

@@SebastianRaschka sounds great :)

[@mehdimaboudi2703]

Many thanks for sharing the lecture video. Page 16: it seems that instead of ceiling function k>ceil(n/2), floor function should be usedm k>floor(n/2). if we have 15 base estimators, then k>7.

[@SebastianRaschka]

oh yes, good catch

[@FarizDarari]

Wonderful illustrations, thanks a lot!

[@wellkamp]

So many thanks to great explication

[@rohitgarg776]

Very nice explanation

[@Jonathanwu_tech]

very helpful video!

[@SebastianRaschka]

Glad to hear!

[@abdireza1298]

Professor Raschka, Please allow me to ask. Is there any theoretical background for what algorithm we can and what we can't combine into a voting classifier? Say, we built a model from the X and y using - Logistic regression with Lasso penalty, - Logistic regression with Elastic net penalty, - Decision Trees, - Random Forest, - AdaBoost, - XGboost for each one showing various accuracy results from stratified k-fold cross-validation. Is it acceptable to create a soft voting classifier (or weighted voting classifier) made from the Decision Trees model and Random Forest? (considering random forest itself is already a combination and soft-voting of several decision trees). Does it acceptable to create a voting classifier consisting of XGBoost and Adaboost? Does it acceptable to create a voting classifier consisting of logistic regression with lasso penalty and other logistic regression with the elastic net penalty? (considering elastic net already a combination of lasso and ridge I understand that we are free to do anything with our data, I believe combining similar models will help at least narrow the standard deviation from the cumulative average of the cross-validation split. But, is it theoretically acceptable? Thank you for your patience. I am sorry for the beginner question. Good luck to everyone.

[@SebastianRaschka]

That's a good question. You can put literally everything into the voting classifier. The caveat is if you have too many not-so-good models though, then it will also drag down the performance of the ensemble. But this is essentially a hyperparameter tuning problem. You try to combine different models and see what works. I recently added example 8 here (rasbt.github.io/mlxtend/user_guide/classifier/EnsembleVoteClassifier/#example-8-optimizing-ensemble-weights-with-nelder-mead) to optimize the weighting of the individual models in the voting classifier. (Btw. later in this course, I also talk about stacking, which is essentially the same concept where we use a logistic regression classifier to weight the classifiers predictions)

[@charukshiwijesinghe5049]

Thank You

[@blogger.powerpoint_expert]

how can i refer to that material? if i want to refer that in my work?

[@diniebalqisay2658]

Hello sir, May I know how voting classifier can be implemented in CNN model approach?

[@SebastianRaschka]

It depends on how you implement the CNN. The simplest way would be to collect the class label predictions and compute the mode.

[@saumyashah6622]

Does all the classifiers have similar weights(in voting) ? If yes then classifier with legit high accuracy would have equal weighted vote as compared to vote by a worse classifier. Please answer me. I am stuck sir. Help.

[@SebastianRaschka]

No, they don't have to have the same weight. So when you look at the code around min 20:00, there is the following line eclf = EnsembleVoteClassifier(..., weights=[1, 1, 1]) You can change the weights to give certain classifiers higher weight than others. How to find good weights? That would be another hyperparameter to tune ...

[@vikramxD]

There seems to be an issue of zip function argument not being iterable in the ensemble.voting classifier method does anyone know how to solve it ?

[@SebastianRaschka]

Hm, are you using the one from MLxtend? Can you share a code snippet of what's not working? E.g., via a GitHub Gist or so? gist.github.com