Multinomial Logistic Regression with R: Categorical Response Variable at Three Levels

Provides illustration of healthcare analytics using multinomial logistic regression and cardiotocographic data.
R and data files: github.com/bkrai/R-files-from...
Includes,
- steps for preparing data for the analysis
- use of nnet package in r
- calculation of probabilities using coefficients from the model
- estimating probabilities using the model
- developing confusion matrix
- calculation of misclassification error
Logistic regression is an important tool for developing classification or predictive analytics models related to analyzing big data or working in data science field.
Machine Learning videos: goo.gl/WHHqWP
Becoming Data Scientist: goo.gl/JWyyQc
Introductory R Videos: goo.gl/NZ55SJ
Deep Learning with TensorFlow: goo.gl/5VtSuC
Image Analysis & Classification: goo.gl/Md3fMi
Text mining: goo.gl/7FJGmd
Data Visualization: goo.gl/Q7Q2A8
Playlist: goo.gl/iwbhnE
R is a free software environment for statistical computing and graphics, and is widely used by both academia and industry. R software works on both Windows and Mac-OS. It was ranked no. 1 in a KDnuggets poll on top languages for analytics, data mining, and data science. RStudio is a user friendly environment for R that has become popular.
0:00 Introduction
0:01 Multinomial Logistic Regression
4:12 Equations for Calculating Probabilities
8:31 Prediction
11:37 Misclassification Error
13:53 Two Tailed Z-test

[@seeonie]

This is excellent. It simplified a complex process for me. And to cap it, the equation derivation aspect was well-explained. You deserve huge kudos.

[@bkrai]

Thanks for the comments!

[@Abouelela1]

I watch a lot of videos, I read a lot of books and online-article, but your videos are the best all over the internet.

[@bkrai]

Thanks for your comments!

[@samarthshrivastava2258]

You are doing an amazing job by uploading videos related to the techniques which can be used using R. I have watched 7-8 videos of yours and I am feeling quite confident with respect to execution of these techniques. Kudozz to your channel ✨🥂

[@bkrai]

Thanks a ton!

[@flamboyantperson5936]

This was really a very helpful and great tutorial. You have so many awesome videos that I spend my holiday learning from your videos. You have enlighten me through your teachings. Thank you so much Sir. Remember our blessings are always with you and you will always be happy in your life.

[@bkrai]

Thanks for feedback!

[@michellelu7336]

Thank you so much for this! You just saved my life!

[@nasamumusa5044]

Easy to follow your explanation. You've done a great job. How I wish I can find a tutorial like this on Ordinal Logistic Regression. Gracias!

[@bkrai]

Thanks for the feedback! Here is the link to Ordinal Logistic Regression: kzbin.info/www/bejne/p5zMp32wn957paM

[@kalyanasundaramsp8267]

i salute you for sharing the knowledge, you are a great human being.please continue to share

[@bkrai]

Thanks for your comments!

[@chrishanni2779]

I have followed many of your videos, thank you!

[@bkrai]

Thanks for comments!

[@federicopascucci9402]

You have excellent didactic skills. Nice tutorial, well done!

[@bkrai]

+Federico Pascucci Thanks 👍

[@dragon171998]

this video was a life saver for my project thank you!

[@bkrai]

You're welcome!

[@2009saurabhrishu]

Very descriptive video. Each step is explained in detail. Request you to cover other modeling techniques as well. Thanks.

[@bkrai]

+saurabh gupta Thanks for your feedback! I'll be adding more techniques from time to time including based on specific requests. Meanwhile here is a playlist with many other methodologies. kzbin.info/aero/PL34t5iLfZddv8tJkZboegN6tmyh2-zr_T

[@jformoso615]

Very clear! Thanks

[@tewodrosgebregergs8230]

This really helpful video, thank you

[@bkrai]

+Tewodros Gebregergs Good to hear that!

[@cmsaboridoify]

Thank you! Very simple and useful!!!

[@bkrai]

+Carlos Martin 👍

[@tanvirrajput3906]

Sir, thank you very much for your efforts

[@bkrai]

Thanks for comments !

[@samanabraham7379]

Great explanation. Very clear and helpful.

[@bkrai]

Thanks for comments!

[@bkrai]

Here is a link to more recent one: kzbin.info/aero/PL34t5iLfZddvv-L5iFFpd_P1jy_7ElWMG

[@samanabraham7379]

@@bkrai Thank you. I am sure they will be very helpful.

[@bkrai]

Welcome!

[@sunilbobb]

really awesome sir

[@micpax1]

Very good video, thank you

[@bkrai]

+micpax1 thanks for the feedback!

[@shivamg1000]

very nicely explained...

[@bkrai]

thanks!

[@vijaymore1239]

Thank you so much, really helped!!!!!! you explained it very well!!!!! Suggestion would be add few graphs along with ROC curve would be helpful!!!!!! Thanks again!!!!

[@bkrai]

+Vijay More Thanks for the feedback 👍

[@bkrai]

you can now find roc curve video here: kzbin.info/www/bejne/r6GyYneGerCMfNE

[@parthdave2501]

Bro aap mahan ho.... poora assignment ho gaya mera

[@bkrai]

+Parth Dave 👍

[@parthdave2501]

Hey can you please tell me how to find the Beta coefficients for each of the 3 levels? Here it took level 1 as a reference and found coefficient values in reference to it. What if i need the actual values of Betas at all 3 levels. Because if i take reference as the 2nd class the betas change. Any help is much much appreciated

[@TheVectorzer0]

thanks it helped me a lot!

[@bkrai]

+vectorzer0 good to hear that!

[@HamJeong]

Thanks for the videos, these are so great and useful! A question: is there any difference between the model you specify here and a proportional odds logistic regression? It seems like there is none, since you calculate the probabilities per category at 10:10 but I wasn't sure. Thanks!

[@devawratvidhate9093]

Thank you for simple detailed lecture always to the point . sir do you have any interview preparation videos for Data Analyst ..?

[@bkrai]

Following link has several short videos on various topics related to data analyst jobs. It can help with interview preparation. kzbin.info/aero/PL34t5iLfZddv9M5HQNbJkB39Gx7t83rft

[@devawratvidhate9093]

@@bkrai Thank you so much

[@sailingintosunshine]

Thank you!

[@bkrai]

+Markus Franke 👍

[@LoizidesGeorge]

Superb, excellent presentation thanks! Γ [Dr. Bharatendra, whenever you are in Marathasa mountains, Cyprus contact me! I owe you many beers and meals for the time you saved me]

[@bkrai]

Thanks for feedback and comments!

[@roopashastri9908]

Awesome explanation on two tailed z-test! Could you please elaborate on AIC metric

[@bkrai]

You can get some info on AIC from this document: avesbiodiv.mncn.csic.es/estadistica/ejemploaic.pdf

[@ghostshiphost5744]

Is it possible to plot sigmoid curves for the 3 levels in one chart in R? Appreciate your sharing :)

[@nassermohammed5844]

Your explanations are very helpful, thank you for posting it! I do have a question though: for the error table at 3:18, why does the function stop iterating after 3 iterations? Also, what is the error being measured?

[@bkrai]

It measures the log likelihood and the iteration stops when difference between successive iterations is too small.

[@secilgulbudak1277]

Thank you very much this very good explanation, i learned multinom function just by watching your video. I want to ask a question about logistic regression in R. Which link function (logit, normit (pro bit), gompit (complementary log-log)) is used by this multinom function in R. I guess it is logit function but i am not sure. And can we use the other link function? Can we write the link function while multinom function is writing in R?

[@RL-vu8jt]

Im sure its logit link function

[@bkrai]

Saw this today. Hope it is resolved by now.

[@ben88hunt]

Thank you

[@bkrai]

You're welcome!

[@ben88hunt]

Thanks again, the video is very helpful. I am just wondering if you know whether it is possible to obtain a p value to indicate significant overall model fit, chi square value, and a KMO/ Bartlett's test of sphericity using the "multinom" function or can this only be done when the outcome has 2 levels? @@bkrai

[@rajkothari6007]

Nice. How did you select the 3 variables initially from the dataset?

[@bkrai]

I used 1st three variables to keep it simple enough for writing probability equations. For developing complete model, you must try all variables.

[@Blacksam52]

Thank you very much, good explanation. I have a question. In this case LB has p value significant for 2 and not significant for 3. So should be remove LB, if a particular variables has p value insignificant for 50% or more variables can we remove them. Also in case of logistic regression when we remove any variables we watch R square value; in this case should we monitor AIC value?

[@bkrai]

Since LB is significant for 2, we should keep it in the model. Because if you remove it, it will be missing from both 2 and 3. Also AIC can be used here for making model related decisions. Lower value of AIC indicates better model. It helps with goodness of fit and also helps to avoid model over-fitting.

[@hellpumpking]

Do you have some example with this model with random effects? Muchas gracias.

[@jenniferamorim9850]

Thank you for this amazing explanation. what method is used to estimate the parameters in this package?

[@bkrai]

It uses multinomial logistic regression method. It could be that I misunderstood your question.

[@jenniferamorim9850]

@@bkrai in fact I would like to know if the function uses the maximum likelihood method to estimate the parameters. Because when I used the function of the VGAM package the estimates were different.

[@santoshbala9690]

Hi Dr. Rai, We are not getting the coeffeicient of "1" - The refernce level, How do we get that... It is essential we get the Interpretation for that level

[@bkrai]

'1' is used as reference level and the 2 equations are relative to that. So '1' is baked into two equations that we get.

[@guerschommugisho5569]

Nice video

[@bkrai]

Thanks for comments!

[@hanivlog774]

Thanks for the information. you have used three-class labels but a two-tailed z test table showing the values for two-class labels, i.e. 2, 3. Please explain this. Thanks again.

[@bkrai]

With 3 classes we will obtain two equations and level-1 is being used as a reference level.

[@new.challenges]

Dr. Bharatendra, is it possible to apply the k fold cross-validation method to the regression model, where some of the independent variables are categorical?

[@bkrai]

Independent variables being categorical should not matter.

[@carolinaquiros4192]

Thank you for the explanation, Prof. Rai. Can I just ask how do we choose the reference level? Is it the level with the most frequencies? Many thanks!

[@bkrai]

It doesn't really matter which category is chosen as reference as far as developing a good classification model is concerned. It can be chosen as per the convenience of interpreting the coefficients.

[@carolinaquiros4192]

@@bkrai Thank you so much!

[@carlfrio8024]

For 4 dependent variables, do we use only 1 reference level also? Is process similar to the one you did on the video? Thanks

[@galan8115]

Good evening doctor, thank you again for your tutorials. If we want to drop variables out of our multinom model, when can we do that? I got some that are not significant for either of my "binomial within multinomial" models, (I got 3 classes so i got 2 contrast), ¿can i eliminate them of my model since they "do not contribute significantly"? And if so, ¿can i do that also with the variables that only contributes to one of my models?

[@bkrai]

You can remove variables by putting a minus sign in the equation. You may also refer to this more recent one: kzbin.info/aero/PL34t5iLfZddvv-L5iFFpd_P1jy_7ElWMG

[@galan8115]

@@bkrai thank you. Yes i just saw that video :D

[@bkrai]

Thanks

[@amanmehta6562]

hello, how to do this if the independent variables are also factors with more than 2 types of values having no order or levels (eg. work experience - 0-6 months, 7-12 months, 12-24 months, 24+ months or educational background - commerce, science, arts, others)

[@bkrai]

Once you specify that the variable is a factor variable, it will automatically take care of it.

[@balajiks6716]

Sir your videos are awesome and I have a doubt. I am using neiuralnet package for classification and how to develop a neural netweork model with multiple inputs and multiple outputs.( I meant with different formulae)

[@bkrai]

You can use this latest link for multiple inputs and outputs: kzbin.info/www/bejne/npWbYnh-ZsxkmKc

[@jaysethi3736]

Hello Sir, Your video was beneficial. I am running my model on the training dataset, and right now I am getting an accuracy of 67% with the error of 33%. So I wanted to know what should I do to reduce the error. The data was biased which I have made it balanced using ROSE package, and I haven't done normalization so should I do normalization. Because I was reading somewhere that normalization won't affect much. Would be a great help on this.

[@themillitarycampaignsofpes3311]

Excellent video! Just one query. If I want to test mymodel on a different and smaller data set, how can I do that?

[@bkrai]

You can use it the way we do it after partitioning the data set. Here is an example: kzbin.info/www/bejne/l4SUgGt7nqx_msk

[@themillitarycampaignsofpes3311]

@@bkrai But when I try that, this is the error I get: predict

[@aradhyamudigonda4513]

Very informative, Sir. Thanks for this. I have a query, though. I had built a classifier with 11 classes and I want to calculate probabilities for the test data in the Excel file as my test data will be coming in from a new survey in excel and I want to run the data through excel, get the probability in real-time. Will that be possible?

[@bkrai]

Yes, you can do that. Also, it will be better to use csv format.

[@ashokchaudhary8204]

I appreciate your tutorial could you add some other videos

[@bkrai]

You can find all my video here: kzbin.infofeatured you can also subscribe to channel to get update when new lecture video is posted.

[@takheru]

Hi sir, may I know where can i get the appropriate citation for this particular script?

[@bkrai]

You should be able to google it.

[@Monkeyhandle101]

thank you for the video, very helpful. I do have one quesiton and im hoping you can answer. Under the predict part of the video, you insert predict(my_model, mydata). What if i wanted to test on new data points. so predict(my_model,testdata). Bharatendra Rai

[@bkrai]

That's correct!

[@ashrafuzzamanshahriar7701]

But in the test set we always omit the classify variable . SO if i omit the classify variable it shows error :(

[@ranjithnair2659]

Very well crafted and detailed Vedio. I still have a doubt that if I'm to using 64 Variables and 25000 line items after running the model how to determine the best variables. In case of LM & GLM command, summary highlights the best possible variables by ***. What to do in NNET to find the same.

[@bkrai]

You can do z-test and look at p-values to see how significant each variable is. This example shows these calculations too.

[@shuxrat2600]

Big thanks for this tutorial Dr. Bharatendra! When I run confusion matrix in my data: cm

[@bkrai]

You have 'ske' somewhere in your code that it cannot find.

[@shuxrat2600]

@@bkrai . Thankk you for response. Yes, I have ske independent variable in my code and data. But why it cannot find, do you have any idea?

[@bkrai]

Check if Canopy_structure_merged$out should be Canopy_structure_merged$ske

[@willianleal5495]

Very good video! I used your example to run a model, but when i make the Z test comes the following error in R: > z

[@bkrai]

+Willian Leal I notice 'd' missing in standard towards the end. Also 'erros' should be 'errors'.

[@willianleal5495]

+Bharatendra Rai Thanks! My mistake.

[@rohittuteja4489]

Is there any method to choose significant variable or we have to take all variables in model

[@bkrai]

Those variables that are not statistically significant, can be dropped.

[@Protect_the]

Very helpful. I have one question however.....How to increase iterations? What if the model is not converged as per the R-output message after running "multinom" function?

[@bkrai]

+Chandan Kumar you can get more information about the function using "??multinom". I'm yet to come across a situation where it doesn't converge.

[@jaycalvert7911]

+Chandan Kumar You can specify additional arguments from the nnet function. set maxit = 1000 for 1000 iterations and so on

[@YatiChoudhary]

Sir, I have gone through many videos on youtube regarding the logistic regression. But the way you have explained is commendable. I am in the final stage of my PhD and these days writing a paper for publication, I am struggling to build my logistic regression model, if possible would you kindly guide me? How may I contact you to discuss it? I shall be highly obliged by your early response and guidance.

[@bkrai]

Here is my email: [e-mail]

[@YatiChoudhary]

@@bkrai Sir I have emailed you the details with my official email ID with name Pooja Chhikara (Yati is my pet name which I use for informal social media platforms). kindly check and revert back. Thank you

[@marinarondon8056]

Dr. Bharatendra, I have a categorical dependent variable (3 levels) and two categorical independent variables (14 levels and 4 levels). Can I analyze the same way? I noticed that I will not have estimates for all levels like I have one level for each independent variable as a reference.

[@bkrai]

Yes, it will work fine. You may also try this link: kzbin.info/aero/PL34t5iLfZddvv-L5iFFpd_P1jy_7ElWMG

[@marinarondon8056]

@@bkrai thank you!!! Great explanations!

[@bkrai]

Thanks for comments!

[@PramodKumar-cf3tr]

I appreciate your tutorial. I did the test on my data and surprise model predicted 100%. Earlier, when i did the test using regular logistic regression(i.e glm) the predictability was very poor. Even the decision tree, and random forest was not good. Could you make some tutorial for data cleaning esp. if dataset is huge or provide any references?

[@bkrai]

Thanks for the feedback! I'll keep it in mind when planning next set of tutorials.

[@yogitasolanki4638]

Sir please make one video of polynomial regression

[@annazaiceva5861]

Is there a video on how to calculate those three probabilities?

[@bkrai]

See the video at 4:15 point. It shows related equations.

[@carlfrio8024]

Sir, is this process same when dealing with 4 dependent variables?

[@bkrai]

That's different.

[@carlfrio8024]

@@bkrai how is it different sir?

[@OwolabiEbenezerselectedsongs]

I can't get to download the dataset. The link only take care of the codes.

[@bkrai]

Here is the link: drive.google.com/open?id=0B5W8CO0Gb2GGMzJGbkdGUGREYjA

[@tay9001]

Sir, in this example why does there not need to be a split of the data into train and test?

[@bkrai]

You can try this more recent one: kzbin.info/aero/PL34t5iLfZddvv-L5iFFpd_P1jy_7ElWMG

[@tay9001]

@@bkrai Thank you! Would it be fine to perform the multinomial logistic regression either way? One way being not dividing the data and the other way to divide the data into train and test? Love your videos!

[@bkrai]

It is always good to partition the data.

[@RafiTAHasani]

Sir, We are trying to run multinomial logistic regression with many independent variables which are categorical, for example, age & religion (4 categories) and education (6 categories). We are unable to get coefficients and P values corresponding to the category of the dependent variable which is kept as reference level. You have replied to the queries in this regard that R will take care of this once we specify it as factor. We have 2 doubts: 1. Can we create reference levels of our choice for independent variables also? 2. which are the coefficients and P values to be reported for the different categories of dependent variable. 3. How to identify the coefficient and P values corresponding to the category of the dependent variable which is kept as reference level. 4. How can we run a multinomial probit model in R?

[@abhishekbansal5182]

@Dr. Bharatendra Rai sir plz show what is the mathematical functions for finding the coefficients Plz help

[@bkrai]

For that you will have to refer to some book. This video only shows how to do it with R.

[@abhishekbansal5182]

@@bkrai Sir plz suggested some book Plzzzz

[@neroksi5595]

This saves my fucking asses ! Thanks dude

[@soumyendupaul9556]

What's advantage of taking reference of levels?

[@bkrai]

That's needed when you do the probability calculations.

[@basic-statisticsjain6693]

Does the method work for four levels of outcome?

[@bkrai]

yes

[@amolamahadik1]

Hello sir, Thank you so much this video, however My train accuracy is 90% but my test accuracy is 20%only, I think mymodel has overfitted, can you pls tell me how to handle this situation, or if you can make video of cross validation for multinomial regression, it will be a great help. Thank you in anticipation.

[@giorgiogentile8566]

How we can get the coefficient of reference levels?

[@bkrai]

It's not required.

[@hyunjungariuka1686]

i dont get it, what is the number 2 and 3 mean in resression? i mean 2 0.0000....... and the next row 3 0.728...... these 2 and 3 numbers what do they tell and why it is needed

[@bkrai]

This link should help to provide more details: kzbin.info/aero/PL34t5iLfZddvv-L5iFFpd_P1jy_7ElWMG

[@swatisrivastava682]

It is possible to construct with more than 50 response variable ?

[@bkrai]

That should work fine.

[@swatisrivastava682]

@@bkrai Thanks a lot sir !! I will try to implement today and will update whether I was successful or not.

[@bkrai]

great!

[@lakshmigurram9911]

Can u explain roc curves for multinomial...

[@bkrai]

You can only do two at a time. For multi class situation, you can try one class versus all others.

[@soumyendupaul9556]

Can u explain how did u calculate z score? I didn't get it...............

[@winnumber101]

My R has begun to freeze whenever I run the multinom command. It didn't always act this way, but it hasn't stopped doing this for a while even after re-installing the program.

[@bkrai]

probably you may have too many things open. Try to clear any open data-sets or graphs. in the work space.

[@vishnukowndinya]

hi Sir, how can we reduce the misclass error for any model ? one way i learned is by adjusting the cutoff value (p=0.5 or 0.45....). i have built a logit model and measured the appropriate cutoff p=0.46 and used in predicting instead of p=0.5. (pred >0.46

[@bkrai]

When original data has class imbalance, following link has some ideas for improving the prediction model: kzbin.info/www/bejne/fqCVfJ-sr8-Ynck

[@vishnukowndinya]

that's a gud explanation. tq sir :)

[@Monkeyhandle101]

Bharatendra Rai do you have an email i can contact? I have some questions about the data set im using. I would love to pick your brain!

[@abhishekbansal5182]

Sir plz suggest some book through which we can learn core concepts or mathematics of machines learning and statistics So we can write own neural network Plz sir help me Plz suggested book which will help me to understand the core concept. .

[@abhishekbansal5182]

Sir how we get 2 different equation For categories 2 and 3 Both categories has different coefficients Plz sir explain How we get two different equation for 2 and 3 categories

[@bkrai]

I have explained that around 4 minutes and 10 seconds

[@abhishekbansal5182]

Sir actually im seeking its mathematics approach to get the coefficients

[@abhishekbansal5182]

@@bkrai sir plz show what is the mathematical functions for finding tha coefficients Plz help

[@murtazaburhani4022]

Can you make a multinomial logistic regression in r from scratch, please🙏

[@murtazaburhani4022]

With maximum likelihood too

[@bkrai]

Thanks for the suggestion!

[@Manas2]

Thanks for nice the explanation. Getting following error from the link which you shared (dl.dropboxusercontent.com/u/50119580/Logit-binary.R): File not found Sorry, that file doesn’t live here anymore. It might have been moved or made private

[@bkrai]

Dropbox has now discontinued public folder, that's why this error. You can now find files from following link" drive.google.com/open?id=0B5W8CO0Gb2GGN21ab0ZiZmo2UWs drive.google.com/open?id=0B5W8CO0Gb2GGMzJGbkdGUGREYjA

[@Manas2]

Thank you very much.

[@sunilbobb]

really awesome sir

[@bkrai]

Thanks!