Machine Learning Fundamentals: Sensitivity and Specificity

In this StatQuest we talk about Sensitivity and Specificity - to key concepts for evaluating Machine Learning methods. These make it easier to choose which method is best for your data.
For a complete index of all the StatQuest videos, check out:
statquest.org/video-index/
If you're not already familiar with the Confusion Matrix, check out the Quest: • Machine Learning Funda...
If you'd like to support StatQuest, please consider...
Buying The StatQuest Illustrated Guide to Machine Learning!!!
PDF - statquest.gumroad.com/l/wvtmc
Paperback - www.amazon.com/dp/B09ZCKR4H6
Kindle eBook - www.amazon.com/dp/B09ZG79HXC
Patreon: / statquest
...or...
KZbin Membership: / @statquest
...a cool StatQuest t-shirt or sweatshirt:
shop.spreadshirt.com/statques...
...buying one or two of my songs (or go large and get a whole album!)
joshuastarmer.bandcamp.com/
...or just donating to StatQuest!
www.paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
Corrections:
7:28 My wording could be improved. The True Negatives for Troll 2 are the people who were correctly predicted to not like Troll 2.
#statquest #ml

[@statquest]

Correction: 7:28 My wording could be improved. The True Negatives for Troll 2 are the people who were correctly predicted to not like Troll 2. Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

[@stefanosmandalas6599]

Thanks for this comment, I was wondering during the video.

[@LockyLawPhD]

Thank you for all the great videos! I have learned a lot from you. Will we see a video on Partial Least Squares (regression / SEM) for ML? There are some videos on KZbin but are all very confusing.

[@statquest]

@@LockyLawPhD I'll keep it in mind.

[@freemind.d2714]

And for those who wonder, yes, "Sensitivity " is "Recall"

[@falaksingla6242]

Hi Josh, Love your content. Has helped me to learn a lot & grow. You are doing an awesome work. Please continue to do so. Wanted to support you but unfortunately your Paypal link seems to be dysfunctional. Please update it.

[@alejandrasevilla2237]

StatQuest should be mentioned in my future Data Scientist Master Diploma. Seriously! Every time I get introduced to a new concept, I come here and.. BAM! it all makes sense.

[@statquest]

Awesome! Good luck with your Master Diploma. :)

[@andyn6053]

@@statquest I have learned more from watching your videos than from my teacher at the university.

[@arvinflores5316]

How's your Masteral now ma'am? I'm starting one too this 2021 :p

[@buytfdip7712]

Same for me with my Bioinformatics master's! He's awesome

[@geosaiofficial1070]

same thing for me. Thank you

[@ShaileshDagar]

Having trained myself on other StatQuest video, I predicted that I would understand this video and I actually understood this video, so that looks like a True Positive right there.

[@statquest]

BAM! :)

[@littleKingSolomon]

It isn't English that u r speaking. U r speaking ML

[@nooreldali7432]

@@littleKingSolomon I read ML as machine language 😂

[@ritikkhatri]

This channel is a gem. Glad I found it!

[@statquest]

Awesome! Thank you very much. :)

[@littleKingSolomon]

Watched confusion matrix statquest, and I forgot to like. So I paused in the middle of this one, went back and liked then came back. Because u deserve more than just likes. But for now that's all I have to offer. God bless u and thanks

[@statquest]

Wow, thanks!

[@nileshsuryavanshi8792]

No one teaches like you, thanks for making such a playlist.

[@statquest]

Thank you! :)

[@DarkLordAli95]

I only wish that I had found your amazing videos at the start of the semester. oh well, better later than ever, so BAM anyway!!

[@statquest]

Thanks! :)

[@PunmasterSTP]

Out of curiosity, how'd the rest of the semester go?

[@adsax1903]

I listen to ur Quests at 2x speed and yet they are super uber crystal clear. god speed man

[@statquest]

Nice! :)

[@alecvan7143]

Your videos are always astoundingly clear and well explained! In fact, you've inspired me to the point of being the first KZbinr I support financially!

[@statquest]

WOW! Thank you very much! I really appreciate the support. :)

[@emmaning992]

Your channel is honestly gold... I used to think stats is a mystery and it didn't make intuitive sense until I met your channel - now everything makes sense and I enjoy using them. Thank you for making these free and accessible, I really appreciate your effort, dedication and intelligence

[@statquest]

Thank you very much! :)

[@aysegulkul1741]

Size ne kadar teşekkür etsek az. Müthiş videolar gerçekten. Bu kadar iyi ders anlatan bir hocam olmamıştı hiç. Başarılarınızın devamını dilerim.

[@statquest]

Thank you!

[@nishantrao5600]

BAM! That's an amazing video. I completely get sensitivity vs specificity now. Thanks!

[@statquest]

BAM!

[@ankitsapra1595]

This is my go to channel for any stats related knowledge. Thanks Josh for this exceptional channel👏👏

[@statquest]

Wow, thanks!

[@amirmohammadjalili2676]

Josh Starmer you're awesome and you're really helping the community so much. Many thanks to you sir

[@statquest]

Thanks! I'm glad you like some of my videos.

[@douglasespindola5185]

That's what I call a great explanation! TRIPLE BAM! Thanks, Josh! Greetings from Brazil!

[@statquest]

Muito obrigado!

[@muthukumarantce]

Always precise and clear video on intuitions. Thank you Josh :) Enjoying Double BAM!!!!!

[@statquest]

Hooray! :)

[@MB-vd6hc]

Thanks, Josh, you explained the concept very clearly.

[@statquest]

Glad it was helpful!

[@umanagaswamy6358]

The prelude is absolutely funny :) And the content is amazing, thanks StatQuest 👌

[@statquest]

Thanks! :)

[@janvichokshi4892]

You explained it so simply ! Just loved this tutorial. Thanks Josh :)

[@statquest]

Glad you liked it!!

[@sonang9495]

easily understood, clearly explained, this channel is precious for newbies, Thanks ad

[@statquest]

Thank you very much! :)

[@RaynerGS]

Amazing class. Way to go! Salute from Brazil.

[@statquest]

Muito obrigado!

[@mohammadsaad7221]

Why didn't I find this channel before?? It's a gem!!

[@statquest]

bam!

[@brpawankumariyengar4227]

All the videos of this Chanel are very well made and are extremely good ….. Thank you so very much ….. I never knew I could get such incredible videos for free …. I will share this channel with All tomorrow.

[@statquest]

Thank you very much!

[@proggenius2024]

I'm lucky to stumble on this channel before my final exam! How cool is your teaching...wow!

[@statquest]

Good luck on your exam!

[@oladamola8690]

Josh, I must confess you’re the best !! I can’t thank you enough 🙌🙌

[@statquest]

BAM! :)

[@gangcheng7322]

you are the doctor who saves my time. many applause.

[@statquest]

Thank you! :)

[@ifmohammed6495]

I really sad that there is no one support such that videos I really think you deserve more

[@statquest]

Thank you! :)

[@xxxsamxxx130]

best explanation i've ever come across.. thanks a lot..

[@statquest]

Glad it was helpful!

[@harshkful]

Amazing videos and explanation ! Thanks ..

[@statquest]

Thank you! :)

[@barath961]

Thank you so much for fantastic lessons :). Love from TN

[@statquest]

Bam! :)

[@haidyabugom3a133]

Great explanation! I would like to tell you that our Prof in the college uses your content and suggests it as a reference to understand ML. Thanks a lot for your effort! Really the best KZbin channel .

[@statquest]

Thank you very much!

[@fabiof.deaquino4731]

Quest on! Happy New Year! 👏👍👏

[@statquest]

Hooray! Happy New Year! :)

[@farzanehjmr7523]

thank you very much for your clear and appealing explanation.

[@statquest]

Thank you!

[@MajedDalain]

such a great explanation and content. thanks a lot

[@statquest]

Thank you! :)

[@saravkumar8058]

Wonderful Mate. Appreciate your work. Your explanation with example is top class!!! Happy that I found this channel. Great work. Thanks a lot for en lighting us on ML concepts.

[@statquest]

Awesome, thank you!

[@saravkumar8058]

@@statquest BAM!

[@adityasaxena3903]

Amazing Content! Keep growing.

[@statquest]

Thank you very much! :)

[@neuodev]

I was alwasy think of Sensitivity and Specifity as if they only can be calculated for 2x2 table. it was a good point from you to apply it on a 3x3 one Thanks!

[@statquest]

bam!

[@punchline9131]

Great BAM for this video!!!!

[@statquest]

Thank you!!! :)

[@Hardson]

Please, keep making these amazing videos!!!

[@statquest]

I will! :)

[@zhspartan9993]

Your videos are awesome, thanks

[@statquest]

Glad you like them!

[@technicalescola]

Brilliant @StatQuest with Josh Starmer.. It is explicitly explained by You

[@statquest]

Thanks!

[@prydefonbah4436]

Gave practical application in a meaningful way.

[@statquest]

Thanks!

[@LorenceLiu]

You are better than all of my undergrad and grad professors :D

[@statquest]

Thanks!

[@JerryWho49]

You made again a great video! Thank you. Maybe you could have mentioned precision and recall (=sensitivity) as both terms are often used, too.

[@statquest]

Thanks! I'll put that on the to-to list.

[@alfonsoreyes9811]

Your vids are awesome!, thank you

[@statquest]

Thank you! :)

[@pratikmishra162]

I am really thirsty and watching statquest! Thank you sire.

[@statquest]

BAM! :)

[@hajarhajar8906]

Thank you for saving my life!

[@statquest]

bam!

[@prachinainawa3055]

Woww! It's very much clear. I love it

[@statquest]

Thank you! 😃

[@chloegu4110]

Great Explaination! The confusion matrix is really confusing... need to go over this video several times.

[@statquest]

bam! :)

[@lightshinichi8056]

This channel is awesome and content is so great ,

[@statquest]

Thank you so much 😀

[@pavoldzama4641]

Hello, my suggestion for video I am missing on your chanell - Harmonic mean and F - score for recall and precision. Or at least when I searched your chanell I havent found video on this topic. Thanks for amazing work you are doing.

[@statquest]

Great suggestion!

[@ejn2178]

To be 100% honest I wasn't a big fan of the intro song hahaha but I gotta admit whenever I see new concept or having difficult time to understand topic while studying I come back to this channel. Great work!! It's sorta addictive now hahahaha

[@statquest]

Bam!

[@aleksandrshipilo4467]

Thank you for the lesson)

[@statquest]

Triple bam!!! :)

[@Yibioa]

Hey, can you make a video about Independant Component Analysis, and how this can be used for biological data, such as gene expression data? Btw, my favorite channel. You rock.

[@dawnmarielagamba3165]

FINALLY - I understand thank you!

[@statquest]

Glad it helped!

[@ChituOkoli]

I just discovered your videos yesterday and I subscribed immediately. Thanks a lot! Could you please make a video on precision and recall? Actually, what I would really like to see, more than just the calculations, is help understanding when sensitivity vs. specificity is more useful than precision vs. recall and vice versa. I imagine that a purely calculations video on precision and recall would be very similar to this one on sensitivity and specificity, but knowing when each statistic is more useful in real life applications would be more helpful to me. You touched on that very briefly in this video when you mentioned which is preferred (random forest for higher sensitivity in detecting heart disease versus logisitic regression for higher specificity in confirming that people don't have heart disease), but it would be helpful if you spend more time on those--to me, it is that kind of application that really brings statistics to life.

[@ChituOkoli]

@techoji No, they're not. That's why I'm asking for a separate video.

[@thotayateeshchandrasai3828]

So good!

[@statquest]

Thank you!

[@hi_10svideos86]

before statquest path was like false positive for machine learning but after watching this channel my path become true positive.

[@statquest]

Bam! :)

[@sanhitasaxena1542]

STATQUEST IS AWESOME!!

[@statquest]

bam!

[@t-ranosaurierruhl9920]

Awesome!!

[@statquest]

Hooray! :)

[@AnkitSingh-fw7xy]

More power to you! :)

[@statquest]

Thank you!

[@stevennye5075]

well done!

[@statquest]

:)

[@phuocnguyenlethanh3104]

I swear this guy release coolest music

[@statquest]

bam! :)

[@mehediazad1780]

thank you so much.

[@statquest]

Bam! :)

[@katwoods8514]

Troll 2 is the best worst movie ever! Love that you included it as an example.

[@statquest]

BAM! :)

[@sumitjoshi7904]

BAM! i love your videos

[@statquest]

Thank you very much! :)

[@maleehahaider8596]

Youre awesome Josh. Keep going! i have recommended your channel to all those bitten by sadistics... ahem! statistics!

[@statquest]

Awesome! Thank you!

[@parzynamea4701]

now i know, when to use which classification model , lovely mates thank you

[@statquest]

bam!

[@hoangminhthai4055]

This video is about the way to calculating sensitivity and specificity. Sensitivity = True positive /( True positive + False negative ) Specificity = True negative / ( True negative + False positive) For the case, we have more than two category, we will get these two numbers according to specific category

[@statquest]

bam!

[@sandipbhattacharjee3304]

Thanks!

[@statquest]

WOW!!! THANK YOU VERY MUCH! BAM! :)

[@wolverine87833]

BAMM!! this guy is a legend.

[@statquest]

Thanks!

[@bashiransari6258]

just awesome

[@statquest]

Thanks again!

[@victoriaogden8322]

What if you are hoping to predict something continuous - like the length of an object: Are sensitivity and specificity still used (perhaps by binning and tuning the sizes of bins)? I guess the metric moves from whether the model was right (True Positives and True Negatives) vs. wrong (False pos. and neg.) to how close the model performed to the true value. Would love to hear thoughts on what you would do to assess continuous models instead!

[@statquest]

When we predict something continuous, we usually use sum of residuals or mean squared residuals.

[@calvinfeng5357]

I rarely see specificity as an evaluation metrics. It seems precision and sensitivity are more commonly used. However, it's very good to know this information!

[@statquest]

I should have a precision video out soon.

[@calvinfeng5357]

@@statquest That'd be great! I look forward to your newer videos!

[@sneezypete382]

Josh Starmer, You rock!

[@statquest]

Thanks!

[@geraldoaugustoguto8442]

Amazing Video. Thanks. What're relations between Precision/Recall and Specificity/Sensitivity?

[@ShivamTiwari-on2kl]

I have spent so much time on this and now I finally understand it. Sensitivity and Recall are the same. They tell us how many of the true values were truly predicted. Sensitivity tells us how many false values are correctly predicted. Precision tells us how many true values of all the true values are correctly predicted.

[@ShivamTiwari-on2kl]

I am sorry if I confuse you further.

[@geraldoaugustoguto8442]

Thanks a million Shivam.

[@liranzaidman1610]

Fantastic

[@statquest]

Thank you! :)

[@mostinho7]

2:10 specificity example from 2x2 confusion matrix Sensitivity is true positives identified by the model / (true positives + false negatives) Specificity true negatives identified by the model / (true negatives + false positives) 4:50 using specificity and sensitivity to compare models Todo continue

[@pranaymehta8471]

Thanks for the video. Truly love it. Quick question1 - You mentioned that if correctly identifying negatives is more important, then we should put more importance on Specificity. But how do we decide which class is positive and which is negative. For instance, I am working on fraud detection problem and only 2% of transactions/examples are rejected. What would be my positive class and negative class in that case? Question 2 - Can we have a video on precision and recall ? I find it very confusing so as which Precision and Recall should I be computing when working with imbalanced classification problems. Should I compute Precision and Recall for Majority Class (accepted transactions) or should I compute these metrics for Minority Class(rejected transactions). Any insights on this would be really helpful.

[@statquest]

First, you have to decide what you want to detect. If you want to be sensitive to the rejected transactions/examples, then you need a method that is sensitive to the minority class. I talk about this towards to the end of my ROC video, so, for more information, see: kzbin.info/www/bejne/apu1c4V6l6-Yo68

[@tanphan3970]

Hello Josh, Thank for directly explain about Sensitivity and Specificity. I read some term about Precision and Recall in Confusion Matrix. And I make confuse about using those concepts when choosing a algorithm for training. When do we take care Precision/Recall and Sensitivity/Specificity? Sorry if my question is in wrong in your flow! Thank you and have a good day!

[@statquest]

First see: kzbin.info/www/bejne/apu1c4V6l6-Yo68

[@ozonefriendlyman]

Love you videos. Correction at 10:16 - The sensitivity for Cool as Ice not 17 / (17 + 175)

[@statquest]

nice!

[@skumarr53]

Thanks for the video. Can you make videos on Bayesian statistics?

[@statquest]

It's on the "to-do" list, but it won't be for a while.

[@arjunkaarat9326]

Hey Josh! Your videos are excellent. Would making similar videos on GLM and probability distributions need significant lift? Maybe it's in there already. I just came across this channel.

[@statquest]

I already have a bunch of videos about General Linear Models, as well as a bunch of videos on Logistic Regression (a type of Generalized Linear Model) and a few videos on statistical distributions. A full index is here: statquest.org/video-index/

[@TheMangz1611]

@statQuest Pls make a video on Precision as well - wth love from India

[@statquest]

Noted

[@ankitsingh961994]

Hi Josh, do you think it would add value to add metrics like Precision, Recall and Accuracy here, since we are talking about Confusion Matrix? I know Sensitivity and Recall are the same, just thought we can mention the term as well.

[@statquest]

I'll keep it in mind.

[@kimtae3446]

Hi Josh, do you have a video about lift chart? Can't find it in your channel :(

[@statquest]

Not yet.

[@allaabdella4794]

Thank you for the video, it's helpful. Can you please clarify the True Negatives ( 7:43). Why would we include 77 and 92?

[@statquest]

Because we are calculating specificity for Troll 2, everything is relative to Troll2. Because those people were correctly predicted to not love Troll2. This makes them "true negatives" with respect to Troll2.

[@vaggelisk9401]

Hi, is there any difference in results if we change rows to Actual and columns to predicted? Great video btw!!!

[@statquest]

No.

[@superfaqer]

Hey Josh, awesome videos! They help me understand the statistics and maths side of machine learning :) I have a question based on your example for the heart disease prediction; is it not possible to just use both models for predicting those 2 separate things? for example using the Random Forest for predicting heart disease and Logistic Regression for predicting free of heart disease? Or does that not make sense. Im farely new to ML and would appreciate an answer :) regards,

[@statquest]

Usually people use several different machine learning methods as an "ensemble" and make the final classification based on votes from each method. If you have a random forest predict if people have heart disease and logistic regression for predicting free of heart disease, then what do you do when the random forest does not predict that someone has heart disease and the logistic regression predicts that they do?

[@superfaqer]

@@statquest Thank you for the quick response :) I see what you mean, that could definitely be a problem. Appreciate the videos and your activeness! cheers mate.

[@bytesizedbraincog]

Amazing one! How does the sensitivity and specificity metrics could be made actionable for more classes or categories (400 Classes) not sure which one to consider the most importance for bulk of classes

[@statquest]

See: kzbin.info/www/bejne/apu1c4V6l6-Yo68

[@geoff1391]

Hi Josh. Let's say one model was better at predicting Sensitivity, whereas another was better at predicting Specificity. Do you foresee any problems or downsides to data integrity to using different models, the model with the highest score respectively? For example, in your 2x2 confusion matrix @4:38, using your Logistic Regression model for Specificity measures and the Random Forest model for Sensitivity measures?

[@statquest]

Not that I know of off the top of my head.

[@geoff1391]

@@statquest Okay, that's good enough for me! Thanks Josh.

[@Samurai_Jack__]

Thank you for the amazing explanation as usual, I have a small question : what is the difference between this metric and precesion/recall metric ... when should we prefer one over the other? because my teachers just mentioned the latter metric

[@statquest]

I talk about that in my video on ROC graphs: kzbin.info/www/bejne/apu1c4V6l6-Yo68

[@Samurai_Jack__]

@@statquest Thank you so much

[@netviz8673]

Sensitivity: true positives/(true positives+false negatives) basically TruePositivePrediction/TotalPositiveCases. Tells about positives getting correctly identified (as positives) Specificity: TrueNegativePrediction/TotalNegativeCases or TrueNegatives/(TrueNegative+FalsePositives). Tells about negatives being correctly identified (as negatives)

[@gauravchaudhari2828]

I really liked illustrations and slides what software do you use ?

[@statquest]

I use keynote

[@Pratik-Kedar]

really nice concept clearance ..........! Keep it up , waiting for more content. BBBBBBBBBBBBAAAAAAAAAAAAAAAAAAAMMMMMMMMMMM

[@statquest]

Thanks! I have a lot of videos coming out this month. :)

[@coolbuddy102]

The 12 dislikers are the college profs going out of business. BAM!

[@statquest]

BAM! :)

[@erick6381]

I love you.

[@statquest]

:)

[@zeynabmousavi1736]

What would be the definition of sensitivity and specificity in the case of non disease? Lets say we need to classify between two classes: class A and class B. Which one is positive which one is negative?

[@statquest]

You decide which one to treat as positive and which to treat as negative.

[@sherryyu1936]

I won't tell you, I am waiting for your neural network videos!!

[@statquest]

I hope to get to those in the coming spring. They are way behind schedule! (almost a year late). But better late than never. I'll start doing neural networks as soon as I'm done with XGBoost.

[@ashutoshpanigrahy7326]

Can the confusion matrix & the metrics be only applied to classification problems? I don't think this would work well for regression, for example.

[@statquest]

That is correct. For regression we have the Sum of Squared Residuals, the Mean Squared Error, R-squared and p-values.