Test Characteristics: How Accurate was that Test?

Does a positive test mean that you have a disease? Does a negative test mean you're healthy? Unfortunately, the answer to both these questions isn't a definitive "yes". How good a test is depends on it's sensitivity and specificity. Learn about both, and why understanding these test characteristics is so important. Next week we'll talk about what this all means in part II of this series, The Bayes Theorem.
The Vermont Study: www.ncbi.nlm.nih.gov/pmc/artic...
John Green -- Executive Producer
Stan Muller -- Director, Producer
Aaron Carroll -- Writer
Mark Olsen - Graphics
/ aaronecarroll
/ crashcoursestan
/ realjohngreen
/ olsenvideo

[@MLEvlog]

Every week this channel gives me such vital and useful information. I had no idea about tests like this- and as someone who has had health problems I'm surprised a doctor never mentioned this to me while I was undergoing tests

[@zoemportlock]

just wanted to say that im really enjoying the content on this channel, specially this weeks as someone who has recently gone through a lot of test and appointments with doctors interpreting the results, had given me understanding and now a different perspective, as always - looking forward to next weeks!

[@healthcaretriage]

Thanks!

[@JBatesProductions]

Your videos are ridiculously informative to watch without it being unengaging. Thank you for these.

[@naminova]

This is really good information! I've been having a TON of blood work done lately, and it's nice to be able to look up whether the tests are specific or sensitive. I've actually had a monospot positive many separate times over the past few years, so knowing that the specificity is nearly 100% explains why my doctors have been so concerned.

[@romantheflash]

I remember doing this during my psychology lecturers when we were talking about diagnosing someone (or even IQ tests.) and how it's just as important for mental health as well as anything medical. Really enlightening and you covered it very well.

[@jklocport]

I can't way for next week; if you explain Bayes Theorem even half as well as you explained sensitivity, you'll have done the world a great service. Thanks for being such an awesome channel!

[@healthcaretriage]

Thanks!

[@BlackRose0555]

really great information and the idea of channel is amazing I hope it get the popularity that it deserve thank you so much

[@themotherbrassica]

Wait a minute... Did you just do an entire video on Bayesian logic without name-dropping it? If so, bravo!

[@CygnusExOne]

I never learn from my mistakes, but I always learn from Healthcare Triage!

[@SyntekkTeam]

I learned this stuff in statistics. Back then we talked about it in terms of power and significance level of tests. It's was interesting having it explained from a medical perspective. I look forward to the next video to hear more about it.

[@MelaniePJones]

This is an absolutely amazing video and I want to say thank you so much for supplying this knowledge! At first I was afraid you were going to be too complex and it would go over people's heads, but nope! You explained it beautifully. I now have more medical knowledge which a) is my favorite subject area, and b) knowledge = power and it never hurts to know just as much about something as your doctor. Being uninformed medically is very dangerous and very costly, in my opinion and experience. Again, thank you Dr. Aaron.

[@blacktongs]

I still go back to this channel years later, mostly for the educational value but you can't sleep on the entertainment value! "Think about a pregnancy test... you're going to figure it out sooner or later anyway." *Cry laughing whilst cramming* Thanks Healthcare Triage team!

[@tanialupin]

Thank you, another brilliant video! As a med student I'm all too aware of society's lack of medical education. Keep educating, you're doing a great job!

[@bassoonatic777]

Excellent explanation of the differences between specificity and sensitivity. I also like the use of probability.

[@JMulvy]

Looking forward to next week.

[@itisdevonly]

Good choice of video topic! Really brings to mind the explanation of Bayes Theorem I read, involving breast cancer rates and the likelihood of false positives.

[@emilybelle2083]

I'm studying for my vascular exam and needed a brush up on Specificity/Sensitivity/accuracy. I tried several videos but this is the only one that gave easily digestable examples. I especially liked the pregnancy example. ^^

[@RobbyOlivam]

Very well done! Keep up the great work!

[@MadaxeMunkeee]

Ooooooh bayes theorem! That's a really good choice for next week. Looking forward to it :D

[@La8Stell]

This is amazingly helpful. Thank you!

[@sarahkatemancini]

Love your videos. Can you perhaps do one on common birth control misconceptions (weight gain, trouble getting pregnant after discontinuation, iud pid, etc etc) ?

[@knobjockey6882]

Awesome, thanks again; as always, I really enjoyed it! Considering the rise of 'gluten-free' foods, and the very small number of people who suffer from Coeliac disease (spelt 'Celiac' in some countries), would you guys consider doing a show about that subject? Is 'gluten-free' really beneficial for you if you don't have Coeliac? What about people with sensitivity to gluten without having Coeliac? What about the health benefits, if any, from abstaining from gluten? And finally, what about the health costs, such as the higher fats and sugars that are found in gluten-free foods?

[@cand33cane]

This is the first time I have actually understood this concept. Thank you 🥳🥰

[@brittanyestrella8714]

This was sooo helpful, thank you ☺️

[@michellechen5253]

coooool!! I am recently reading Baron-Cohen's paper and I encounter these two new concepts which confuse me a lot! Thanks for such detailed explanation!!! Amazing!!

[@Feinebea]

As a medical student I highly appreciate this. Great explanation!

[@PutuDharmaMahaYusa]

Very clear and simple, I just learn this subject

[@dimbulb23]

Good stuff and excellent real-world examples. Even I understand it.

[@sakuradeva555]

Thank you this was pretty awesome!

[@therealdrag0]

Every video you did is excellent! Need more subscribers!

[@hatorigirl1202]

This really would have helped me understand that concept in statistics. It was one of the few I had a hard time following.

[@JuJuLiams]

Holy shit, I was literally *just* thinking about Bayes Theorem as a way to assess the probability of testing positive or negative just as the end title came up!

[@GregTom2]

Time for an exam question: I am a spanish inquisitor trying to discriminate real witches from false witches in a gypsy village. As we all know, witches burn, and wood also burns, which obviously means they both float. As we all know ducks float as well, therefore if the woman is lighter than the duck, she is a witch. Question 1: What do you assume is the specificity of this test? Question 2: Unsatisfied with the amount of witches burnt, I measure them on a scale where the duck side is made out of heavy rock to make it heavier. Is the specificity going up or down? Question 3: What is the sensitivity of my test? (trick question) ... I'll give the exam correction when I get enough answers.

[@QvsTheWorld]

Spanish Inquisition? I wasn't expecting this!

[@GregTom2]

But seriously guys, answer the questions

[@sammichbread]

f in chat for this guy, he never got his answers 😔

[@dbmeboy]

Would have loved to see positive and negative predictive value covered here as well... but perhaps those deserve a video of their own.

[@kathrayres]

This video is extremely informative. I never knew that medical tests were measured in this way!

[@atheeralmutairi8102]

YOU SAVED ME THANK YOU SO MUCH

[@IDecisive]

Took me a couple of watches to feel like I understand at least 90% of it. Great video, thank you.

[@carriehope1641]

Ok I need to know where you got the pill-shaped candy dispenser and your art of the head/brain. I enjoyed the video too!!

[@brycepatties]

So would a good plan of action - at least to cover all the bases - be to have a very specific test followed by a very sensitive test?

[@emiliatan2509]

Clear and interesting!

[@mfelixd1]

Thank you.

[@kennyc002]

SNout, SPin is usually how we memorized it. Sensitivity is useful for ruling out something. Specificity is useful for ruling in.

[@kennyc002]

One way of dealing with this issue is to have 2 different tests. One that screens well and has a high sensitivity (so that people who don't have the disease won't have to worry), and then one with a high specificity afterwards (to confirm if disease actually exists).

[@EthanMagnuson]

When are you going to do the analysis of France's system? I was looking forward to that.

[@pcljet]

This seems like it would be a useful video to show in a statistics class.

[@brandonfrancey5592]

Just one question, what happened to Healthcare system of France? Was there a mix up somewhere or was it posted in French on your French channel I don't know about?

[@healthcaretriage]

It's coming!

[@tanialupin]

And I might confuse people, but there are also two more things we calculate from the table, the positive and negative predictive value, which is how many of the positives are true positives (a/(a+b)) and how many of the negatives are true negatives (d/(c+d)). That means that while a test might be sensitive, that is most people who have the disease test positively), it might have a low positive predictive value, that is too many people who don't have the disease test positive. That might make the test impractical. Similarly for negative predictive value!

[@AboALjazy1]

Excellent

[@TBSkyen]

Lot of room noise/reverb in this one. Might want to check up on the audio setup :)

[@geniusmp2001]

This looks like a job for Bayes' Theorem!

[@AjaxEcthelion]

Good stuff.

[@LZKS]

Thanks for the educational insight. Learned something crucially important today. Should be taught in high school biology, but obviously grade school education system of the world (not just USA) is a fail.

[@ZeMole420]

oh that was awesome!

[@NethDugan]

I'd guess this is why our lab runs HIV confirmations through a different test than the first one, more info to base results on. Not entirely sure (not a BMS) but would make sense.

[@bi1iruben]

Yes, two different approaches to reduce false positives on the initial test. 1st test is with ELISA test and then as a 2nd test Western blot. The first test has a sensitivity of 99.7% and specificity of 98.5%, which is good but considering implications of get it wrong the 2nd test takes us to a false-positive rate (i.e. positive result in those who do not have the infection to 1 in 250 000 which means about 99.9996% of positive results are correct)

[@healthcaretriage]

Yep! One is likely sensitive and the other specific.

[@courtneyreneelane7929]

Positive and negative predictive values next then?

[@ParadoxicalParadigms]

So what happens when you combine the data of both high sensitivity/ low specificity test and a high specificity/ low sensitivity test?

[@KarolaTea]

Thanks for the math lesson! ^^

[@nicolademaio6899]

I have the impression that the WBC count threshold examples are swapped: shouldn't one increase the threshold to get more sensitivity and less specificity?

[@megamax888]

This is amazing! Does anybody know if Aaron Carroll is someone's actual doctor? If so what specific type?

[@healthcaretriage]

I'm a pediatrician

[@megamax888]

What is the weirdest or most odd question a child has asked you?

[@DetrimentallySane]

Would you consider making a video on chronic illness?

[@jpheitman]

OH MY GOD.... The relationship between specificity and sensitivity... It's the Heisenberg Uncertainty Principle. ...Except different...

[@Grungir2]

I would like to see an episode about vitamines ,are they safe? Should we take them, when should we take them ,how many and for how long.... I hope you would consider it .keep up the good work

[@CoJau911]

he answered this question in the Q&A video he did with John Green. I'll let you check it out but he said vitamins basically make expensive urine with no health benefit...

[@Grungir2]

okay thanks i will check it out

[@Nailkita]

Grungir2 Basically, if you don't eat properly a multivitamin is good, but there's no point to over due it because having excess your body just flushes out. (though excess of some vitamins/minerals can be bad for you as well). I'd like to see more information though for vitamin deficiencies, as a lot of my family has b12 shortfall and have to take shots since b12 vitamins don't do what you need (so the doctor told them) and I know I've been told to take a LOT of vitamin D from my doctors due to my lifestyle, depression, and lupus. Do specific vitamins really do anything? Fish Oils Omega 3s are they worth it?

[@Grungir2]

Nailkita i just wanted to post something like that ,he said if you eat normal meals and have normal weight you will just pee them out and i agree ,if you are healthy you dont need them but i was actualy interested in the stuff you said + for example people that lost weight due to illness or depresion or eating problems ,do vitamins help? im actually taking B complex for my depression :P and i can say for a fact that it helps to calm things down for a while but do they have a long therm effect ?

[@otter969]

Should have made this video a month ago when we went over this in my Stats class. Good video though.

[@jordanj5877]

I want to know where to buy the plastic pill capsule in background.

[@MelodyRaindo]

Can you please do healthcare of New Zealand and if supplements are effective or not?

[@JohnWGermanProject]

Could you do an episode on second-hand smoke?

[@zelda12346]

Got confused with the jargon. I'm used to referring to them as Type I and type II error or Alpha/Beta level. You could have also used the results of a test to give seemingly subversive ideas. Using the cancer example, the probability of women in the sample who had cancer given that they tested positive was only 4.4%. The probability of women who had cancer even though they tested negative was 0.9%. Only about 4 out of a hundred women had something to worry about, and about 9 out of 1000 had a nasty surprise later down the road. If we wanted to take this a step further, we could apply this to population with a margin of error on each probability of roughly +-0.1% with a confidence level of 99.7% (very confident we got dis down). Definitely a good study to use. Only one problem: we don't actually know how many women had cancer except through other tests that also posses type I and II error. Without cross-referencing multiple tests such that the probability of the final result giving an incorrect answer being very, very small, the only thing we can rely on is the significance level and a power CURVE (not a singular number for Type II error) which gives the power/sensitivity of the test as a function of how many people actually have cancer. This is somewhat moot because the study obvious cross-referenced it when publishing the findings on this test, but I'm point out that the information given in the video didn't come out of thin air. We take it for granted but the fact of the matter is that we don't know for sure how many people had cancer (until we could extract a tumor or something) and the numbers given are just our best guess. That's both the beauty and nightmare of statistical analysis.

[@zelda12346]

Additionally, the video implies that Type I and Type II errors are binary trade-offs. This is not true. If you want a low alpha but also a low beta, then your option to reduce beta is restricted to having a low standard deviation in the sample. You can naturally lower your standard deviation by having larger sample sizes. However, sampling spends resources. Larger samples spend more resources. In an ideal world, you plan out your alpha and beta beforehand then math out what the minimum sample size is to achieve those goals. However, a lot of the times, you are given a budget and must do it backwards: use the money to grab a sample size and then project the errors as functions of each other and then make decisions there. too math; didn't read: trade-offs are more accurately specificity, sensitivity, and cost of test, not just the first two.

[@ranueltiota]

Menea6587 In a lot of instances I think small sample size could be attributed to the disease being tested. For a large number of diseases the amount of patients infected is probably low enough that a good statistical sample may not be available.

[@jakemcclure]

Z-Statistic go watch the Bayes theorem video. Sensitivity & Spec are independent of the disease prevalence.

[@zelda12346]

1. Dear God, was I that annoying in undergrad? 2. What does independence have to do with anything?

[@metabeard3788]

I have a completely unrelated question. I recently encountered a few people who refused to eat or drink anything that had been microwaved because they believed that it alters the molecular composition and thereby more likely to cause cancer. It sounds crazy to me, and I've read a few articles that seem to disprove it, but was wondering if you had any experience with this, or knew of any controlled tests that might dis/prove it.

[@gmosphere]

Ooh Vermont, a friend of mine is from Vermont

[@cathdaws3396]

How many times did he have to practice saying specificity.....

[@somewony]

How do you determine how many cases are false positives and false negatives? You can't know for sure whether or not the patient has or doesn't have the disease, because that's what the test is for.

[@C0nc0rdance]

In practice, we determine the specificity and sensitivity by comparison to other tests, called a test concordance. Hence my username. This is a practical matter. Most diagnostic tests have been compared to direct observation of a clinical definition of the disease or pathogen. For example, I could develop a molecular test for leprosy (Hansen's disease), and use known samples collected from Hansen's patients, or use reference samples that have been cross-tested by a different test with a known specificity.

[@somewony]

C0nc0rdance Thanks! One question though: Did you come here because i mentioned you in a nother comment or is this just some freaky coincidence?

[@healthcaretriage]

You can know by a better test. Or because later it becomes obvious.

[@RaineAvina]

Is there a way to determine the threshold that would maximize both sensitivity and specificity at the same time? That way it's not super lop sided. Probably would be best for tests that have an already great rate of true positives and true negatives.

[@Paulginz]

You can take the value for which sensitivity=specificity. The best approach is usually to minimise a loss function, i.e. quantify how bad/expensive false positives are compared to false negatives and minimise the expected loss. For example, in criminal court convicting an innocent person is considered much more harmful than letting a criminal go free.

[@gorms13]

Should have tossed out a Sci Show plug on the odds of predicting anything. I've gotten a lot of mileage out of the mammogram example at bars and dinner parties.

[@xiomoreira-ruiz7904]

These concepts are covered in Statistics. We would call the false positive a Type I error and a false negative a Type II error in the "Fakitis" example.

[@michaelmolter6180]

Interesting that this is less relevant for COVID-19 testing where the prevalence is so high. It makes the specificity and sensitivity metrics perform more intuitively.

[@sanadbenali6993]

what about likelihood ratio

[@FNPetersen]

Any chance you could do a video about PET scans? I'm trained as a nuclear operator, so I know about radiation. I know not to be scared of most radiation medicine. However, it seems to me that positron emitters might cross a line. Seems to me that antimatter radiation just has so much more potential to be ionizing and potentially harmful. Any medical insights on what makes the procedure safe would be appreciated.

[@DavidSilverGames]

A query. Flouride. I've heard and read a lot about its toxicity and its dubious use as a dental reinforcement. Can you shine the light of cold hard science upon this subject? Thank you!

[@InorganicVegan]

Oh, conspiracies. Fluoride is a dental reinforcement. It's also used to keep water from freezing in your pipes in winter. The more stuff dissolved in the fluid, the lower the freezing point. It's why people put salt on ice. Here's a link about colligative properties. en.wikipedia.org/wiki/Colligative_properties

[@DavidSilverGames]

Diana Peña Sure... except most of what you said can be done by anything. Anything that dissolves can help my pipes. I'm not worried about my pipes, they can be replaced. I'm more worried about me. I'm harder to replace, at least in my very self-centered opinion. So, Flouride! Bring in the science!

[@InorganicVegan]

David Silver It's not bad at the levels used today. www.livescience.com/37123-fluoridation.html Also, it's not just your pipes. It's the pipes of the entire country we're talking about. If those go out, we're all fucked.

[@DavidSilverGames]

The article you linked does not shine a very flattering light on flouride when read through.

[@itisdevonly]

Putting fluoride on your teeth = helpful ingesting fluoride = harmful (with the caveat that the dose makes the poison, of course)

[@yourbabyscorpse]

Will there be an episode about the Simpson's paradox?

[@Xsinthis]

Anyone else find the audio quality kinda poor at some parts of the video?

[@Xsinthis]

Bob Tahoma It's weird because the Green brothers' videos are usually top notch, I wonder why this channel/series is the exception (with regards to audio)

[@FistyMcBeef]

Please do an episode on medicinal cannabis!

[@healthcaretriage]

Wait for it..

[@meedoamr]

You are the real mvp

[@pjw2000]

Are you still going to do a video on the French healthcare system?

[@healthcaretriage]

It's coming!

[@AnthonyBrusca]

Where is the healthcare system of France? What happened to the series of healthcare systems across the world?

[@healthcaretriage]

It's coming, I promise!

[@edoist16]

Hey Healthcare Triage, with more places in the US and around the world legalizing cannabis, could you look at the myths and realities of the drug? Is it better health wise than alcohol or tabaco? Thanks! :)

[@sion8]

Where is the French Health system video!

[@TheShabooka]

i never understood this although i studied it 50 times at the most but now it is as clear as fuck thank you so much love u man

[@Enke796]

Please make an episode about heart diseases.

[@jakemcclure]

Enke796 check out this week's episode

[@FPSchazly]

loving me some healthcare triage

[@jenacorn]

As I understand it, the moral to the story appears to be: tests are a tool, not an end-all be-all; which is fair.

[@SpySappingMyKeyboard]

Seems like there is an error coming from labeling tests as 'positive' or 'negative'. For the fakitis example, it seems we could just get the number of white blood cells back, with a probability - you have 18 WBC so you have a 80% chance of fakitis, you have 11 WBC so you have 5%, etc (making up the numbers here, but I hope you get the idea).

[@jakemcclure]

SpySappingMyKeyboard Nope, the statistics are well-designed. You're looking for the math that will be explained in the PPV and NPV video.

[@SpySappingMyKeyboard]

Maybe error was a bad word to use - ironic considering I was criticizing the choice of words :P I didn't mean the mathematics was wrong, just that how the results were communicated could be improved! What I mean is that if instead of saying 'your mammogram came back positive', if we instead said 'your mammogram shows a 70% chance of cancer', it would eliminate this problem (or error).

[@jakemcclure]

SpySappingMyKeyboard that's not how these 2 test statistics work. They're a %. But they're not a "% risk". You have to know how common the disease is in the population to calculate predictive value of the test. That is the PPV and NPV video I was referring to.

[@SpySappingMyKeyboard]

Yes. What I'm saying is that the way that these stats are presented causes confusion. If instead of having these stats we had the % chance that we have the disease, then this confusion would not exist.

[@jakemcclure]

please just watch the PPV and NPV video. sensitivity & specificity statistics are helpful for certain scenarios and mathematically represent something slightly different than what you're looking for. % chance of a disease requires that you know the prevalence of disease. Sensitivity and specificity calculations do not involve prevalence, thus you need to utilize different formulas to calculate the statistical "% risk" that you're looking for.

[@malaney2004]

No cliffhangers!

[@fernandojackson7207]

I volunteered to administer mammograms. But I was ultimately told I need to ask for permission first.

[@Aetherdynamo]

Oh god I think I have fake-itus. What should I do doctor Dan!

[@sherrivance1254]

I LOVE Healthcare Triage.

[@herpaderpidyderp]

After watching a lot of your videos, i feel like you should have been a statistician not a doctor. You even talk about things like bayes rule, which while applicable to medicine is really a statistics thing

[@christianmatthe1]

Stochastics* Triage :D

[@markruffruffalo9923]

If tests aren't 100% accurate, how would we know who the false positives and

[@Imbalanxd]

This creator needs a subbable profile

[@healthcaretriage]

Hmmmmmm.

[@in2horses247]

Can you discuss whether hair and skin hygiene products (such as shampoo/conditioner/underarm deodorant) are cancer-causing? Or maybe if there is a link between bra wearing and breast cancer? All the conspiracy theorists that I'm friends with on Facebook love to post "studies" about these issues.

[@jakemcclure]

in2horses247 wise of you to remain skeptical of their cancer-causing studies

[@TheWolphren]

Who's betting next week deals with serial testing?