Do USMLE scores *really* predict patient care outcomes?

[@EndoExcision]

Sir, it’s not called journal club because “journal” is the type of club you use to beat an author into a pulp. Another banger!

[@sheriffofsodium]

😂 I warned you this wasn’t your program director’s journal club.

[@npvaughn]

I wouldn’t hate journal clubs so much if they were all like this

[@janetpurkey]

Logical, yes. Thank you for speaking the words most of us think but can’t articulate so precisely. Follow the money and there you have it.

[@misteratoz]

As a hospitalist, I chuckled many times throughout this and I thank you so much for making this video.

[@sheriffofsodium]

Thank you for watching.

[@ifrqi]

No hospitalists in a paper about hospital outcomes? I find that to be perfectly reasonable if the purpose of the study was “make USMLE print more money”. Great video as always

[@GREGhere15]

Stopped at 7:30. Thanks for a great video!!

[@sheriffofsodium]

lol

[@theotherredmeat]

Using the "attending of record" for an up to 40 day hospitalization in a tertiary care center is laughable. The actual attending could change 15 times in that period.

[@sheriffofsodium]

So I take it you don’t believe in The One Doctor (33:06), whose omnipotence and total responsibility is enshrined in the PHC4?

[@gaurav.raj.mishra]

Can we have more journal club videos? You make it very entertaining, with the logic, the jokes and the sarcasm. Very educational too. Teaches critical thinking and research methodology; things the world could use more of.

[@sheriffofsodium]

Thank you for the kind words - and I will try. I actually have a few papers on my desk that I’ve thought about (or am still thinking about) turning into Journal Clubs… I just get pulled in other directions.

[@BurntToast44242]

Can't wait for the following "MCAT correlation with good patient outcomes" funded by Kaplan.

[@theotherredmeat]

"ACT/SAT scores tied to better patient outcomes" "Performance on multiplication tables tied to better patient outcomes" "APGAR scores tied to better patient outcomes"

[@mc_dibia]

😂 not the APGAR scores​@@theotherredmeat

[@olive_n]

I'm so confused by the decision to exclude doctors who identified themselves as hospitalists. The best logic I could imagine is that they were only interested in patient outcomes for community FM, and they used the PHC4 dataset as a surrogate because, theoretically, quality of primary care correlates with outcomes in the hospital. But then the results would be confounded by the quality of care received IN the hospital, so why add the extra steps? Maybe I should've stopped listening at the seven minute mark after I got my dopamine...

[@DrAyeshaJB]

If nothing else, one should remember the maxim "Correlation does not imply causation" before citing/retweeting/extolling any study, no matter which discipline it belongs to. Your video analyses are always spot on Dr Carmody 👍👍

[@sheriffofsodium]

Thank you for taking the time to watch.

[@vistaiscool2]

Fascinating watch. Would you be able to share any resources for those who want to appraise literature in a similar fashion? I believe it's a skill that is becoming more important in this social media age.

[@sheriffofsodium]

The BMJ has a lot of good stuff - e.g., bestpractice.bmj.com/info/toolkit/learn-ebm/understanding-statistics-other-resources/

[@noicedrinkzv2.015]

He never misses- sheriff is back in town

[@ZVMed]

The hero of Henle returns. Awesome vid!

[@sheriffofsodium]

Thank you for taking the time to watch.

[@LJ-cp6qs]

Thanks for making this video. I didnt get a 280 on step 2 and at times I feel bad about it, but this certainly puts things into perspective.

[@dagozasteroide]

Transparency left the chat when is related to NBME/USMLE.

[@Aaron-cc7yq]

Shout out to sketchy micro! Lol. Imagine a patient seeing a doctor looking at a cartoon drawing to figure out their health management 😁

[@jjonsolomon]

I kept suggesting to my program to split into 2 groups to admit: 1 by usmle score, and 1 randomly to try to study a similar thing. Of course, it was never listened to seriously.

[@beccaburrington9196]

11.2 patients over 3 years. It is incredible to me NBME was bold enough to publish this study. I think I would be so embarrassed to have my name attached to this Swiss cheese. What I thought you were going to bring up was blending all 3 USMLE scores together instead of analyzing each of the 3 separately. I would love to know what happens when you compare step by step. I'm guessing not all 3 steps were "statistically significant" in their model if they wanted to just throw everything in a blender.

[@austinbradshaw3636]

For a physician with even a small daily inpatient census, pneumonia alone should easily surpass 11.2 hospitalizations in a year let alone 3 years. Unfortunately laymen and healthcare workers long removed from inpatient medicine will not be familiar with this horrible flaw in the study.

[@sheriffofsodium]

On the one hand, you could try to justify the composite USMLE score by claiming that the goal of the study is to provide validity evidence for the USMLE program overall. On the other hand, it’s hard to hypothesize that performance on each Step has an equal, 33% contribution to patient outcomes when the tests are taken at different times and cover different content. My suspicion is that the composite score was used to minimize differences between US medical graduates (who do not prioritize Step 3 scores) and international medical graduates (who often do).

[@gaurav.raj.mishra]

​@@sheriffofsodium yeah there's no way Sonny, Michael and Fredo have the same significance

[@sheriffofsodium]

@@gaurav.raj.mishra You just ratted on yourself for being a regular viewer. (Thank you for watching!)

[@celsomiranda6293]

Excellent video and information. It will be better to use a Instantaneous Hazard Ratio (HR) Model instead of an OR (RR) model for this outcomes & purposes. An OR of "2" (event is 2x more probable by one-unit increase in the "predictor"); an Instantaneous Hazard Ratio of 0.8 (the event will occur "twice" (1.6) as often given a one-unit increase in the "predictor". HR model will analyzed better the impact on survival analysis (Kaplan Meyer RR: "Time-to-Event" Outcome.). Thank you for your videos!!

[@mukwellekong-mbonekarlaach1941]

Beautifully done.Another Rémi der not to take things at face value.

[@sheriffofsodium]

I appreciate you taking the time to watch.

[@adamisbetterthanluke]

I absolutely love your videos and have been a long-time subscriber--any plans to make a podcast?

[@sheriffofsodium]

Thank you for the kind words and support! I’ve thought about it… honestly, these videos are really just podcasts with a few visual aids. But the audience here is growing - and I think I’m reaching the kind of thoughtful, curious viewers that I most want to reach. And I have only a limited amount of bandwidth to make videos/posts/social media posts/etc. so I have to try to be selective in where I put my energy. So at the moment - no. But I might revisit that in the future.

[@adamisbetterthanluke]

@@sheriffofsodium Understood! Honestly, I wonder if you could save the audio and just release it concurrently as a podcast? Either way, I'm sharing your videos with friends and telling them to watch/listen as though they're podcasts. Thanks for your hard work, SoS!

[@rojanamjadi6281]

Outstanding program. Thank you

[@isaacjamestea9652]

Ah yes another Sheriff video. Now my weekend is complete. I'll finish my late EPIC notes and procedure reports later.

[@sheriffofsodium]

I’m flattered. Thank you for watching!

[@ajabisong]

Fantastic analysis!

[@richardly438]

The results were Inevitable!

[@austinbradshaw3636]

It seems odd that the mortality and length of stay for acute MI and stroke are attributed to a family medicine or internal medicine doctor that does not identify as a hospitalist. Outcomes for both ailments involve timely diagnosis and intervention usually performed in the emergency department. Additionally both ailments usually involve consultation from specialists that are not being assigned to these patient outcomes. Idk how a generalist is anymore responsible for patient outcomes over a cardiologist for an acute MI or a neurologist for a stroke. I’m curious why those two diagnoses were even included in the study. The other three were much more reasonable, but the study was still horrible overall.

[@sheriffofsodium]

Great question. I’ve gotta think that, at some hospitals, at least, some patients with these diagnoses wouldn’t even be admitted to a general internist or family physician - they would be admitted to cardiology. Comparing hospitals where some/most patients get admitted to an entirely different service is another great source of selection bias and confounding.

[@austinbradshaw3636]

@@sheriffofsodiumAfter considering your additional point, I believe that more likely than not the authors of the study cherry picked certain criteria to get the results that they wanted. I don’t believe that they are capable of being this incompetent by chance alone.

[@DonMoney123]

Another banger 🎉

[@sheriffofsodium]

Thank you for watching, @DonMoney123.

[@jjjjjjjyang]

Strong work

[@sheriffofsodium]

Thank you for watching.

[@DrBrandonBeaber]

Nice video. Any reasonable person can understand the absurdity of this study. One patient admitted to the hospital could have a >90% expected mortality (like a 95 year with CAP but 10 comorbidities who is already in septic shock). Another could have a 1% expected mortality (I was once a young marathon runner admitted to the hospital for CAP). Different doctors treat dramatically different patient populations. A doctor can inherit a patient who has been terribly mismanaged by another as you suggest. The problem with multivariate analysis is there are a thousand unknown and unaccounted for confounders. The patient's diet, exercise regimen, compliance with treatment, availability of certain specialists/tests. The quality of the radiologist could influence the internist's ability of manage pneumonia, COPD, or heart failure. The vigilance of the pharmacist to catch medication errors matters. The quality and patient ratio of nursing matters.

[@epicleetness9287]

You should read the same bullshit papers they did for abim, it's like they copied the same paper

[@sheriffofsodium]

Great points. Too bad you weren’t a reviewer for the paper.

[@truthteller2711]

What about a score comparing step 2 scores only?

[@volodiasmorschok5416]

to be short, do USMLE scores really predict patient care outcomes?

[@sheriffofsodium]

I doubt it. To the extent they do, I suspect the effect is small and mediated primarily by better training (unlocked by higher scores).

[@rileysmith8086]

As far as the doctors, how did they take into account consults(there is obviously large differences between different specialists)? So, if a pt comes in and has an acute MI, obviously they will see a cardiologist via consult. Doesn't that "taint" the "pure" generalist physician group?

[@sheriffofsodium]

They didn’t consider it at all (see my mockery around 33:06). I agree - it’s a fundamental problem for the study.

[@rileysmith8086]

@@sheriffofsodium I'm glad I prefer to see an uncertified doctor that hasn't taken any exams. I actually prefer a doctor that hasn't attended medical school at all.

[@tal8871]

haha this is great. 11.2 pts in 3 years. That's a hefty schedule... Thanks for speaking truth. Be careful... NBME is about to have this channel flagged. LOL!

[@sheriffofsodium]

My guess is that they’ve already watched the video - and will pretend they didn’t. Kind of like when I first saw this paper in November.

[@lympho]

Wow You took a study that sounds legitimate and exposed it for the Exel fishing trip it realy is

[@sheriffofsodium]

Gotta say, that was the goal. Thank you for watching.

[@shans9939]

Just Amazing!

[@augustortiz]

Seems like the people behind this research were padding their CV’s to apply for the C-suite fellowship of USMLE LLC.

[@theotherredmeat]

They basically already are: One of the authors is the VP of operations management for the NBME One of the authors is the VP of the USMLE for the NBME One of the authors is VP of competency-based assessment for the NBME The study was funded by the USMLE (i.e., the NBME)

[@anaos1004]

I mean, what's next, physician's APGAR scores determine better patient outcomes?

[@Joongsu]

USMLE/COMLEX serve as the foundation information required to understand the pathology and clinical medicine but in no way can they replace clinical medicine experience. You become a doctor by well... doctoring, not reading things off of a book. Knowing the information is great, but it is the experience that truly matters in the end. As for board exams, just pass them.

[@EtherealPurple]

I also find it very bizarre that they decided to use a composite score that combined Steps 1, 2, and 3. Many students -- I'd guess the vast majority of students -- try *extremely* hard on Step 1, try reasonably hard on Step 2, and basically just try to pass Step 3. Why include exams that many students put much lower effort into? I'm not sure what the answer to that question is, but it's not what I would have done.

[@sheriffofsodium]

I have a theory about this. And obviously I can’t know - but I suspect that it was done to minimize the differences between US medical graduates and IMGs. You’re right, almost no US grad cares about their Step 3 score, because they’re typically already in residency when they take it. But many IMGs take Step 3 before they apply - and a higher score may open additional doors. Including Step 3 scores makes the z score for USMGs and IMGs similar, which may have helped prevent an unwanted association between better or worse outcomes of care for one group or another. (Remember, there are significant systematic differences between the locations and types of patients USMGs and IMGs care for - and it’s hard to imagine those differences wouldn’t have resulted in differences in outcomes in a study like this. But that kind of association - in either direction - would have detracted from the point the authors wanted to make.)

[@EtherealPurple]

@@sheriffofsodium That's interesting and seems right to me. The next question is, of course: should USMGs and IMGs be lumped together in this study? Given that many IMGs come to the US (and take the USMLEs) after already practicing for several years in their country of origin, it seems like including them wholesale introduces a big confounder, the kind of thing you allude to when you talk about the temporal relationship of the exams to the point that the physicians' performance can be measured.

[@sheriffofsodium]

@@EtherealPurple I honestly don’t know. Since the goal of paper is to assess the validity of the USMLE program, then including IMGs makes sense… but there are systematic differences between IMGs and USMGs in terms of the settings in which they practice, the patients that they care for, etc., and I think it would be have been highly undesirable for the authors to report a statistically-significant association between IMG/USMG and differences in mortality/LOS.

[@healthcareplanforus1246]

Considering there was still 37 min. left I figured there was a LOT more nuiance. We need to look @ everything wrt medical education & policy as if it were a clinical trial.

[@claytontindell9939]

What about the other medical staff that play a role in patient care? This study is a bit ….. bold in a foolhardy sense.

[@sheriffofsodium]

Naw. The only relevant input is that of The One Doctor and their USMLE score. Those people don’t even take the USMLE!

[@claytontindell9939]

@ 😂

[@Alich79865]

Its just a screening tool right now. This test was made to check the basic knowledge to license as a physician and not to check how good a physician is. I guess your video proves that lol.

[@hamidg]

yes the data had 70 male doctors that's probably the reason for the gender difference, and I think mortality and patient length of stay are too vague to be used as a indication of patient care outcomes. and the choice of diseases selected probably played a role in the outcome, I think including diversity in disease selection might change the outcome