Reproducibility in Research: An Introduction

[@GeneticallyModifiedSkeptic]

More videos on how the medical research process works, please! Our culture is hurting for scientific literacy.

[@bradwilliams7198]

Two things that are needed in this effort are 1) journals willing to publish negative results (if they were competently done) and 2) funding for replication studies. My first question about the replication of the 100 papers was "Who funded this?"

[@karandex]

Thank you thank you for covering this topic. We need to people to understand science from scientist

[@AvgJane19]

#let'smakereproducibilitysexy2k17

[@chargingbig4317]

Is this segment on reproducibility going to be focused on medicine, like the channel's emphasis, or more broad so those of us in other scientific fields can learn how to improve more easily?

[@somenerdyblonde]

+

[@Overonator]

I bet you will not discuss prior plausibility, that's the step that is often ignored and we end up doing clinical research on things that have an extremely low prior plausibility of working. For example, why test homeopathy in clinical trials when the prior plausibility of it is so low given all we know about chemistry and physics and biology.

[@therrydicule]

This comment has reproduced and now is stuck with multiple sets of triplets and a low income job.

[@bookworm83197]

"This is otherwise known as FRAUD." *long pause* xDDDDD

[@alexanderjones1265]

This is great stuff! As someone who used to be in research this video raises many of the problems that I saw in my 4 years as a PhD student. I won't mention too much here as I'm guessing you'll cover this in your videos but I hope you mention open-sourcing code used in experiments (I spent about half my time programming yet it was deemed irrelevant for publication, even my thesis!)

[@cm3655]

WOOOT PATREON WISH SATISFIED. THIS IS SURELY THE MOST MEANINGFUL CHANNEL EVER BESIDES SCISHOW!!! Yeah, a bit funny you can get credits from watching youtube vids

[@InorganicRubble]

I love that Healthcare Triage is doing this. Please keep it up guys.

[@XiaosChannel]

The point should be made that, without science, we can't know we're wrong.

[@nicoangenent-mari69]

A lot of this has more to do with methods descriptions not being detailed enough. Science has advanced to such a stage (especially biology) that complexities in experiments are MADDENINGLY sensitive. Something as inane as buying the same reagent from a different vendor can make the difference between getting significant and insignificant results (because that product is lower quality, contained in a different medium, meant for different purpose but not well advertised). I had the exact same thing happen to me in trying to reproduce a study's results - and being unable to reproduce them was my fault, not the authors. I bought crappier reagents without realizing it and made other mistakes that only a very nuanced expert who understood the issues intimately would have been able to pick out. Basically, science is too specialized, to the point where only the most highly specialized experts are often able to do the experiments. When other scientists try to replicate them, they fail, due to their own (relative) inexperience.

[@thaismcrc]

I'm an academic translator and would like to write subtitles in Portuguese for this series, so the videos can be shared with more people. Would you be interested in that? And if so, would you be willing to share the episode scripts? That way, I don't have to transcribe them and can finish the work faster. (I'm volunteering to do this for free because I think these videos can be really helpful)

[@juliomoraification]

Thais Camargo +

[@healthcaretriage]

Yes! Let's talk about this. I think you can use the community captioning. What's the best way for me to connect you with scripts? -stan

[@thaismcrc]

Awesome! Send me an email: [e-mail]

[@seandees3028]

If I had the kind of money that Gates or the like had, I would set up labs that do nothing but the boring but necessary work of replicating and reviewing scientific studies. A legacy that no one would remember, and no one would think about, but would be the best thing that I could do to help the most people, even if in the abstract and detached way of providing better science and more rapid scientific advancement. I would also build awesome but useless things (like castles near my favorite beaches).

[@luxweaver2706]

this comment won't be reproduced

[@rparl]

Olan Williams This commentary will'n"t be etaoin shurdlu....

[@BigBoss-sm9xj]

Ross Parlette lol

[@AlanmanAaron]

It's really cool of you all to have made resources like that, thanks for doing it even tho I won't necessarily be consuming them.

[@TheyCallMeNewb]

This is all very interesting. It takes me 8 months to compile a report, the idea that I myself have tainted all throughout, makes me sick.

[@billyclabough9835]

There should be two types of science journals: one type for findings that have been independently reproduced and the other type for findings that still need to be independently reproduced.

[@BeccerTehRecker]

Can you do a video on the PACE trials for ME/CFS please? From the data that's been obtained through FOI requests it looks like the epitome of bad science (Patients could simultaneously be sick enough for entry to the trial and classed as recovered by the outcome measures)

[@BeccerTehRecker]

There is a lot of links on this page. The ones titled "trial by error:......" are about the PACE trial. www.virology.ws/mecfs/

[@hotdrippyglass]

Congratulations on getting funding from the NIH. This means the quality of your past work merits this investment and shows the recognition you and the team deserve.

[@somenerdyblonde]

+

[@Ziraya0]

wait, this is like, turbo legit? congrats!

[@annali4138]

Is the credit only for physicians?

[@willemvandebeek]

thank you for making this video

[@hotshot619]

Does Dr. Carroll's face look orange to anyone else? Either your white balance is off on this video or its on my end but for some reason he looks orange in this video.

[@rparl]

hotshot619 It's the DJT effect.

[@rikwisselink-bijker]

I think it is a tan. Oh, and don't google DJT, use wiki instead. The urban dictionary has some not safe for work definition for the acronym.

[@broudwauy]

I think either the in-studio lighting is off, or they ran out of Dr. Carroll's makeup tone. The base of his neck looks like his normal skin tone!

[@XiaosChannel]

and holy shit, this is a whole playlist!!!!

[@tombombadillo1]

this comme n t won't be reproduced

[@itsthabenniboi]

i think this is hilarious that it counts for CME

[@rikwisselink-bijker]

I agree, although I think it is a very positive indication. It means that this is not only a gap in public knowledge, but it is also high-quality enough to be considered for credit.

[@khandarwilliam5439]

this is scientific method

[@kcallamajaji]

LOVE this! I worked a little on the compliance side of of Research, with my organizations IRB and IACUC teams. So much education is needed in these communities.

[@lightbox617]

Thank you so much. It has been increasingly difficult, over the last 15 years for me to believe reported scientific studies. It would help me a lot just to know who funded the project.

[@Airehcaz]

Timothy Dingman It seems to me that it shouldn't matter who funds the research, as long as the science is done with integrity. What really matters is the structure of the experiment, and wether or not the result can be re-produced. A poorly designed experiment is useless, and poorly designed experiments hardly ever are reproducible. I try not to judge the validity of a study based on the funding party (which can be hard, since we all have biases) because the truth is important to me. I'm not arguing with your statement or anything, just sharing my thoughts :)

[@lightbox617]

If Monsanto, Astrazenica or any big pharma group funds the research, it is going to be slanted towards the money. Think.

[@mac533]

+Timothy Dingman But if the experiment can be repeated even by persons who aren't funded by Monsanto et al., then the research is valid regardless of who funded it.

[@qbNone]

Bravo! So amazing that you guys are moving from critique to actively helping the community. Critique is needed, but I hope your efforts will remove some of the need to critique in the first place; how wonderful.

[@redwoodferrari]

As someone who works in academic health research, this is quite spot on. Yeah, science is stochastic but not to the degree of complete non-reproducibility. I would love to see more episodes!!

[@janakhabouri8271]

I love this

[@sjwimmel]

Truly making the world a better place here, guys, awesome work!

[@bradleyfrueh2761]

i'm really excited for this series. Keep up the amazing work

[@osianoisekenegbe9401]

I couldn't catch the word at 4:30 and it's driving me crazy! The closest-sounding word I can think of is "apropos." But in any case, I'm excited about these upcoming series.

[@friendly_alkali]

Sounds like "equipoise" (a balance of forces or interests) to me.

[@queuebit]

+

[@guest_informant]

"Equipoise and disinterest" Equipoise basically means being in the middle. However in this context *clinical equipoise* can have a specific meaning en.wikipedia.org/wiki/Clinical_equipoise _Clinical equipoise, also known as the principle of equipoise, provides the ethical basis for medical research that involves assigning patients to different treatment arms of a clinical trial. The term was first used by Benjamin Freedman in 1987. In short, clinical equipoise means that there is genuine uncertainty in the expert medical community over whether a treatment will be beneficial_ Not sure that fits with what was said, but I think it's interesting nonetheless. Basically, We say we're objective but in reality the researcher will often skew the design.

[@osianoisekenegbe9401]

*****​ Guest Informant​ Thanks!!

[@nab-rk4ob]

Awesome guys. You are getting through and going to be able to make a difference.

[@HelixChaga]

SO TAN!

[@piaopiaokeke]

As someone who works for a medical device company, I can confirm that these videos are sorely needed. Please help us save science!

[@maledicenttails]

This is an absolutely awesome video! Keep it up!

[@wrrrdnrrrdgrrrl]

Please oh please cover demographic variety. Drawing conclusions about human health based only on straight white college boys can be so dangerous for the rest of us.

[@somenerdyblonde]

I'm sharing this with the people in my lab! This is great!

[@ArtArtisian]

*feels very smug about mathematics research*

[@MathAndComputers]

In math (and theoretical science), some related-but-different issues are that 1) bogus claims will often be made about the applicability of research/results to sciences or other areas of math, (it used to be all the rage to claim that something had to do with quantum computing), 2) complicated proofs can often take a surprisingly long time after publication to be truly vetted thouroughly, and sometimes, by the time a proof has been found to be incorrect, there are already other published papers building on the possibly-incorrect result, and 3) assumptions are sometimes unclear, particularly in chains of proofs that depend on previous proofs of results that only apply under certain circumstances, but several proofs down the chain, it's difficult to determine whether the circumstances required for the first proofs are actually satisfied or not in the situation examined by the later proofs. I encountered all 3 in my short run doing research.

[@ArtArtisian]

Sure, but: 1) is obvious to any practitioner and does no harm to any theoretician. Marketing does strange things to people. 2) Such complicated and wrong proofs are relatively rare - a good proof is also well written. Plus: results assuming another result are still both interesting and valid mathematics (see everything that depends on RH). 3) Good writing is rare in all of academia - mathematics does this much better than, say, TCS.

[@ExPwner]

Reproduced results in economics: 1. Failure in collectivized systems. 2. Decreases in employment following increases to the minimum wage. 3. Increases in third party payment leading to higher healthcare prices. More data needed, clearly.

[@mac533]

How this for reproducibility: the countries with the healthiest, happiest, best educated populations are almost all "collectivized". OTOH, can you show me even one, ONE libertarian country whose people have a decent quality of life? Just one. Also... if that's true, then why are countries that are more collectivized than America paying less for healthcare than America?

[@ExPwner]

Healthiest - based upon what criteria? Life expectancy? Because that's more than healthcare quality. Happiest - no. This metric was completely debunked as a difference in cultural expectations. Best educated - many of the worst educated are in collectivist systems. Libertarians don't have a country. The prevalence of collectivism isn't some sort of testament to its successes. All of the countries that you could list are doing worse economically as a result of collectivism than before it. The US has a halfway collectivist system already. The fact that other collectivist systems that are more collectivist have better outcomes doesn't prove that collectivism itself is a good idea. This is analogous to the Laffer Curve.

[@mac533]

+James Adams There's different types of collectivism. Well, I say collectivism, but I'm using the libertarian definition of collectivism, which is anything that dismantling the gov't and worshipping corporations. What data do you have that libertarianism works, then, if no country works by libertarian principles? Seems like a huge guess to me. Why yes, life expectancy IS more than healthcare. It's not allowing corporations to dump poison in people's drinking water, or use deadly pesticides around children, or put trans fats and HFCS in the food for school children. Shoot, if you want people to walk more? How about better public transportation, more bike lanes, more pedestrian-friendly cities in general? Really? And what cultural expectations are those? Not having to work three jobs in order to have middle class life? Not becoming bankrupt because you broke a finger? Not being tens of thousands of dollars in debt after college? Not getting cancer from all the hormones and chemicals big companies put in food? Not have poison in the air and water? Really? What countries are less collectivist than the US and do better in education? And yeah, the US is a halfway collectivist system. I mean, corporations aren't allowed to murder people yet. That's what collectivism is, right?

[@ExPwner]

Libertarians don't worship corporations. We uphold voluntary interactions and the NAP. Huge difference. I don't need any. I'm not making claims about libertarianism. I'm making claims about the failures of collectivism which are many. I'm also arguing that these socialist utopias that you and many leftists like to uphold are on roughly the same path. Libertarians don't support companies dumping poison into people's water. However, if you have an unwavering support of government monopoly then you'd have to answer for the government of Flint poisoning their water. As for walking, people can walk for free. No coercion needed. Cultural expectations? I don't recall saying anything about that. Not my claim. I didn't mention less collectivist countries. However, the private model of education can and does do better in the US than its public counterpart, which spends roughly $11k per student per year on inferior results. Hell, for $11k per student per year I could organize a more effective system and still take home a salary for doing so. And the kids would have a marketable skill after leaving, too. No, that's not what I mean by collectivism because even an individualist society would ban murder. What I mean by a collectivist system is that the government pays about 50% of the healthcare costs in this country. Only a small percentage is directly paid by consumers. Out of the other half, much of that is still paid by insurance companies. Meanwhile, we have places like the Surgery Center of Oklahoma delivering much cheaper healthcare based on direct payment. Not even socialist countries can do that kind of cost savings.

[@mac533]

+James Adams Every system has failures. If you're arguing collectivism is bad because it isn't perfect, then that's a poor argument. The real question is, does collectivism have more failures than libertarianism? If so, prove it. Libertarians don't support companies dumping poison into people's water, they just won't do anything to stop it. And if Flint were a corporation-owned city, rather than a government-owned city, would things be any different? Other than the fact that you wouldn't be able to vote the ownership out of power? You did say something about cultural expectations: "Happiest - no. This metric was completely debunked as a difference in cultural expectations." I'm saying I don't think their cultural expectations are that much different. I wouldn't be happy if I were one broken leg away from homelessness either. You could make a more productive school system with $11K, but why would you? When the shareholders would be way happier if you just ran the cheapest possible system, producing children who can barely write their own names and massive profits. (You know, like for-profit colleges do.) Once again, I'm sure corporations can produce good products cheaply, but why would they, when they can simply produce a terrible product and jack up the price? Competition? Any competition can be easily stifled by a sufficiently large corporation. They'd find a way.

[@Reinier020]

this comment won't be reproduced

[@999is666upsidedown]

this comment has gender identified as an Apache attack helicopter