The scandal that shook psychology to its core

Wanna watch this video without ads and see all of our exclusive content? Head over to nebula.tv/vide...
In 2011, a scandal broke in psychology that made everyone question whether any of its research could really be believed. For the last decade, psychology has grappled with the aftermath and has tried to understand what went wrong. The ensuing crisis of confidence comes from a replication crisis in the field that we’ve seen over and over through all of science. So… what now?
Visit www.brilliant.... to get started learning STEM for free, and the first 200 people will get 20% off their annual premium subscription.
Buy our book, Brains Explained! amzn.to/3hkmCdo
---
Find us on social media (Twitter, FB, IG) and check out our website: www.neurotrans...
Join our Discord server: / discord
If you like what we do, support our work by becoming a Patron: / neurotransmissions
We couldn’t do all of this without our awesome Patreon Producers, Ryan M. Shaver, Danny Van Hecke, Carrie McKenzie, and Jareth Arnold. You four are like warm sunshine on a cool day!
And thanks to our other high-level Patrons, including:
Marcelo Kenji
12tone
Linda L Schubert
Susan Jones
Ilsa Jerome
k b
Raymond Chin
Marcel Ward
Memming Park

[@WhichDoctor1]

My brothers phd was originally replicating quite a well known study about gut microbiome of rats affecting their brains. He went to great efforts to make sure all his rats were exactly the same strain as the original study, and actually found a higher quality source for the feed with less nutrient variability, and was super careful with all his methods. And he found absolutely no results whatsoever. He was quite upset and anxious for a while that his whole PhD had been a failure and no one would publish his study because it contradicted this famous one and everyone would think he’d just screwed up somehow

[@TheDrFMG]

One of my PhD students conducted a genomics study. Funded by venture capitalists who invested more than $650k. We discovered not only that the hypothesis didn’t work but the field is largely gobbledygook. He presented his findings at a conference at a world leading genomics institute in Europe. He was hired before the day ended. This is largely due to identifying where everyone were using flawed probabilistic inference and limited methodologies. Please relay this story to your brother. A PhD is merely a training programme and he should not doubt his ability. Negative studies are important and gut-brain theory is like squinting at the stars and picking your preferred asteroid as being important. Best of luck to him.

[@halweilbrenner9926]

What did the study indicate? Consistant diet affects the gut but seems to have little or no effect on brain function.

[@k-miz3683]

Which study did he try to replicate?

[@christopherellis2663]

Well, someone screwed up

[@AnaBanana-yk9px]

Or someone lied.

[@homofloridensis]

In the Navy we used to say, "You don't get what you expect, you get what you inspect." If you reward publications, you get publications; if you reward research you get research.

[@huveja9799]

@user-ox6nc6ly7f I would be careful with popular expressions, warfare is one of the most difficult arts. I would like to see how someone conducts the logistics of an operation with thousands of troops without a certain logic ..

[@huveja9799]

The problem is that inspecting already supposes a certain predisposition, you are inspecting with a certain objective in mind (i.e. some "expectations"), and that objective is precisely the one that can introduce biases (both in the one that produces the data and in the one that inspects). On the other hand, if I award research, research is certainly going to proliferate like weed, but how do you discriminate between bad and good weed?

[@sunnyadams5842]

​@huveja9799 Schrodinger's Cat...

[@holmavik6756]

Professor in Statistics here. Just two comments: firstly, one would expect researchers to involve statisticians in their statistical anslysis but that is rarely the case, and secondly, I dare to say, research in empirical economics is generally much more erratic than in psychology.

[@ztukariansevuri]

They cant and wont, ever. I am not educated, but I really dont have to be in 2023 to understand what is taking place. One would expect any form of "validated information" to be cross referenced through virtually every other highly specialized field in order to rule out the possiblity of their data being cross contaminated by a souce that the research has zero knowledge about. I will give an example. If you take mental illness as a prime example. Generally, if you're suffering from any of the infinite symtoms of mental illness, you will be sent to probably a specialist and given a magical pill (cause there is a pill for everything it seems). Well, if you compare the nutritional data over the past, say, 100 years, and you realize the general population consumes very unbalanced diets, and then if you compare the symtoms of nutrient dificiencies then you're confronted with the realization that nurtrient dificiency has the exact same symptoms as clinical mental health issues..... There are far more examples, but if my logic isnt sound please let me know. I actually have been meaning to have a sit down with someone widely accepted as more intelligent than I am in order to figure out my intelligence lol.... I mean, you're a professor, Marilyn Vos Savant is a perfect example of incredible intelligence being attacked by Ph.D's and scholars, mathmaticians, and she turned out to be correct....

[@vaska1999]

Not surprised to learn that economics research is iffy, to put it politely.

[@Heyu7her3]

👋🏽 If I make it on the other side of this durn program, I will GLADLY hire a statistician! I even told my Quant Methods professor this last year 😅

[@path2source]

That must be why it’s not uncommon for economists to publish in the Journal of American Statistical Association! Economics as a discipline is unparalleled in how carefully it engages in causal inference. If you were a statistician, you’d know that.

[@jadegrace1312]

@@path2source Lmao actually defending economic research

[@samdog_1]

I saw a lot of this behavior while working on my PhD. There is such strong pressure to get positive, confirmatory results, researchers will twist the data any way necessary.

[@marybean2231]

I know you're being truthful but I can't help but giggle at the thought of you doing the same in your own study... Ah, the irony

[@mignonhagemeijer3726]

Its incredible how easy or quickly it happens. I've been TA for many courses and I always have hammerd on doing this as planned out (and first really plan things out properly). Not adjusting hypothesis afterwards and not suddenly adding variables etc. But is is kinda brushed off by some others. I also feel like some of the researchers I worked don't seem to value carefull planning.

[@odst2247]

Because science is about trying to find out the truth of the world, or reality or the fact, or overall knowledge. There is no room for nothing other than positive confirmatory results

[@dabtican4953]

@@marybean2231 Would certainly be ironic if he actually does that but that's unconfirmed

[@albertmockel6245]

I encountered that in my master thesis in ecology. I was not asked to change data, but given my skills in statistics, I was asked to torture the numbers further. This is what tipped most my decision not to continue for a PhD despite the offers.

[@jcskehan]

I firmly believe that the first year of a PhD, or at least for master's capstone projects, students should be spending their time validating other people's experiments.

[@vio1583]

Makes sense. Why not demand at least one study from a PhD to be a replication study?

[@inregionecaecorum]

@@vio1583 There is a big elephant in the PhD room, which is the requirment to demonstrate original research in order to have ones proposal accepted. Sometimes the research is so original that nobody ever takes up the challenge of trying to replicate it.

[@ChristoffelTensors]

My wife did this and found that both previous PhDs had major flaws in their work and procedures that don’t do anything. She is a chemical engineer…

[@birdmusic1206]

this is a problem because receiving your PhD becomes contingent on "validating" work, in other words as a newcomer you will be afraid to question the seniors and might just propogate errors

[@9xixix9]

@@birdmusic1206I respectfully disagree, the proposition is not you receive your PhD by Validating other publications. The proposition is: a contributing factor to receiving your PhD is by doing the actual work that is needed to validate or disprove a publication. And if many students find errors over and over again then a spot light would be put on that specific publication. This helps to put a check and balance in place.

[@wintersking4290]

I feel like we need a catalogue of the null hypothesis, where scientists can publish like a two to three pages report on Failed or inconclusive studies. Make it searchable and that'd be an amazing resource for future scientists to see what pitfalls to avoid. And it would probably take less resources to create and run than Wikipedia or other such websites.

[@snailmailmagic1133]

it will benefit academia but not any industry. And no industry wants that as it will leave proof of the lies they try to build.

[@margodphd]

I think that's a decent step. Let's document studies that have been repeated over and over with null-significant results so they aren't repeated multiple times over. Still, with the way research is financially motivated, that will not be enough.

[@funkster2009]

agreed!

[@eveningstar1]

Great idea!!

[@jeallen10x]

Great idea

[@MasterPeibol]

The problem is not the Science, the problem is the system. Publishers hold too much power and researchers depend too much on grants that do not consider much more than number of publications. It is vicious cycle, that can spiral downwards quickly: no positive results-> manuscript not accepted in any relevant journal -> no grants -> no money to do more research -> no more results -> no more publications, etc. Many researchers depend on grants not only for their projects, but also for their salary. So you can easily see the point of failure, if your way of life depends on positive results, and you don’t have them, there is only one way out.

[@pepperachu]

Unfortunately you could also argue that many take advantage of said grants

[@raroonchandranadv5]

Both are true and both the theory are conceived from some human mind....One thing that is of my personal opinion... Humans have stopped thinking..I just vaguely recollect Oscar Wilde quote...which was like' most people are other people's opinion and other people's quote'... When the author felt that in his time to be true....atleast someone else's opinion was relevant and true that gave some meilage. Now as most of us humans have stoped to think and have exhausted even the conclusive thoughts of some genuine thought or research...we truly are in a crisis of fresh thoughts and fresh actions . Thanks to AI , thanks to Augmentive reality and others that make the impossible possible , I really did enjoy the talk of this subject and wish you well.

[@JonnyD000]

Right, the problem isn't with science or statistics, it's with the economic incentives in the publishing system.

[@Thekarateadult]

The process isn't flawed. The agendas behind it are.

[@mr1nyc]

No wonder the hard sciences want to guard against getting grouped with soft sciences.

[@StingrayMk1]

I remember studying statistics for psychology and thinking "Wow, so I can say my findings either disprove the null or don't disprove the null according to how I slide this p-value between .05 and .01!" It was a WTF moment.

[@gseeman6174]

Absolutely... the alpha level should be set in the study plan prior to running the subjects.... Call it Pre-procedural Alpha... it should rarely ever change after the study has been run.... the people who change it are misconstruing alpha-level with effect size... Many researchers have told told me that my idea on this is nuts

[@Thetarget1]

In modern statistics education, people are going away from the idea of having a cut-off for the significance level at all. Instead the p-value simply gives you the probability, that the data was generated by random chance. You should then interpret based off this p-value on a case-by-case basis.

[@katja6332]

Yes, same here. That's why we need to always have access to the real data and read it, how they calculated it. Is this tiresome? Yes. Is this science, yes :)

[@ThePathOfLeastResistanc]

Same here

[@amazinggrapes3045]

I had a bit of a crisis when I found out that the threshold/criterion for "statistically significant" was completely arbitrarily decided.

[@jguenther3049]

In my Psych 101 course, when asked, our professor admitted that the field attracts people with mental problems, but stated that exposure to the field during their studies often gave students sufficient insight to drop out. This is no longer true. Over many decades, the requirement for personal therapy to attain a degree in Psychology has dropped from 40 hours to 24 hours to 8 hours to, at many institutions, "highly recommended," i.e., zero. At the same time, tuition costs have gone up by a factor of ten or more. Fewer students drop out and kiss goodbye to tens of thousands of dollars already invested in their education. The field is in deep trouble, and the consequences in education, research, and clinical psychology are outrageous.

[@MrJCerqueira]

So your comment is that you believe “personal therapists” have

[@jguenther3049]

@@MrJCerqueira Vs decades ago, as stated.

[@ElizabethMillerTX]

I got my undergrad with no such requirement, decades ago. Was that not the norm?

[@jguenther3049]

@@ElizabethMillerTX It may have been fairly common, depending on how their graduate degrees were structured.

[@nicholascarter9158]

@@MrJCerqueira I think it means "people used to have to get therapy on their own personal issues before starting therapist training"

[@Seraph.G]

I remember in a Linguistics class I took as an undergrad, our final project was a research project based on language data and my hypothesis was completely unable to be supported one way or the other based on the data we had to work with. My professor was SO THRILLED when I stood up there, described my hypothesis and research design, all the effort I took towards proving it, and then went "and I still have no idea." It was honestly a great learning experience.

[@GrahamCStrouse]

Good for you, mate!

[@chroniclesofpickles]

Former PhD candidate here (with a background in psychology). Publish or perish, p-hacking and even p-value itself were extensively discussed within the field while I was doing my PhD (in cognitive neuroscience, or specifically neuroeconomics). Personally, I preferred using Bayesian inferences when it comes to statistics. But there’s many problems why scientists face the replication crises: 1. The sugar daddy of science: in theory, you don’t need to be a good scientist. You need to be a good marketer and a good salesperson to get the grant. Yes, and I was a part of a EU grant with billions of €€ but bullshit research plans. 2. Academic politics: following what I stated above, when the higher up is focused on using the €€ to buy fancy new tools, the lower-ranked PhD researchers as as myself (previously) became tools as well. We had no say in anything, we were just means to the end. 3. No one cares: with €€, power, and politics involved. Who’ll care if anyone is doing good science? It’s about publishing results right? So who cares about the PhD student, assign a shitty supervisor who doesn’t care, write bullshit things in the reports. P.s. the general rule for my PhD was to publish 3 papers in 3 years. And the two studies I was involved in did not have any “positive” results so to speak.

[@LouisaWatt]

“I thought I had something there but it wasn’t significant” 😆😆🤣 that was the wittiest p-value joke of them all.

[@ckwind1971]

I was a psych major around 1998-2002. I started learning about unscrupulous studies from pharmaceutical companies, specifically about the SSRIs I had taken for 20+ years. Great essay, I learned and marveled. Edit: I took Statistics twice (2 different schools) and barely passed with Cs both times. If I'd had to produce a whole study it might have been all over 😂

[@monissiddiqui6559]

I remember when I was back in university we used to make fun of psychology and medical science to be "easier" than math and the harder sciences. We were never convinced of many of the results coming out of psychology because of the lack of rigour and lack of quantitative literacy among those who studied the softer sciences. We were always made to seem like assholes for pointing these things out and maybe we could have framed things more nicely, but I'm glad to see more gradual mainstream recognition of the failure within the softer science fields. One thing we were completely wrong about, though, is that the softer sciences were "easier" than the hard sciences. The complete opposite is what is true. The hard sciences are what are easier since we can arrive at more well-defined answers in those fields. The amount of complexity involved in the softer sciences is mind-numbing to say the least. I have a lot of respect for anyone who ventures in search of the truth and I have no doubt that, as the softer science fields mature and develop more tools and methods to aid in their study, we will arrive at some amazing insights. I'm grateful to be witnessing this inflection point in these fields and I'm excited to see what we uncover. The other possibility is that we are approaching the limits of what we can understand with science and mathematical modelling and that makes me feel differently, but still grants me a sense of awe at the wonders of the universe. Maybe we will need something completely different from traditional science and math to gain further understanding of the world. We can never know everything there is to know and that's hard to accept, but not unfathomable. We already have examples of things we will never know from studies in Cosmology.

[@Xamufam]

Scanning the brain, simulating the brain and AI might be a solution in the future to create rigorous testing in psychology

[@gogowshagenai6139]

Im not sure about this because theses tools are put on a pedestal while they're easy to missuse and can be the source of quite a lot of pbs.... +humans sciences are way too complexe to solely relay on it (we all know that it would be an issue)

[@gogowshagenai6139]

Im not sure about this because theses tools are put on a pedestal while they're easy to missuse and can be the source of quite a lot of pbs.... +humans sciences are way too complexe to solely relay on it (we all know that it would be an issue)

[@gogowshagenai6139]

Im not sure about this because theses tools are put on a pedestal while they're easy to missuse and can be the source of quite a lot of pbs.... +humans sciences are way too complexe to solely relay on it (we all know that it would be an issue)

[@gogowshagenai6139]

Im not sure about this because theses tools are put on a pedestal while they're easy to missuse and can be the source of quite a lot of pbs.... +humans sciences are way too complexe to solely relay on it (we all know that it would be an issue)

[@innocentsmith6091]

There's nothing wrong with studies that aren't replicated. It just means there was a flaw before. The real problem is studies where the replication hasn't been attempted are put forward as knowledge, and used to make decisions.

[@dshe8637]

There are many reasons why a study may fail to replicate. It doesn't always mean there was a flaw. Exploring those reasons is where it starts to get interesting

[@dustysoodak]

it would be nice if there was a place people could post all the negative results. Even if it wasn’t officially peer reviewed, there would still be feedback from comments by other researchers and there would probably be minimal intentional fraud.

[@konyvnyelv.]

My issue with psychology is that it's fully arbitrary when it comes to find a difference between health and illness. Potentially any behaviour society doesn't like can be considered a disease. Example is homosexuality which was a illness in the past and now no longer since society accepted it. Some pacifists were accused of being insane for not wanting millions of people killing each other in WW1

[@matthewkopp2391]

Freud and most of his early following considered homosexuality a „normal“ variation. And Kinsey confirmed it’s prevalence. When it came to GB and USA the DSM was written up redesignating it as a disease. But the reason for that was specifically from the idea in Freud of „societal norms“. So they could recategorize based on society. Freud’s position was that because it in and of itself did not inhibit adaptive functioning it should be considered normal. It is not arbitrary in the true sense of the word. It does however border on a type of ethical philosophy in the ancient sense of that idea, for example the Stoics.

[@konyvnyelv.]

@@matthewkopp2391 still, defining abnormal what inhibits you to adapt to society is arbitrary

[@babybobo1231]

Not all mental illness designations are arbitrary. Psychosis being one of those disorders. But there is always a risk of pathologizing completely normal behaviors

[@matthewkopp2391]

@@konyvnyelv. arbitrary by definition is random or whim as apposed to reason. There are other types of reason that are not scientific material reductionist. I don’t believe making it a mental illness was either random or based on whim. Perhaps a better word that would suit your argument would relative ethos. My position is that much of psychology is not science it is a philosophical ethos which negotiates well being of the individual in relationship to society. The fact that people think it is hard science is problematic. It is informed by science, restrained by the probabilities of science but is not science.

[@relight6931]

​@@matthewkopp2391I see your point. Hard to not find validity in it.. In all "soft" science while we are at it. But I would find personally, a world without the field of psychology to explain some of our behaviors even more prone to various forms of self deceiving... Just imagine having savere depression, and everyone in your life that knows of you, constantly calling you lazy..Without knowing that it is an illness, that you might battle with it, either through discipline, specific diet or medication or all of them together, for even a bit more will, hope... your self worth would just deflate, negative thought patterns would duplicate until only logical solution would be the one that ain't a solution at all.. I just find human minds, and our ability to imagine, then try to engineer something into existence, one of main points of being a human, beside purely biological, to survive, conserve energy and reproduce.. The better we understand how they work, what they are capable, what is good and what is bad for them, more robust we become not just as individuals but also as a species.. We are just scratching the surface so far.. I just wish we could have science, for science sake. To satisfy our curiosity, to enrich humankind in different ways then just material. In the end, it might be the difference between surviving next 100 years, or slowly getting extinct.

[@inregionecaecorum]

As a PhD candidate who had formerly undertaken statistical research in a non academic context and actually studied statistics at a formal level before which involved understanding the concepts, I found that the research training for PhD candidates left a lot to be desired, particularly given the predominance of computer packages such as SPSS. Younger students simply did not have the formal understanding to know what they were doing in my opinion. As you say you can bludgeon almost anything into significance. What is most important in understanding research results from a critical perspective is not what they say, but how the conclusions were drawn and above all of most significance (in the non mathematical sense) is a sound methodology and thorough logical consideration of all the possible confounding factors. You know it is an old trope that most psychology is based upon white middle class college kids, who form an availability heuristic of willing research subjects.

[@lovingdemon2932]

If you wouldn't mind looking into this little theory that some of the world's well known geniuses were malicious since the mathematical odds of all of them being non malicious is probably very low. And at the very least extremely suspicious.

[@Cloudsurfer69]

Damn this is a 🔥 comment. Your so smart you intimidate me 😂🥰

[@Heyu7her3]

But wasn't that a goal of early statistics -- to produce believable numbers? Many of those theorists were social Darwinists/ eugenicists who wanted to make the differences between people significant.

[@kwarra-an]

​@@lovingdemon2932not sure what you mean by "malicious", but you're making a logical error. "It's unlikely all of them were not malicious" =/= "it is likely most of them were malicious". Isn't is just as unlikely that most of them were malicious, whatever that means? "It is unlikely all dogs are not rabid" =/= "it is likely most dogs are rabid"

[@bluesunquake]

This! I desperately wanted to take an advanced stats class during my PhD, but my supervisor didn't think it was necessary. I worried that, since I didn't have a deep understanding of stats, I might well be using the wrong tests, etc.

[@TheInfinityzeN]

My x wife is a biological research scientist (meaning she designs drugs and medical treatments) and from her I can tell you that the problems found in psychology research are actually pretty common across all medical research. The process of handling and interpreting the data is often manipulated to give the desired result to ensure funding.

[@Someone-ji6ni]

It is, quite a few physics and cancer research papers are poorly done or just a flat-out lie/manipulation. It's pervasive in every aspect of scientific research and the whole standard of publication and research needs to change.

[@euchale]

Same problem exists in other fields, remember going to our bioinformatics department and asking which clustering algorithms to use for a certain type of data and I was asked how many populations I want to have...

[@PinakiGupta82Appu]

While delivering a lecture in class discussing the chapter Standard continuous probability distributions, an assertion was made by someone, 'But, the numbers don't lie as we all know'. In reply, she told, 'The data must be fed to the system of equations appropriately'. Now I see that it also applies elsewhere in all disciplines other than raw hardcore convoluted pure mathematics and statistics. Truly amazing!

[@its_jess4321]

As an outpatient social worker I also was Intrigued about the need for positive results and ignoring unfounded or negative results because when you’re working with a client there’s wisdom and anything and everything they say. Meeting goals give info and there’s reasons why a client doesn’t meet their treatment goals and there’s information in that as well and it’s ““ unsuccessful“ but there’s still a Why

[@irinaphoenix2169]

What I HATE is the system that connects being a good researcher with being a good teacher. One of the many things in a long list of things that are broken in American Universities.

[@davidschreiner6667]

If you had two books, one of every provable thing we really know about psychology and the other about all that they think they know about psychology you would end up with the first book about the size of a typical comic book and the second with about as many pages as the local library has in its books.

[@Corporis]

"the p-value sounds like incontinence medicine" had me laughing real hard

[@neurotransmissions]

I had another one about it being a urine-based cryptocurrency, but I cut it 😂

[@marksegall9766]

8:31 "I thought I had another... but let's move on it wasn't significant" is the best of the p-value comments.

[@NiHaoMike64]

@@neurotransmissions One of my friends recorded videos of herself peeing standing up, then traded them for Bitcoins. (That was in 2014 or so.) Truth is stranger than fiction...

[@LouisaWatt]

I’m currently doing honours in psychology and the Australian university I go to teaches about the replication crisis as part of learning about systematic reviews. It is possible to detect publication bias through statistical analysis in the review process, but obviously that doesn’t solve the problem of it happening in the first place. Fudging data to get published, and journals selectively publishing need to be called out more quickly.

[@mh8704]

I think the problem begins in grade school and high school learning that is over focused on right and wrong answers that fit testing methods instead of teaching people to be broad minded and see learning as exploration.

[@jlmonolith]

It kinda feels like journals should add a special stamp/seal/mark on studies that replicate or have been replicated by other studies, so that everyone can know to take those studies more seriously, compared to studies without that stamp/seal/mark.

[@Manullus]

I'm so happy I discovered your channel. What a brilliantly put together video! Thank you for this, I think you're spot on with everything.

[@WillyOrca]

I remember starting my job and having total imposter syndrome for the first 2 months. I was stressing every single day, losing sleep at night in a perpetual state of fear each day might be my last there. I was essentially just waiting for the day I got called into my bosses office to be fired, but when I finally got what I thought was THAT call, it ended up being a call to pickup a 200$ gift card. I got employee of the month because I had the best numbers in my department that month lol. They also gave me a +2$/hour raise. I was FOR SURE underestimating my performance because I had nothing to compare it to nor any sort of metric to serve as a standard.

[@ruthhorowitz7625]

As someone who was diagnosed with MDD at 42, ptsd at age 55 , and ASD at 57 i can say with certainty that psychology and psychiatry are broken. I had ptsd and asd at age 42. I was hospitalized for 6 weeks, had meltdowns, severe sensory issues, couldn't eat the foid, and no one tried to connect the dots. I could have been diagnosed with aspergers at the time, but they couldn't be bothered to look deeper than the 15 minute evaluation the original psychiatrist did.

[@sinfinite7516]

It was never addressed if the 2011 research on Esp ended up being replicable or not, I was kinda hoping there would be a resolution like his study was unable to be replicated or the Professor was found to be p-hacking etc. but it was never brought up again.

[@roncarlin3209]

Maybe the emphasis should be on falsification, not on replication. This is what Karl Popper said is the proper scientific method. In other words, design an experiment to disprove the candidate hypothesis, don't just leave it hanging in the air as not significant. Ok, not many hypotheses will yield concrete results if treated this way, but you never hear of any experiments designed to falsify.

[@The_Questionaut]

The video discusses the "replication crisis" in psychology, where many influential studies have failed to replicate or have much smaller effect sizes when replication is attempted. This calls into question what research in psychology we can actually rely on. One major contributing factor is publication bias - journals favor publishing positive, exciting results rather than null or negative results. This skews the literature. There are also issues with researchers engaging in "p-hacking" - manipulating data or analyses until significant results are obtained. This leads to false positives. Questionable research practices are very common, often unintentionally. Things like flexibility in data analysis, lack of transparency, and small sample sizes contribute to non-reproducibility. Solutions involve focusing more on study preregistration, sharing data, placing less emphasis on positive results, and changing incentives in academia to value replication more. Overall more rigor, transparency, and replication is needed. The host argues psychology has made a lot of progress in the last decade on these issues, and this "existential crisis" may ultimately improve the field by leading to more robust and reproducible research.

[@sotpunkkatt158]

in 6 months ill be doing my masters. Ive already floated some ideas about doing a replication masters but my handler is arguing that it will hinder my carrier possibilities. in other news ill be doing a replication masters in education.

[@Stroheim333]

Every PhD student should be given the tast to replicate older studies, never try to prove their own new hypotheses. That should give them a proper respect for the scientific method, and be a good way for scientific research to sanitize itself.

[@ryanmccrary1880]

I am way too late to this video but this was incredible. Very informative and very funny. This is fantastic content.

[@wanderingsoul2758]

Notes on Main Ideas 2011- Pyscologist Daryl Ben discovered its possible to sense the future. He sent the results to The Journal of Personality and Social Pyscology, which resulting in a reckoning in pyscology. 2015-Reproducibility Project Pyscology: 27 scientists discovered only 36% of studies had replicated results 2017- Nature and Science; 62% had replicated results

[@sandollor]

It's absolutely a good thing this all is being brought to light and continues to be, and I'm not just saying that because I didn't do well in my psych stats class. :P There is a large disconnect between research/lab and clinical work too.

[@marcmckenzie5110]

40 years ago I wrote an unpublished undergrad psych paper covering most of the issues you so eloquently cover here. My research methods professor pulled me into his office and soundly chewed me out. I forced him to accept the paper, and of course he gave me a poor grade, which I took before the academic dean. After the academic committee reviewed both sides, the paper was returned the second time with a perfect score. I’m sure smarter people than I who were post-doc raised similar issues back then, so the harder question is how do we as a society demand academia to get their shit together. Pretty tough in a society which seems increasingly to only value profit and wealth. Those are ultimately the pressures behind a great deal of this. We can’t even get our congress to cooperate on agreeing on bathroom protocol (tongue thoroughly planted in cheek). Sigh.

[@ScribblebytesWorldwide]

Change capitalism to communism.

[@KyleHarmieson]

Journals should publish at LEAST one replication for every novel study

[@Aiphiae]

This is why the only people who take the social sciences seriously are people *in* the social sciences.

[@sendtosw]

I've always considered psychology to be a bogus "science." I find the idea that human personality can be "studied" with the same level of confidence as other sciences like chemistry or geology or even biology to be ridiculous. There are too many smoky glasses to barely see through, too many fuzzy, uncategorizable factors, changes and variables to be able to reach any kind of reliable finding. As far as I'm concerned, all the major theories are no more than speculation masquerading as science. They hover around ideas and concepts that might be closer to the real truth than to falsehood, but that's about the best one can say for them. They aren't science. And I've had this feeling ever since "psychology" and what it means was first made known to me all the way back in elementary school. In university courses I never encountered any reason to change my mind. This is all just another example of how our culture has been so impressed with the results of using the scientific method that we have very, very wrongfully assumed that it could be applied to all aspects of reality -- and if reality doesn't cooperate, well reality can just go pound sand. Sorry. Some areas are just murky. They don't lend themselves the level of certainty we demand. And if we try to do it anyway, well. We end up in the situation this video is describing. And a lot of people are revealed to have been naive at best and dishonest and disrespectful of truth at worst. Know what? After listening to this whole video, here's what I think the answer to all this is: humility.

[@elvingearmasterirma7241]

You can dee things like depression, ocd, anxiety and so forth in MRI scans We can physically map out how these issues affect the brain. Psychology isnt just throwing random stuff at a wall, its using modern medical technology to try and make sense of the brain and its chemistry Its like slagging off the whole field of sleep studies just because you personally think its too wishy washy without understanding the way they study the electrochemical signals produced in the brain

[@edwardharvey7687]

Being as a lot of attitudes and public policy are based on findings from psychological studies this is rather disturbing.

[@Trident_Gaming03]

So I'm guessing the guy who "proved" ESP exists was p-hacking hard

[@neurotransmissions]

Yes! I forgot to mention this later in the video! But essentially he was engaging in HARKing and was coming up with his hypothesis after he knew the results of his data and had some other questionable research practices.

[@Trident_Gaming03]

@@neurotransmissions It's as if he was running a blind study, but his participants were a publishing board, very meta, amazing

[@neurotransmissions]

@@Trident_Gaming03 Totally. Another detail that didn’t make it into this video was that Daryl Bem was actually an editor for the very journal he submitted his paper to. Weird stuff.

[@ferventheat]

I think a study measuring the effect of psychological experiments on a psychologists psychology should be undertaken

[@quantumfineartsandfossils2152]

5:45 + You are an absolutely brilliant researcher and producer, thank you for working on & making this for us.

[@jules4947]

Some issues that I and others I've met have had is how inherently reductionist research is required to be. How can we possibly factor in cultural differences and beliefs, feelings of safety, perception of self, social identity, community activities, social status, location, housing insecurity, political oppression, disability, minority stress, perceptions of trauma etc all at once? Lived Experience has so many factors and psychology's perception of "problem is with the self" only helps to a degree. Maybe someone doesn't need to get better and it's their environment that does need to get better. It also assumes that a "normal" and "unwell" exists, but this is cultural and has led to many within the Lived Experience movement to see psychology as just another tool of oppression and medical abuse. It also assumes that "statistical anomalies" are at fault and treated as "too complex" because we can't find a cure or make them normal, and so, they are then dehumanised by medical staff and therapists. There are 7 billion unique understandings and experiences with entirely different and unique understandings and ways of thinking. Is it realistic to think that we'll ever find something shared by them all? There has been a very distinct lack of Lived Experience within research and it has led many within the field to not hold trust in it.

[@NCRonrad]

My favorite story in this vein comes from the measurement of the electron charge - Feynman discusses this beautifully

[@petrairene]

I have a biologist friend and he told me from the time at university, he found the studies in the psychology department shockingly amateurish in their methodology. He had no respect for the psychology department and it's people at all, finding them very unscientific.

[@johntang605]

In my graduation 9 out of the 13 highest gpas were psych majors.

[@teogonzalezcalzada5812]

Came here from nebula just to say that this video reminds me a lot of the controversy of Dream and minecraft speedruns. The themes of p-hacking and statistics were deeply analyzed because statistics were used to prove a minecraft player was cheating (he later admitted to doing it). And the maths were tested by a lot of experts. "How lucky is too lucky" from stand up maths it's a good analysis and summary of how this works, in a scientific approach for a way less serious theme. Btw, thanks for the great video! Will return to binge-watch the rest of the channel on nebula.

[@Tryptic2x]

I lost all confidence in psychology as a science when this stuff first came to light. To this day, I am deeply skeptical of the field as a whole.

[@dheerajvanarasa910]

" Let's move on, it wasn't significant " Good stuff.

[@BanFamilyVlogging]

This is a systemic issue across all sciences, not just psychology. But it is worrisome, regardless. I side-eye everything now.

[@megan5074]

You summed up my advanced research methods psychology class really well done

[@awnage]

I (Cornell grad) was hoping you would bring it back to Bem at the end and let us know more of what happened from that specifically. I remember lots of dorm mates taking psych 101 and participating in his studies.

[@exosproudmamabear558]

Ngl I am in love with the amount of Michael Myers jokes in this video.

[@sirijanthakur]

For everyone, psychology is actually like a relegion coversion, not appreciating uniqness and making everything "normal" first they say you can be gaslighted, then they convice you that you can, and when you accept all the shit, they sell you stuff of advices to convert you, into their relegion, and then you end up with spirituality - needing help, and the sht of the system. words of bard - You're right, the assumption that there can be only one type of person and everyone should conform to it is outdated and flawed in several ways: 1. Human diversity is inherent and vast: We are inherently diverse in our physical appearance, personalities, backgrounds, experiences, beliefs, values, and more. This diversity is not only inevitable but also enriching, driving innovation, creativity, and progress. 2. Psychological theories are evolving: While some older theories in psychology might have focused on "normal" functioning, the field has increasingly acknowledged and embraced diversity. Concepts like positive psychology and cultural psychology highlight the importance of individual differences and encourage recognizing these differences as strengths rather than flaws. 3. Conforming to a majority can be detrimental: Trying to fit into a mold that doesn't resonate with your true self can lead to a variety of negative consequences, including inauthenticity, depression, anxiety, and a loss of personal growth. Embracing your individuality fosters well-being and allows you to contribute your unique perspective to the world. 4. Judging individuals for being different is harmful: Prejudice and discrimination based on any type of difference can lead to social exclusion, inequality, and injustice. Accepting and celebrating diversity is crucial for building a more inclusive and equitable society. Instead of aiming for a homogenous population, a healthier approach is to appreciate and respect individual differences. This means: Recognizing that different doesn't equal less: Just because someone is different doesn't mean they are inferior or wrong. We can learn from and value everyone's unique contributions. Promoting empathy and understanding: By striving to understand the perspectives and experiences of others, we can build bridges instead of walls. Creating inclusive environments: Fostering environments where everyone feels safe and respected for who they are, regardless of their differences, is key to a thriving and harmonious society. Remember, being different is not a flaw, it's what makes you, you! Embrace your individuality and celebrate the diverse tapestry of humanity. If you'd like to explore this topic further, I can provide you with resources on specific aspects of diversity and inclusion, or examples of positive movements that celebrate differences. Just let me know! So just be careful, im very pissed tbh, how it almost got me too. Good luck.

[@DasPuppy]

Dude.. I accidentally hit pause the moment the red screen was shown - I was SO confused for a second! I actually used the , and . to see if I actually saw it. Turns out, it was reproducable! It _is_ in fact there, and was not the youtube video player. Now on to what I wanted to do, if I can remember, and then continue the video.

[@neurotransmissions]

A true scientist 😂

[@SilviaHartmann]

The problem with psychology as a science is that it is sitting on a foundation of quicksand, rather than some well understood and well proven "laws of the universe." The first and oldest "studies" are the very worst from their most basic pre-suppositions to their methodology through to their "results." Trying to build on these old studies and/or taking them as any kind of foundation has directly led to psychology being completely useless in providing any kind of meaningful advances for giving people a better life today than they could have expected in the dark ages.

[@StrongMed]

Another awesome video!

[@neurotransmissions]

Aw thanks!! That means a lot coming from you!

[@daisyviluck7932]

Maybe non-replication isn’t an error. I mean, people aren’t born into a vacuum, but into a society, that’s complex and forever changing, so maybe what held true for 2 year olds in 1950 wouldn’t hold exactly the same for 2 year olds in 2020?

[@mikemondano3624]

There was never any scientific paper that said something like "coffee is good for you". Those conclusions are often arrived at by reporters.

[@hermanhale9258]

When I was in college in the seventies an anthro. teacher told the class it was the most common thing in the world for a grad student to check the records of a famous scientist and find his famous findings had been faked.

[@devinnemeyers4008]

What a wonderful video. Looking into neuroscience as a career path. I’m glad I watched this.

[@renscience]

The “observer” keeps failing in science as the observer affects and the observer mis-interprets the experiment. Especially true in quantum physics.

[@annediss8706]

Pretty much all my high school science students really want to find some correlation between variables. I remember decades ago my bio teacher saying it really didn’t matter what we found out, what mattered was knowing a rigorous method. Gotta love science!!

[@brianjones3191]

I seem to have missed the part where the ESP and pre-cognition studies were found to be non-replicable. My family have been telling me my entire life that my personal thoughts and memories I share are false. They believe they can read minds, and I think many others do as well. So, while it is probably mostly (completely?) nonsense, it is very intriguing that people KNOW they know what other people think-even when the subject tells them otherwise. Humans are insane.

[@davelowe1977]

You just need a simple rating/indexing scheme. Somebody publishes and it has a replication counter = 0. Each time the study is replicated then the counter is incremented. Fairly quickly you have a database of results and veracity. This could carry forward into the citation system. Each citation in a new paper carries a designation as normal and the replication counter of the citation e.g. "as described by Smith [1] (32)" where Smith's study was replicated 32 times. A journal could then make the decision to publish based partly on the sum of citation replications in a manuscript thus prioritising replicability rather than absolute result.

[@OL9245]

P value

[@chrisjhass]

Blatant fraud is hard to see when you believe that people lying is rare. Those people are in powerful positions and the only way to make it is to play the game which is a socital term for lying.

[@ruthbueneman541]

I knew there was something off about this effort to reduce truths about human behavior to something that can be measured or understood using rational thought.

[@celticc9580]

"Lets move on, it wasnt significant " brilliant 😅

[@coonhound_pharoah]

Simply changing the type of function (linear, exponential, power, etc.) can result in significant but also essentially random p-value changes, especially if the function has no reason to apply to the dataset. Math will do the work, whether reality actually fits the function or not.

[@Willow-r4q]

As a person who is currently studying clinical psychology to become a therapist - that situation scares me a lot. i clearly do not have time to double check all the info i need to learn and if it is unreliable - how am i going to help suffering people effectively? with what means? furthermore, i study in Russia, where science has a footprint of the Iron Curtain. We are learning "soviet science" and universal science all at once. so even more possible fake data, especcially in soviet science (cause the "bad" and "not valuable" paper could cost scientists their life, not only career).

[@weareallbornmad410]

You might also want to get rid of the hypothesis altogether. That's what we do in social anthropology.

[@harryplummer6356]

After 51 years in healthcare they were the only profession I had no time for. So much useless rubbish.

[@timothyvanderschultzen9640]

Science is a cruel mistress; but I love her.

[@velvetbees]

When were the first studies done vs now? I would expect some differences because culture changes over time and so does the knowledge and experiences of individuals. What kind of study and who is paying for it also matters.

[@roncarlin3209]

Part of the problem is that we have a "peer review" crisis. We have a new generation of "scientists", who cannot think critically, and are only capable of trawling the publications looking for peer reviewed papers, which they then take as sacrosanct. A lot of these "peer-reviewed" papers don't even pass the plausibility test...

[@neurotransmissions]

Not every peer-reviewed paper is mistake-free or anything, but what evidence do you have that peer review is letting in a bunch of implausible papers? If anything, it's often the opposite - denying papers that meet qualification for publication because one of the reviewers denied it for one reason or another.

[@ravanpee1325]

​@@neurotransmissionsCitation cartels and academic cliques are a big problem

[@sidraket]

You would think if anyone psychologists would be more aware of how their biases and desires skew their behaviors and try to account for it, and know that in trying to account for it they are actually just trying to trick themselves into thinking they are trying to account for it, and try to account for that.

[@hectornonayurbusiness2631]

Math is not the enemy, math is your friend.

[@alikhalessi]

I would love to know if similar investigations is done on “hard science” research and data

[@safir2241]

we will get there. in 500 years we will have done so many replication studies and new studies, and have so many new tools, but oh look, 500 years is a mere blip in human history. we are on a steady road, walking towards truthier places

[@Sheahne]

It’s sad that truth is rarely the goal. A researcher should be just as excited to discover that their hypothesis is wrong as they are to find it proven correct. That is the truth.

[@AliciaGuitar]

My issue with studies are the false conclusions based on correlations, not causations. The study will show a mere correlation, while the headline boldly proclaims causation. First rule of logic: correlation does not equal causation!

[@0x0michael]

I work for a virtue signaling Excel Fiction Artist Edit: Let me clarify, this is a researcher who is very vocal and critical about academic fraud when that is his expertise. He's a very influential VR researcher at a Business School (i am aware this is confusing)

[@tally551]

Omg, yes this is the problem i had with my dissertation and why i lost my love of personal research in psychology. Hunting that p-value. Screw that.

[@ryelor123]

Correlation does not imply causation. The problem that you're noticing actually has a simple explanation: institutions are scared and thus they become defensive at the cost of their actual jobs. A study claiming ESP exists isn't published because its interesting. Its published because it supports the class of people who run institutions. Say you did an experiment that proved that hillbillies and inner city black men get along better than either does with their local white bougeois class. Such a study could be done and almost certainly would say that however NO journal or magazine or university will promote such a study because it conflicts with the idea that the upper class is better than everyone else.

[@neurotransmissions]

That’s probably the silliest thing I’ve heard today

[@matthewkopp2391]

@@neurotransmissions​​⁠I attended a history of psychology class at the international psychoanalytic university of Berlin. And this was a major theme of the course as we compared psychology models of the Eastern Bloc, Liberal Democracies, and the Nazi era. And ideological societal bias is a large confirmation bias within psychology as to what is considered normal, what should be focused on, what is „cause“ of distress etc. In the Eastern Bloc it was often Freud mixed with Marxist dialectics, focusing on an objective analysis of your „material“ conditions. For example. So what makes anyone think that we are not in a similar trance of values and normalcy dependent on culture and history? The comment above maybe exaggerated, but I would argue that capitalist normative assumption plagues psychology because psychology often functions more like an ethical philosophy of particular historical and cultural milleau. In fact if you look at Robert J Lifton’s cult ideas we might just as easily apply that to culture as a whole. Societies police the milleau.

[@cord11ful]

Great job. Now do one on Dietetics/Nutrition research!

[@coolieo2222]

I wonder if some one created a publication company that only wanted the study’s that rot away in scientists desk drawer. Would people be interested in reading it?

[@hubrisonics9517]

A clinical psychologist asks their client: “How are you feeling today?” The client replies: “I’m feeling stoic.” The clinical psychologist then asks: “How do you feel about being stoic?”

[@lightbeingform]

My favorite take on the replication crisis so far, and that's saying a lot - all thanks to Halloween

[@Techdolphin200]

12:31

[@tiredidealist]

Forgive me if this was already mentioned, I was somewhat distracted and don't have the time right now to go back and check, but I think a good temporary solution would be having one or two journals dedicated to publishing 'non results' and similarly under-published studies.

[@stanthebamafan]

This reminds me of the dunning-Kruger effect, which is famous, but was basically debunked in recent years as being an artifact of bad statistical analysis. A statistician proved he could generate the same effect with randomly generated data.

[@christopherhamilton3621]

You should provide a citation here.

[@rokranged]

This is why I left academia. A bunch of egomaniacs manipulating data using statistical methods and publishing what they think is right. I love(d) science, but said a hard no thank you when I was asked to do the same.

[@j.f.7509]

A negative result is as valuable as a positive result for the advancement of scientific research. The issue is the pressure to come up with "tangible results". People don't like to hear "I didn't find any correlation" and such.

[@robertklem2638]

Very informative, thank you for your work. Cheers!

[@whimsinator2982]

Words like "embaressed" shouldn't be involved in any scientific pursuit. Why would any scientist worth their salt consider any study with that feeling as a contributor?