How Software Defaults Threaten Science

Science (and anything else) that relies on advanced statistical analysis is made possible these days through the use of computer software and programming languages. But that software comes with defaults, and those defaults influence results! How much are our scientific results dependent on some semi-arbitrary default that some coder selected years before we used that software on our data?
You can find GiveDirectly at givedirectly.org
Links to the articles referenced in the video, in order:
www.newscientist.com/article/...
journals.plos.org/plosbiology...
www.newyorker.com/magazine/20...
ryxcommar.com/2019/08/30/scik...
stats.stackexchange.com/quest...
stackoverflow.com/questions/5...
www.aeaweb.org/articles?id=10...
bcallaway11.github.io/did/
ideas.repec.org/c/boc/bocode/...
(again) stats.stackexchange.com/quest...

[@joffreymayer9194]

I have an economics background but found myself diving into the software-world since 2 years. It is incredible how many things are sneakily implicit when using a package / user-interfaces designed for non-programmers. The first time I realized those things can have massive implications on your results was when I was writing my master thesis: there, I got a half-cleaned dataset. However, when I tried to understand how some categorical variables were calculated (based on other values in another column), I found some errors in how those categories were formed. This variable happened to be my dependent variable. After I re-coded it correctly the results were very different. Starting from that day, I knew that paying close attention to the code „working under the hood“ (in my case, the results from others that used code to produce an output) is very important. This may be obvious for people trained in building software, but for people using software to build stuff / produce results (like researchers), it is important to highlight those „dependency problems“. So thanks alot for this video and the example that illustrates these issues! 😊

[@brodriguesco]

Very interesting! Another thing people don't think about is picking the seed... different seeds shouldn't have an impact on the result, but if they do...well...

[@lena6608]

Knock, knock! Just wanted to stop by and tell you that we have been reading your book for our causal inference seminar! I am a student at Seoul National University so your work is discussed here half around the globe!

[@NickHuntingtonKlein]

That's awesome! Thank you for letting me know, and hope you are enjoying the book.

[@lena6608]

@@NickHuntingtonKlein oh we are!

[@hippityhoppity657]

underrated

[@123eorl]

great video! although I feel like it could also (at least partly) be called 'lazy scientists threaten science'..

[@juliensiebert359]

Great video, my phd advisor used to say "the model is the code". By the way, numpy.average computes a weighted mean (where the default is no weight at all...)