Metascience Matters

Tim Errington is the Senior Director of Research at the Center for Open Science. He led the Reproducibility Project: Cancer Biology, Systematizing Confidence in Open Research and Evidence (SCORE), as well as the implementation and evaluation of initiatives such as Registered Reports, Registered Revisions, responsible conduct of research trainings, and open science badges.CONTACT RANDY:Feedback: metasciencematters@gmail.comEPISODE LINKS:Investigating the replicability of preclinical cancer biology (Reproducibility Project: Cancer Biology led by Tim Errington):https://elifesciences.org/articles/71601Bayer replication study:https://www.nature.com/articles/nrd3439-c1Amgen replication study:https://www.nature.com/articles/483531aReproducibility in Cancer Biology: Challenges for assessing replicability in preclinical cancer biology (Companion paper to Reproducibility Project: Cancer Biology):https://elifesciences.org/articles/67995What is replication?:https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000691Study comparing standard reports and registered reports in psychology:https://journals.sagepub.com/doi/10.1177/25152459211007467Blog post on the seemingly magical success of revision experiments:https://rajlaboratory.blogspot.com/2013/06/the-magical-results-of-reviewer.htmlGoogle's AI co-scientist paper:https://arxiv.org/abs/2502.18864Machine-readable documents:https://journals.sagepub.com/doi/10.1177/2515245920970949How open science helps researchers succeed:https://elifesciences.org/articles/16800ZBW's Expedition to Open Science Land:https://expedition-open-science.org/OUTLINE:0:00 - Introduction4:47 - Tim's origin story as a cancer biologist6:38 - Initial interest in metascience 9:24 - Starting the Reproducibility Project: Cancer Biology12:07 - How were the studies that were replicated chosen?14:41 - Publishing the Reproducibility Project: Cancer Biology as a registered report17:26 - Results from the Reproducibility Project: Cancer Biology20:28 - Tim's experience throughout the years running the Reproducibility Project: Cancer Biology25:21 - The difficulty of running cancer biology studies27:54 - Judging whether a replication is successful31:23 - What has the response to the Reproducibility Project: Cancer Biology been?37:52 - Why aren't replication rates higher?40:26 - Challenges of running cancer biology replication studies45:43 - Caveats of preclinical disease models49:13 - The incentive for positive data in science57:05 - Systemic intervention vs. Individual policing1:01:04 - The value of registered reports1:07:38 - Registered revisions1:10:48 - Falsifying theories early at the preclinical stage1:15:21 - Different institutions (e.g., academic, industry) conducting different studies (e.g., preclinical, clinical)1:17:34 - New initiatives at the Center for Open Science (Replication project of social/behavioral sciences, automated tools for predicting replication success, LifeCycle journal)1:23:02 - AI scientists are trained on biased literature; distrust of academic literature in drug discovery1:28:46 - Peer review1:32:51 - Narrative in science1:35:06 - 100-200 years into the future1:40:29 - Advice for high school/undergraduate listeners1:42:51 - Metascience manifests in every field1:44:46 - Philosophy of science1:47:49 - Outro

Brian Nosek is the cofounder and Executive Director of the Center for Open Science. He co-developed the Implicit Association Test, a method that advanced the study of implicit bias. He then co-founded three non-profit organizations: Project Implicit to advance research about implicit bias, the Society for the Improvement of Psychological Science to improve the research culture in his home discipline, and the Center for Open Science (COS) to improve rigor, transparency, integrity, and reproducibility across research disciplines. He led the Reproducibility Project in Psychology, a replication of 100 studies from psychology, as well as multiple other replication projects, along with policy reforms such as open science badges, and the investigation of prediction markets for study replication.CONTACT RANDY:Feedback: metasciencematters@gmail.comEPISODE LINKS:Estimating the reproducibility of psychological science (Reproducibility Project in Psychology led by Brian Nosek):https://www.science.org/doi/10.1126/science.aac4716False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant:https://journals.sagepub.com/doi/10.1177/0956797611417632What is replication?:https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000691Metascience conference:https://metascience.info/International Conference on the Science of Science and Innovation:https://icssi.org/International Congress on Peer Review and Scientific Publication:https://peerreviewcongress.org/MetaROR:https://metaror.org/OUTLINE:0:00 - Introduction2:56 - Brian's origin story as a psychologist6:35 - Initial interest in metascience 10:57 - "Scientific Utopia," starting the Reproducibility Project in Psychology, and major fraud cases17:57 - How did the Reproducibility Project in Psychology come together?23:40 - Results of the Reproducibility Project in Psychology27:03 - Response from the field, institutional changes30:39 - Advice for high school/undergrad listeners34:03 - Outro

Eugenie Reich is an attorney committed to taking on scientific fraud, understanding the incentives that drive it, and recovering misdirected research funding. She is also a former investigative science journalist committed to correcting the scientific record. Her 2009 book Plastic Fantastic, details a major fraud cause in physics at Bell Labs. Two of the cases she has litigated cases were against Biogen ($900 million settlement), and the Dana-Farber Cancer Institute ($15 million settlement).CONTACT RANDY:metasciencematters@gmail.comEPISODE LINKS:Cargo Cult Science:https://calteches.library.caltech.edu...The PubPeer Conundrum:https://www.tandfonline.com/doi/epdf/...New Scientist:https://www.newscientist.com/Skeptical Inquirer:https://skepticalinquirer.org/Special thanks to Dylan BouscherOUTLINE:0:00 - Introduction4:20 - How Eugenie became interested in science6:25 - Interest in scientific fraud7:38 - Deciding to become a science journalist8:58 - Fraud in physics at Bell Labs (the subject of her 2009 book Plastic Fantastic)20:15 - Bell Labs coming under private ownership22:53 - Interviewing scientists for her book26:50 - The response from the physics community31:10 - Deciding to become a lawyer33:28 - The False Claims Act36:29 - The qui tam provision of the False Claims Act37:46 - The $900M case against Biogen44:02 - The day-to-day of working on this case46:30 - Impact of the case on biotech and pharma48:40 - The $15M case against Dana-Farber50:42 - Do universities have an incentive to protect accused researchers?54:15 - Is the scale of fraud too large to be fixed?57:56 - Does fraud damage public trust in science?1:00:44 - Tools/solutions to combat these problems1:04:32 - Advice to practicing scientists1:05:55 - Advice and resources for listeners1:08:28 - Outro

James Heathers is the Founder and Director of the Medical Evidence Project, a venture of The Center for Scientific Integrity. He aims to reduce medical harm and improve patient outcomes by identifying and publicizing errors and miscondcut in the medical literature. He uses forensic meta-analytical techniques to detect and deconstruct errors arising from low-quality science and fraudulent work in areas that involve large numbers of patients.CONTACT RANDY:metasciencematters@gmail.comEPISODE LINKS:The original GRIM test paper: http://www1.psych.purdue.edu/~gfranci...Machine-readable documents: https://journals.sagepub.com/doi/10.1...Chaos in the Brickyard: https://www.science.org/doi/10.1126/s...Books:The Man Who Only Loved Numbers: https://en.wikipedia.org/wiki/The_Man...The Emperor of All Maladies:https://en.wikipedia.org/wiki/The_Emp...OUTLINE:0:00 - Introduction5:27 - How James has been the last couple days, and his radio voice6:40 - Writing the first book on Forensic Metascience10:47 - James' PhD work on heart rate variability14:12 - Starting to work on metascience18:40 - The GRIM test26:40 - Programmatically scaling statistical checks33:25 - Centering quantitative results and peripheralizing narrative in scientific papers39:36 - AI for metascience, unearthing data underneath narrative48:11 - Pursuing cases of misconduct52:45 - Founding and Directing the Medical Evidence Project56:10 - Incentives for positive data, post-publication review, challenging the binary of positive and negative data1:05:41 - Advice and resources for listeners1:12:40 - Optimism over pessimism1:16:58 - Outro

Mu Yang is a behavioral neuroscientist at Columbia University, and a scientific sleuth responsible for more than 300 retractions. She led an effort that discovered more than 130 fraudulent papers in the publication record of Eliezer Masliah, former head of the Division of Neuroscience at the National Institute on Aging at the National Institutes of Health. Her sleuthing work has been documented in the book "Doctored" by Charles Piller, Science Magazine, and other outlets, and is unpaid.CONTACT RANDY:metasciencematters@gmail.comEPISODE LINKS:Books: Doctored by Charles Piller: https://www.simonandschuster.com/books/Doctored/Charles-Piller/9781668031254Unreliable by Csaba Szabo: https://cup.columbia.edu/book/unreliable/9780231216241/Scientific integrity blogs: Dorothy Bishop: https://deevybee.blogspot.com/Leonid Schneider: https://forbetterscience.com/Podcasts for critical thinkingPlain English by Derek Thompson: https://www.theringer.com/podcasts/plain-english-with-derek-thompsonThe Gray Area by Sean Illing: https://www.vox.com/the-gray-areaThe Ezra Klein show (NYT): https://www.nytimes.com/column/ezra-klein-podcastOUTLINE:0:00 - Introduction2:58 - Mu's origin story4:35 - Moving to Columbia6:15 - How Mu became a sleuth8:13 - Reporting her first case13:09 - Red flags Mu looks for in papers17:30 - Reductionism in behavioral neuroscience18:04 - Standardization vs. Generalizability19:58 - Data sharing standards across fields21:09 - Difficulties of reporting irregularities in papers, university incentives23:54 - Allocating time between images, numerical, other kinds of data24:37 - How she searches through papers25:45 - Examining the chemistry literature31:10 - Types of misconduct vary by field, risks of reporting35:43 - The case of Eliezer Masliah40:31 - Why demonizing individual scientists isn't productive; the system isn't working56:59 - Academic incentives for positive data1:07:31 - Hard to publish null data; "unhealthy codependence" between academia and publishing1:13:08 - Changing incentives1:21:42 - Are we even making a dent in the scale of scientific misconduct? 1:27:35 - Mu's toolkit1:29:38 - Mu does this work because it's fun!1:34:38 - Protecting students; telling them that null data is ok1:37:52 - Evaluating researchers 1:43:15 - Is peer review still relevant?1:51:38 - How much better could science be?1:55:14 - What will science look like in a century?1:58:13 - Advice and resources for listeners2:00:54 - Online presence2:01:35 - Outro

Florian Naudet is a Professor of Therapeutics at Rennes University. As a metascientist and psychiatrist, his research interests lie in developing and evaluating methodological solutions to treatment assessment, primarily but not exclusively for mental health conditions. His work has also made inroads to quantifying and understanding research waste and the prevalence of substandard data-sharing practices.CONTACT RANDY:Feedback: metasciencematters@gmail.comEPISODE LINKS:Efficacy and safety of esketamine for “treatment resistant depression”: registered report for a systematic review with an individual patient data meta-analysis of randomized, double-blind, placebo-controlled trialshttps://link.springer.com/article/10....Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysishttps://link.springer.com/article/10....Pharmageddon (book) https://davidhealy.org/pharmageddon-i...A manifesto for reproducible sciencehttps://www.nature.com/articles/s4156...Using reporting guidelines to improve the reproducibility of cooking Christmas tree meringues: the “People tasting trees” cluster-randomised controlled trialhttps://pmc.ncbi.nlm.nih.gov/articles...OUTLINE:0:00 - Introduction0:36 - Florian's origin story as a psychiatrist and metascientist4:46 - How Florian became interested in metascience11:55 - Systematic review of clinical trials of Esketamine for depression28:45 - Publishing this study as a registered report40:49 - Vibration-of-effects analysis of double-blind randomized controlled trials assessing nalmefene and/or naltrexone for treating alcohol use disorders59:58 - Advice for listeners interested in pursuing research like Florian's 1:03:38 - Outro

Chirag Patel is an Associate Professor of Biomedical Informatics at Harvard Medical School, renowned for his expertise in using computational methods to understand human health and disease from high-throughput data streams. He specializes in understanding the role in the intersection of genetics and environmental exposures (the exposome) in human health, as well as various disease areas such as diabetes, cardiovascular disease, and Alzheimer’s disease.CONTACT RANDY:Feedback: metasciencematters@gmail.comEPISODE LINKS:Chirag’s Lab: https://www.chiragjpgroup.org/TEDx talk on the exposome:    • Exposome: decoding human health and diseas...  OUTLINE:0:00 - Introduction0:37 - How do you decide what to work on?1:57 - Where does metascience fit into your work?3:57 - Vibration-of-effects analysis6:41 - How does VoE change how we see existing scientific work?9:22 - The biggest challenges in the replication crisis within biomedical informatics11:48 - Environment-wide association study of Type 2 diabetes15:04 - The exposome17:30 - Policy vs. precision medicine applications of the exposome20:09 - The patient exposome22:38 - HBA1C for diabetes as a bridge to the exposome23:54 - Broader metascientific issues of the exposome25:01 - The effects of extreme weather events on human health29:35 - AI for biomedical informatics, the exposome, metascience31:19 - Advice for listeners interested in pursuing research like Chirag’s 32:53 - Outro