ReproducibiliTea Podcast

Today, Sarah, Will and Jan sit down to discuss the last season of ReproducibiliTea. We talk about Will's terrible terrible taste in puns, Jan's terrible taste in pizza, and Sarah's fall into FORRT(.org). While we are wrapping up Season 3, stay tuned for a few more fun & exciting episodes we have planned for this year! Not many footnotes this time: Sarah mentioned sysmus: https://sites.google.com/view/sysmus/home For more info go to ReproducibiliTea.org For comments, questions, tips and tricks, and praise use our feedback form: forms.gle/H6jgUzbbpyauLxUC8

Today, Will sits down with Max Korbmacher, Thomas Rhys Evans, and Flavio Azevedo, some of the authors of the paper "The replication crisis has led to positive structural, procedural, and community changes" to talk about the paper, FORRT, and Open Science communities. Show notes: The paper we discuss for this episode: Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., ... & Evans, T. (2023). The replication crisis has led to positive structural, procedural, and community changes. Nature Communications Psychology, 1(1), 3. https://www.nature.com/articles/s44271-023-00003-2 FORRT – The Framework for Open and Reproducible Research Training: https://forrt.org Getting involved with FORRT: https://forrt.org/about/get-involved/ Charlotte Pennington’s new book: A Student's Guide to Open Science: Using the Replication Crisis to Reform Psychology https://www.mheducation.co.uk/a-student-s-guide-to-open-science-using-the-replication-crisis-to-reform-psychology-9780335251162-emea-group UK Reproducibility Network: https://www.ukrn.org/ Project Teaching Integrity in Empirical Research (TIER): https://www.projecttier.org/ Reproducibility Wiki: https://replication.uni-goettingen.de/ Paper Trail: https://thepapertrailjc.squarespace.com/ Brain Imaging Data Structure (BIDS): https://bids.neuroimaging.io/ Collaborative Replication Education Project (CREP): https://www.crep-psych.org/ The Center for Open Science: https://www.cos.io/ Nowhere Lab: http://nowherelab.com/ Advancing Big-team Reproducible Science through Increased Representation (ABRIR): https://abrirpsy.org/ Open Life Science: https://openlifesci.org/ Turing Way: https://the-turing-way.netlify.app/index.html For more info go to ReproducibiliTea.org For comments, questions, tips and tricks use our feedback form: forms.gle/H6jgUzbbpyauLxUC8

Today, Will talks to David Reinstein about scientific publishing and The Unjournal. The Unjournal: https://www.unjournal.org The Unjournal on LinkedIn: https://www.linkedin.com/company/unjournal/ To get the latest updates: https://bit.ly/ujupdates To apply for positions at the Unjournal: https://bit.ly/Ujwork Will and David’s extended notes for the episode: https://docs.google.com/document/d/13rL6mq71GD6gPv5wBHr_lfX6YSKkLK9n6EAVfGhZhCk/edit?usp=sharing For more info go to https://ReproducibiliTea.org For comments, questions, tips and tricks use our feedback form: forms.gle/H6jgUzbbpyauLxUC8

Today, Will sits down with Björn Jorges and Sabrina Hansmann-Roth to discuss the role of academic societies in the science reform movement. The poster session: https://www.visionsciences.org/2023-pre-data-collection-poster-session-satellite/ Korbmacher, M., Azevedo, F., Pennington, C., Hartmann, H., Pownall, M., Schmidt, K., ... & Evans, T. (2023). The replication crisis has led to positive structural, procedural, and community changes. Communications Psychology. https://gala.gre.ac.uk/id/eprint/42730842730_EVANS_The_replication_crisis_has_led_to_positive_structural_procedural_and_community_changes.pdf As always: For more info go to ReproducibiliTea.org For comments, questions, tips and tricks use our feedback form: forms.gle/H6jgUzbbpyauLxUC8

Today Sarah chats to Zoltan Dienes Live from SIPS! Zoltan's keynote: https://www.youtube.com/watch?v=BxdGXLOC1Cc Reviewing labor: https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-021-00118-2?ref=refind Peer Community In Registered Reports: https://rr.peercommunityin.org/ Flourishing Science Think Tank paper: https://mindrxiv.org/4zrmd There are a few little audio problems in this weeks episode, so special shoutout to our transcript, which is here: https://docs.google.com/document/d/1zd4o40QLn0o5q1Eyp966EoDdvu5XtT6O/edit?usp=sharing&ouid=115964780222242468834&rtpof=true&sd=true For more info go to ReproducibiliTea.org For comments, questions, tips and tricks use our feedback form: forms.gle/H6jgUzbbpyauLxUC8

Today, Sarah is joined by Agata Bochynska and Matthew Good from the University of Oslo's Open Research Team to talk about how vital libraries are to Open Science. Links from this episode: QualiFAIR: https://www.uv.uio.no/ils/english/about/organization/tlvlab/qualifair/ TIER2: https://tier2-project.eu/ Carpentries: https://carpentries.org/ ReproducibiliTea at UiO: https://www.ub.uio.no/english/libraries/dsc/open-repro-research/reproducibilitea/index.html Preprint mentioned by Agata in the last segment: https://osf.io/kcvra/ Agata Twitter: https://twitter.com/AgataBochynska Agata mastodon: https://fediscience.org/@agata Digital Scholarship Centre website: https://www.ub.uio.no/english/libraries/dsc/ For more info go to ReproducibiliTea.org For comments, questions, tips and tricks use our feedback form: forms.gle/H6jgUzbbpyauLxUC8

Today, Sarah and Will discuss the invisible workload of making open science. The paper on invisible workload: https://journal.trialanderror.org/pub/the-invisible-workload/release/1 The replication crisis has led to positive structural, procedural, and community changes: https://osf.io/r6cvx/ For more info go to ReproducibiliTea.org For comments, questions, tips and tricks use our feedback form: forms.gle/H6jgUzbbpyauLxUC8

Today special guest Nora Serres talks with Sarah Sauve about Registered Report and how cool Bayesian statistics are. Show notes: Appetezer paper: Bayes factor design analysis: Planning for compelling evidence https://link.springer.com/article/10.3758/s13423-017-1230-y Nora’s thesis: https://www.duo.uio.no/bitstream/handle/10852/95323/5/Master-thesis_NS.pdf On registered reports: https://www.cos.io/initiatives/registered-reports https://osf.io/preprints/metaarxiv/x7aqr https://www.nature.com/srep/journal-policies/registered-reports PCI Registered Reports: https://rr.peercommunityin.org/PCIRegisteredReports/help/guide_for_authors On Bayesian/statistics: Bayesian Spectacles: https://www.bayesianspectacles.org/ Understanding Psychology as a Science by Zoltan Dienes: https://link.springer.com/book/9780230542303 For more info go to ReproducibiliTea.org For comments, questions, tips and tricks use our feedback form: forms.gle/H6jgUzbbpyauLxUC8 The transcript is available here: https://docs.google.com/document/d/1CCk__36WxOxuuFc1JX8lou03HvGVRci4/edit?usp=sharing&ouid=115964780222242468834&rtpof=true&sd=true

Today, Will and Jan sit down with special guest and author of the paper "A large-scale study on research code quality and execution" Ana Trisovic to talk about the reproducibility of analysis code. Also windows. Links from this episode: Ana's paper: https://www.nature.com/articles/s41597-022-01143-6 Gratitude package: https://github.com/Pakillo/grateful For more info go to ReproducibiliTea.org For comments, questions, tips and tricks use our feedback form: forms.gle/H6jgUzbbpyauLxUC8

! Important Update ! Since recording, UCU have announced that they will "ringfence £250,000 for members facing deductions for undertaking forms of ASOS, including the marking and assessment boycott." This is reassuring news for workers at universities where the VCs have threatened 50-100% pay deductions. But still no new offers from employers. Today, Sophia and Jan are sitting down with @IlsePit(@fediscience.org) and @KatarinaWarren (k-almeidawarren.com) to chat about the ongoing UCU strikes in the UK, and how it is for ECRs to take industrial actions. "The University and College Union (UCU) represents over 120,000 academics, lecturers, trainers, instructors, researchers, managers, administrators, computer staff, librarians, technicians, professional staff and postgraduates in universities, colleges, prisons, adult education and training organisations across the UK" (https://www.ucu.org.uk/article/1685/About-UCU) You can support the UCU here: https://www.ucu.org.uk/ For more info go to https://ReproducibiliTea.org For comments, questions, tips and tricks use our feedback form: forms.gle/H6jgUzbbpyauLxUC8

Will, Helena and Jan talk about how we assess research and that it is weird we never really learned how to do Peer Review. Papers we discussed: Responsible Research Assessment Should Prioritize Theory Development and Testing Over Ticking Open Science Boxes: https://psyarxiv.com/ad74m/ A consensus-based tool for evaluating threats to the validity of empirical Research: https://psyarxiv.com/fc8v3 Will's blog post on peer review: https://williamngiam.github.io/blog/my_peer_review_process The tool for checking p-Values: statcheck.io The tool for checking mean values: https://journals.sagepub.com/doi/10.1177/1948550616673876 For more info go to ReproducibiliTea.org. Do you have questions? Comments? Wanna say hi? You can reach us here: https://forms.gle/H6jgUzbbpyauLxUC8

We mentioned lots of resources this week, brace yourselves! Books that changed our worlds How To Read a Paper: The Basics of Evidence-Based Medicine by Trisha Greenhalgh https://www.amazon.co.uk/How-Read-Paper-Evidence-Based-Medicine/dp/1444334360 How to Win Campaigns by Chris Rose https://www.amazon.co.uk/How-Win-Campaigns-Steps-Success/dp/1853839620 Pollution is Colonialism by Max Liboiron https://www.dukeupress.edu/pollution-is-colonialism A Student’s Guide to Open Science by Charlotte Pennington https://www.mheducation.co.uk/a-student-s-guide-to-open-science-using-the-replication-crisis-to-reform-psychology-9780335251162-emea-group The Art of Statistics: Learning from Data by David Spiegelhalter https://www.penguin.co.uk/books/294857/the-art-of-statistics-by-spiegelhalter-david/9780241258767 Tools we mention: FORRT Academic Wheel of Privilege: https://www.youtube.com/watch?v=mzEdTyA06cU&ab_channel=FORRTproject Ioannidis et al. (2019): https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000384 who provide a means to adjust research impact for self-citations using a newly developed tool Baccini et al. (2019): https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0221212 calculate an “inwardness” metric which shows bias to researchers’ own country Citation Diversity Statement: tools that can help authors assess the citation diversity statistics of their papers (see, for example, https://github.com/dalejn/cleanBib#instructions) Jane Lawrence Sumner’s Gender Balance Assessment Tool (GBAT) https://jlsumner.shinyapps.io/syllabustool/ Groups/projects working on highlighting under-cited work: ABRIR (https://abrirpsy.org/), Marginalia (https://www.marginaliascience.com/) and FORRT (https://forrt.org/neurodiversity/) Other relevant links/resources: Evidence-Based Toxicology https://www.tandfonline.com/journals/tebt20 Evidence-Based Toxicology Collaboration https://www.ebtox.org/ Living a Feminist Life by Sara Ahmed Peer review episodes of Secret Feminist Agenda: https://secretfeministagenda.com/2020/04/07/bonus-episode-season-3-peer-review-of-secret-feminist-agenda/ and https://secretfeministagenda.com/2020/04/07/bonus-episode-response-to-review-of-season-3/ Algorithms are racist & sexist: Noble, S. U. (2018). Algorithms of oppression: How search engines reinforce racism. nyu Press. #MHAWS: Mirya Holman's Aggressive Winning Scholars Newsletter: https://miryaholman.substack.com/ MetaDocencia - https://www.metadocencia.org/

We are back! And we brought... a whole list of reading recommendations! Hosted and Produced by @Sarah_Sauve, @Will_Ngiam, and @VornhagenJB@hci.social Edited by Jan Vornhagen For more information visit ReproducibiliTea.org and send your feedback here: https://forms.gle/8nNLZ92YUcU1mGhc6. For a transcript, please refer to our youtube video: https://youtu.be/S7Ng6t34cSQ This weeks tasty Tea-dbit/Appe-tea-zer: Dienes' paper on democratic governance: https://royalsocietypublishing.org/doi/10.1098/rsos.220808 Things we talked about: Dienes' paper on democratic governance: https://royalsocietypublishing.org/doi/10.1098/rsos.220808 Paul Feyerabend "Against Method" Seductions of Clarity by C Thi Nguyen: https://philarchive.org/archive/NGUTSO-2 Transparency is Surveillance by C Thi Nguyen: https://onlinelibrary.wiley.com/doi/abs/10.1111/phpr.12823 Dissident Knowledge in Higher Education: https://uofrpress.ca/Books/D/Dissident-Knowledge-in-Higher-Education Aspiring to greater intellectual humility in science: https://www.nature.com/articles/s41562-021-01203-8 The questionable positionality statements paper: https://journals.sagepub.com/doi/abs/10.1177/17456916221144988 Also definitely check out Charlotte Penningtons new book "A student's guide to open science": https://forrt.org/educators-corner/014-students-guide-to-open-science/

Sophia and Jan serendipitously meet at a conference and immediately drag special guests @lonnibesancon, @lahariG, Wesley Willett into a room to talk about transparency, Open Science and Human-Computer Interaction. For more Infos about the seminar visit https://www.dagstuhl.de/en/program/calendar/semhp/?semnr=22392 Or go to ReproducibiliTea.org If you have questions or comments, throw them into the pot: forms.gle/HXHHvrNHn54MVHKb9

Today, Sam and Will have choice words about a website. For more Infos go to ReproducibiliTea.org If you have questions or comments, throw them into the pot: forms.gle/HXHHvrNHn54MVHKb9 Transcript: https://drive.google.com/file/d/1TL9lnW_HOL_BuN_wRDrWZb7e-kWUZedr/view?usp=sharing

Links from this Episode: - Paper on the limits of open data for music science: https://emusicology.org/article/view/7646 - Feminist papers on open science: https://journals.sagepub.com/doi/pdf/10.1177/03616843211030926?casa_token=x6twMszrtQgAAAAA:7Wu2Z2V0X58Lo403dJPiLT3jL65YBErd4f6FE_bhcfnie_Sg4oqax9yaMF1R02WEsB4KKNvLu7b32xs and https://journals.sagepub.com/doi/full/10.1177/03616843211036460?casa_token=ZbT7O7sn04YAAAAA:gQ3xo-t3aEh38sIVEfs5SAMa_hJwz_iyDnjuM_q6J-7k8gLn-RuWxtuu-baSkGc1QcybOCFftdAncY0 - My Shiny App: https://sarahsauve.shinyapps.io/BrandAuditDashboard/ -Simulating LMEMs: https://shiny.psy.gla.ac.uk/lmem_sim/ -Preprint of the paper I mention working on reviews for: https://psyarxiv.com/bt6zn/ -That paper covers issues of universality and ethics with citations, but here are two major ones: Decolonizing Methodologies by Linda Tuhiwai Smith and https://journals.sagepub.com/doi/10.1177/1747016117739935 -Will's shiny app: https://williamngiam.shinyapps.io/CDAPower/ -A billion-dollar donation: Estimating the Cost of Researchers’ Time Spent on Peer Review by Balazs Aczel, Barnabas Szaszi, Alex O. Holcombe: https://osf.io/preprints/metaarxiv/5h9z4/ For more Infos go to ReproducibiliTea.org If you have questions or comments, throw them into the pot: https://forms.gle/HXHHvrNHn54MVHKb9

Find Kohinoor here: https://www.kohinoordarda.com/ Find all the stuff mentioned in this episode here: The talk by a former senior admin, Dr. Max Liboiron: https://www.youtube.com/watch?v=Rya5Gom5o20 The book on the not-for-profit industrial complex: https://www.dukeupress.edu/the-revolution-will-not-be-funded The organization Sarah is a part of and mentions in the episode, the Social Justice Cooperative of Newfoundland and Labrador: https://www.sjcnl.ca/ The Preprint on Open Science in India: https://osf.io/preprints/aj9gw/ Open data sharing and the global south, who benefits? https://www.science.org/doi/10.1126/science.aap8395 Psychology should generalise from - and not just to- Africa https://www.nature.com/articles/s44159-022-00070-y As usual, you can find more info on ReproducibiliTea here: ReproducibiliTea.org/ Questions? Answers? Comments? Feel free to drop them here: forms.gle/bRFcfiGQof43stoq6 Transcript: https://drive.google.com/file/d/1KyTSdjN8H9qrCWvTbpBSKcLTvt6b6MSZ/view?usp=sharing

What is the point of having a T in your name without being able to use it for a tea pun? ConTeabutorship was right there. More information on CRediT here: https://credit.niso.org/ And more info on ReproducibiliTea here: https://ReproducibiliTea.org/ Questions? Answers? Comments? Feel free to drop them here: forms.gle/bRFcfiGQof43stoq6 Transcript: https://drive.google.com/file/d/1xi7sm5ExV7_sNejMIFUdH3CU7J4LWypH/view?usp=sharing

Today, Sarah, Will, Sam and Jan discuss what kind of community we would like to foster and how you can get involved. One of those ways, send us your questions and comments, in audio or text right here: https://forms.gle/bRFcfiGQof43stoq6 Another way: Find us on Twitter as @ReproducibiliT or send uns an E-Mail via ReproducibiliTea.org. The latter is also the best place for all infos regarding how to start your own journal club and get involved! Courtesy of Will, please find the transcript here: https://drive.google.com/file/d/1C3P0oc0tf5gje2jUyahhNPvSdBRznREr/view?usp=sharing

Sarah and Sam take a meandering discussion through slow science and what "slow" actually means in this context. Along the way, they also discuss some recommendations to promote slow science made in "Fast Lane to Slow Science" and how often scholarly critique often runs ahead of this slower pace. Links and references: Frith, U. (2020). Fast lane to slow science. Trends in Cognitive Sciences, 24(1), 1-2. Pownall https://www.bps.org.uk/psychologist/slow-science-scholarly-critique https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ For more infos, visit ReproducibiliTea.org