DiscoverNullius in Verba
Claim Ownership
Nullius in Verba
Author: Smriti Mehta and Daniël Lakens
Subscribed: 53Played: 1,191Subscribe
Share
© Copyright {2023} All Rights Reserved
Description
Nullius in Verba is a podcast about science—what it is and what it could be. It is hosted by Smriti Mehta from UC Berkeley and Daniël Lakens from Eindhoven University of Technology.
We draw inspiration from the book Novum Organum, written in 1620 by Francis Bacon, which laid the foundations of the modern scientific method. Our logo is an homage to the title page of Novum Organum, which depicts a galleon passing between the mythical Pillars of Hercules on either side of the Strait of Gibraltar, which have been smashed by Iberian sailors to open a new world for exploration. Just as this marks the exit from the well-charted waters of the Mediterranean into the Atlantic Ocean, Bacon hoped that empirical investigation will similarly smash the old scientific ideas and lead to a greater understanding of the natural world.
The title of the podcast comes from the motto of the Royal Society, set in typeface Kepler by Robert Slimbach. Our theme song is Newton’s Cradle by Grandbrothers.
We draw inspiration from the book Novum Organum, written in 1620 by Francis Bacon, which laid the foundations of the modern scientific method. Our logo is an homage to the title page of Novum Organum, which depicts a galleon passing between the mythical Pillars of Hercules on either side of the Strait of Gibraltar, which have been smashed by Iberian sailors to open a new world for exploration. Just as this marks the exit from the well-charted waters of the Mediterranean into the Atlantic Ocean, Bacon hoped that empirical investigation will similarly smash the old scientific ideas and lead to a greater understanding of the natural world.
The title of the podcast comes from the motto of the Royal Society, set in typeface Kepler by Robert Slimbach. Our theme song is Newton’s Cradle by Grandbrothers.
45 Episodes
Reverse
In advance of the next three episodes discussing the Philosophical Psychology lectures by Paul E. Meehl, we present a brief reading from his autobiography in A history of psychology in autobiography.
Meehl, P. E. (1989). Paul E. Meehl. In G. Lindzey (Ed.), A history of psychology in autobiography (Vol. 8, pp. 337–389). Stanford, CA: Stanford University Press.
In this episode, we discuss objectivity and disinterestedness in science. We talk about norms, values, interests, and objectivity in research practice, peer review, and hiring decisions. Is it possible to be completely objective? Is objectivity a feature of epistemic products or epistemic processes? And most importantly, how would you objectively rate this podcast?
Shownotes
Armstrong, J. S. (1979). Advocacy and objectivity in science. Management Science, 25(5), 423–428.
Declaration of Interest by Stephen Senn: http://senns.uk/Declaration_Interest.htm
Djørup, S., & Kappel, K. (2013). The norm of disinterestedness in science; a restorative analysis. SATS, 14(2). https://doi.org/10.1515/sats-2013-0009
Elliott, K. C. (2017). A Tapestry of Values: An Introduction to Values in Science. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780190260804.001.0001
Feyerabend, Paul. "How to defend society against science." Philosophy: Basic Readings (1975): 261-271.
Jamieson, K. H., McNutt, M., Kiermer, V., & Sever, R. (2019). Signaling the trustworthiness of science. Proceedings of the National Academy of Sciences, 116(39), 19231–19236. https://doi.org/10.1073/pnas.1913039116
Janack, M. (2002). Dilemmas of objectivity. Social Epistemology, 16(3), 267-281.
John, S. (2021). Objectivity in science. Cambridge University Press.
Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. University of Chicago Press.
Mitroff, I. I. (1974). Norms and Counter-Norms in a Select Group of the Apollo Moon Scientists: A Case Study of the Ambivalence of Scientists. American Sociological Review, 39(4), 579–595. https://doi.org/10.2307/2094423
Mitroff, I. I. (1974). The subjective side of science: A philosophical inquiry into the psychology of the Apollo moon scientists (First Edition). Elsevier.
A Russian polar researcher has been charged trying to stab a colleague to death at a remote Antarctic base https://www.businessinsider.com/sergey-savitsky-alleged-attempted-murder-at-antarctic-bellingshausen-2018-10
Stamenkovic, P. (2023). Facts and objectivity in science. Interdisciplinary Science Reviews, 1–22. https://doi.org/10.1080/03080188.2022.2150807
In this episode, we discuss the role of criticism in science. When is criticism constructive as opposed to obsessive? What are the features of fair and useful scientific criticism? And should we explicitly teach junior researchers to both give and accept criticism?
Shownotes:
Babbage, C. (1830). Reflections on the Decline of Science in England: And on Some of Its Causes.
Prasad, Vinay, and John PA Ioannidis. "Constructive and obsessive criticism in science." European journal of clinical investigation 52.11 (2022): e13839.
Lakatos, I. (1968, January). Criticism and the methodology of scientific research programmes. In Proceedings of the Aristotelian society (Vol. 69, pp. 149-186). Aristotelian Society, Wiley.
LOWI: https://lowi.nl/en/home/ As an independent advisory body it plays a role in the complaints procedure about alleged violations of principles of research integrity.
Holcombe, A. O. (2022). Ad hominem rhetoric in scientific psychology. British Journal of Psychology, 113(2), 434–454. https://doi.org/10.1111/bjop.12541
Daniel C. Dennett: I've Been Thinking https://wwnorton.com/books/9780393868050
Phillip Stark textbook chapter on logical fallacies: https://www.stat.berkeley.edu/~stark/SticiGui/Text/reasoning.htm
Gelman, A., & Tuerlinckx, F. (2000). Type S error rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics, 15(3), 373–390. https://doi.org/10.1007/s001800000040
Popper, K. R. (1959). The logic of scientific discovery. Routledge.
PubPeer: https://pubpeer.com
In this episode, we continue discussing Dubin’s 8-step method for theory building. We discuss the measurement of theoretical constructs, using logical propositions to make falsifiable predictions from theories, and the importance of specifying boundary conditions.
Shownotes
Jaccard, J., & Jacoby, J. (2010). Theory Construction and Model-building Skills: A Practical Guide for Social Scientists. Guilford Press.
McGuire, W. J. (1973). The yin and yang of progress in social psychology: Seven koan. Journal of Personality and Social Psychology, 26(3), 446–456. https://doi.org/10.1037/h0034345
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on Generality (COG): A Proposed Addition to All Empirical Papers. Perspectives on Psychological Science, 12(6), 1123–1128. https://doi.org/10.1177/1745691617708630
Norm Macdonald: The Professor of Logic
Raven Paradox: https://platonicrealms.com/encyclopedia/Hempels-Ravens-Paradox
Pavlov, I. (1936). Bequest of Pavlov to the Academic Youth of His Country. Science, 83(2155), 369–370. https://doi.org/10.1126/science.83.2155.369
In this episode we discussed the 8-step method of theory building proposed by Robin Dubin in his classic 1969 book Theory Building.
Shownotes
Dubin, R. (1969). Theory building. Free Press. http://catalog.hathitrust.org/api/volumes/oclc/160506.html
Lynham, S. A. (2002). Quantitative Research and Theory Building: Dubin’s Method. Advances in Developing Human Resources, 4(3), 242–276. https://doi.org/10.1177/15222302004003003
Elms, A. C. (1975). The crisis of confidence in social psychology. American Psychologist, 30(10), 967.
Meehl, P. E. (1978). Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology. Journal of Consulting and Clinical Psychology, 46(4), 806–834. https://doi.org/10.1037/0022-006X.46.4.806
Swedberg, R. (2014). The art of social theory. Princeton University Press.
Ben Wright: https://en.wikipedia.org/wiki/Benjamin_Drake_Wright
Yarkoni, T., & Westfall, J. (2017). Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning. Perspectives on Psychological Science, 12(6), 1100–1122. https://doi.org/10.1177/1745691617693393
Isaac, M. G., Koch, S., & Nefdt, R. (2022). Conceptual engineering: A road map to practice. Philosophy Compass, 17(10), e12879. https://doi.org/10.1111/phc3.12879
In this episode, we discuss the barriers to cumulative science, including inconsistent measurement tools, overreliance on single studies, and the large volume of research publications. Can replications, interdisciplinary collaborations, and prospective meta-analyses help us solve this issue? Can AI solve all our problems? And do most scientists treat their theories like toothbrushes?
Shownotes
Opening quote by George Sarton
Sarton, G. (1927). Introduction to the History of Science (Vol. 376).
Is Science Cumulative? a Physicist Viewpoint: https://link.springer.com/chapter/10.1007/978-1-4020-6279-7_10
Psychological Methods. (2009). Special Issue: Multi-Study Methods for Building a Cumulative Psychological Science.
Walter Mischel, Becoming a Cumulative Science
Dorothy Bishop - Why we need cumulative science (AIMOS)
Watkins, J. W. (1984). Science and Skepticism. Princeton University Press.
A reading of:
Forscher, B. K. (1963). Chaos in the Brickyard. Science, 142(3590), 339–339. https://doi.org/10.1126/science.142.3590.339
In today’s episode, we continue our conversation about preregistration. How flexible can we be when we preregister, without increasing flexibility in our analysis? How well do people preregister, and what does a good preregistration look like? And how do we deal with deviations from preregistrations?
Shownotes
Dubin, R. (1969). Theory building. Free Press. His full quote is: "There is no more devastating commendation that the self-designated theorist makes of the researcher than to label his work purely descriptive".
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), 211037. https://doi.org/10.1098/rsos.211037
Akker, O. van den, Bakker, M., Assen, M. A. L. M. van, Pennington, C. R., Verweij, L., Elsherif, M., Claesen, A., Gaillard, S. D. M., Yeung, S. K., Frankenberger, J.-L., Krautter, K., Cockcroft, J. P., Kreuer, K. S., Evans, T. R., Heppel, F., Schoch, S. F., Korbmacher, M., Yamada, Y., Albayrak-Aydemir, N., … Wicherts, J. (2023). The effectiveness of preregistration in psychology: Assessing preregistration strictness and preregistration-study consistency. MetaArXiv. https://doi.org/10.31222/osf.io/h8xjw
Sequential analysis and alpha spending functions https://lakens.github.io/statistical_inferences/10-sequential.html
Bishop, D. V. M. (2018). Fallibility in Science: Responding to Errors in the Work of Oneself and Others. Advances in Methods and Practices in Psychological Science, 2515245918776632. https://doi.org/10.1177/2515245918776632
FDAAA Trial Tracker https://fdaaa.trialstracker.net
Ensinck, E., & Lakens, D. (2023). An Inception Cohort Study Quantifying How Many Registered Studies are Published. PsyArXiv. https://doi.org/10.31234/osf.io/5hkjz
Quantitude episode on preregistration https://quantitudepod.org/s3e07-in-defense-of-researcher-degrees-of-freedom/
Lakens, D. (2023). When and How to Deviate from a Preregistration. PsyArXiv. https://doi.org/10.31234/osf.io/ha29k
In this two part episode we discuss the fine art of preregistration. We go back into the history of preregistration, its evolution, and current use. Do we preregister to control the Type 1 error rate, or to show that we derived our prediction from theory a priori? Can and should we preregister exploratory or secondary data analysis? And how severe is the issue of severe testing?
Shownotes
ClinicalTrials.gov
You can preregister on AsPredicted and the OSF
Johnson, M. (1975). Models of Control and Control of Bias. European Journal of Parapsychology, 36–44.
SPIRIT Checklist
Bishop, D. V. M. (2018). Fallibility in Science: Responding to Errors in the Work of Oneself and Others. Advances in Methods and Practices in Psychological Science, 1(3), 432–438. https://doi.org/10.1177/2515245918776632
FDA trials tracker: https://fdaaa.trialstracker.net
Ensinck, E., & Lakens, D. (2023). An Inception Cohort Study Quantifying How Many Registered Studies are Published. PsyArXiv. https://doi.org/10.31234/osf.io/5hkjz
van den Akker, O. R., van Assen, M. A. L. M., Enting, M., de Jonge, M., Ong, H. H., Rüffer, F., Schoenmakers, M., Stoevenbelt, A. H., Wicherts, J. M., & Bakker, M. (2023). Selective Hypothesis Reporting in Psychology: Comparing Preregistrations and Corresponding Publications. Advances in Methods and Practices in Psychological Science, 6(3), 25152459231187988. https://doi.org/10.1177/25152459231187988
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of preregistered studies. Royal Society Open Science, 8(10), 211037. https://doi.org/10.1098/rsos.211037
Bakan, D. (1966). The test of significance in psychological research. Psychological Bulletin, 66(6), 423–437. https://doi.org/10.1037/h0020412
Rosenthal, R. (1966). Experimenter effects in behavioral research. Appleton-Century-Crofts.
Johnson, M. (1975). Models of Control and Control of Bias. European Journal of Parapsychology, 36–44.
de Groot, A. D. (1969). Methodology. Mouton & Co.
Claesen, A., Lakens, D., Vanpaemel, W., & Dongen, N. van. (2022). Severity and Crises in Science: Are We Getting It Right When We’re Right and Wrong When We’re Wrong? PsyArXiv. https://doi.org/10.31234/osf.io/ekhc8
In the first episode of 2024, we discuss the double-edged sword: reverence to authority. Should scientists respect others on whose shoulders they stand? Or should they be wary of appeal to authority? How should scientists deal with other sources of authority in science, like for example, the government or academic societies? And how can we differentiate true expertise from mere authority? Enjoy.
Shownotes
Frank, P. (1956). The role of authority in the interpretation of science. Synthese, 10, 335–338.
Barber, B. (1952). Science and the social order. Glencoe, Ill. : Free Press. http://archive.org/details/sciencesocialord0000barb
Barber, B. (1961). Resistance by Scientists to Scientific Discovery. Science, 134(3479), 596–602.
Kitcher, P. (1992). Authority, deference, and the role of individual reasoning in science. In E. Mcmullin (Ed.), The social dimensions of science. Notre Dame: The University of Notre Dame Press
Polanyi, M. (1962). The republic of science. Minerva, 1(1), 54–73
The practice of two-spaces after the end of a sentence comes from when type-writers used monospaced typefaces: https://slate.com/technology/2011/01/two-spaces-after-a-period-why-you-should-never-ever-do-it.html
The Fixation of Belief. Charles S. Peirce. Popular Science Monthly 12 (November 1877), 1-15.
http://peirce.org/writings/p107.html
In this second installment of The Anticreativity Letters, we continue discussing the Tempter's tactics for stifling creativity and how to overcome them.
In the first of a two-part episode, we discuss The Anticreativity Letters by Richard Nisbett, in which a senior "tempter" advises a junior tempter on ways to prevent a young psychologist from being a productive and creative scientist.
Nisbett, R. E. (1990). The anticreativity letters: Advice from a senior tempter to a junior tempter. American Psychologist, 45(9), 1078–1082.
BMJ Christmas issue: https://www.bmj.com/about-bmj/resources-authors/article-types/christmas-issue
Quote by Ira Glass: https://www.goodreads.com/quotes/309485-nobody-tells-this-to-people-who-are-beginners-i-wish
A reading of:
Nisbett, R. E. (1990). The anticreativity letters: Advice from a senior tempter to a junior tempter. American Psychologist, 45(9), 1078–1082. https://doi.org/10.1037/0003-066X.45.9.1078
In today’s episode, we discuss the role of mentorship in academia. What are the characteristics of a good mentor-mentee relationship? What are the qualities of good mentors and good mentees? Does mentorship play a role in the development of scientific knowledge? And could mentors and mentees benefit from couples therapy?
Note: D.I.H.C is pronounced 'dick' but this is meant to be a family-friendly podcast :)
Shownotes
https://www.psychologytoday.com/us/blog/emotional-fitness/201303/10-things-your-relationship-needs-thrive
Roberts, L. R., Tinari, C. M., & Bandlow, R. (2019). An effective doctoral student mentor wears many hats and asks many questions. International Journal of Doctoral Studies, 14, 133.
Sarabipour, S., Niemi, N. M., Burgess, S. J., Smith, C. T., Filho, A. W. B., Ibrahim, A., & Clark, K. (2023). Insights from a survey of mentorship experiences by graduate and postdoctoral researchers (p. 2023.05.05.539640). bioRxiv. https://doi.org/10.1101/2023.05.05.539640
In this episode, we discuss the role of trust in science. Why should we verify but trust other scientists? What are the prerequisites for building trust within the scientific community? Who is ultimately responsible for verifying our claims and practices that bolster those claims? And should we give personality tests to everyone who enters academia?
Shownotes
Hardwig, J. (1991). The role of trust in knowledge. The Journal of Philosophy, 88(12), 693–708.
Hendriks, F., Kienhues, D., Bromme, R. (2016). Trust in Science and the Science of Trust. In: Blöbaum, B. (eds) Trust and Communication in a Digitized World. Progress in IS. Springer, Cham. https://doi.org/10.1007/978-3-319-28059-2_8
Strand, J. F. (2023). Error tight: Exercises for lab groups to prevent research mistakes. Psychological Methods, No Pagination Specified-No Pagination Specified. https://doi.org/10.1037/met0000547
Duygu Uygun-Tunç: Trust and criticism in science, Part I: Critical rationalism instead of organized skepticism: https://uyguntunc.wordpress.com/2020/10/30/trust-and-criticism-in-science-part-i-critical-rationalism-instead-of-organized-skepticism/
Vazire, S. (2017). Quality Uncertainty Erodes Trust in Science. Collabra: Psychology, 3(1), 1. https://doi.org/10.1525/collabra.74
Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case. Nature, 480(7375), Article 7375. https://doi.org/10.1038/480007a
Fricker, E. (2002). Trusting others in the sciences: A priori or empirical warrant? Studies in History and Philosophy of Science Part A, 33(2), 373–383. https://doi.org/10.1016/S0039-3681(02)00006-7
In advance of our episode Verify but Trust, a reading of John Hardwig's paper The Role of Trust in Science.
Hardwig, J. (1991). The role of trust in knowledge. The Journal of Philosophy, 88(12), 693–708.
In today’s episode, we discuss the peer review process---its history, its present, and its future. How does peer review work? How long has it existed in its current form? Should reviews be open and signed? Should reviewers be paid for their hard labor? Should we just abandon the peer review process, or does it have a positive role to play?
Shownotes
Peer Community in Registered Reports: https://rr.peercommunityin.org/
Suggestion to Darwin to publish a book about pigeons instead of The Origins of Species: https://www.darwinproject.ac.uk/letter/DCP-LETT-2457A.xml
Baldwin, M. (2018). Scientific Autonomy, Public Accountability, and the Rise of “Peer Review” in the Cold War United States. Isis, 109(3), 538–558. https://doi.org/10.1086/700070
Burnham, J. C. (1990). The evolution of editorial peer review. JAMA, 263(10), 1323–1329.
In this episode, we discuss Quantifauxcation, described by statistician Philip Stark as “situations in which a number is, in effect, made up, and then is given credence merely because it is quantitative.” We give examples of quantifauxcation in psychology, including errors of the third kind. We spend the second half of the podcast discussing how to develop quantitative measures that are meaningful and bridge the divide between qualitative and quantitative observations.
Shownotes
Statistics textbook by Philip Stark.
Stark, P. B. (2022). Pay No attention to the model behind the curtain. Pure and Applied Geophysics, 179(11), 4121–4145. https://doi.org/10.1007/s00024-022-03137-2
Burgess, E. W. (1927). Statistics and case studies as methods of sociological research, Vol 12(3), 103-120. (Thanks to Andy Grieve!)
Nick Brown's role in pointing out flaws in the positivity ratio.
Retraction notice of the positivity ratio paper.
Blog by Tania Lombrozo on nonsensical formulas in abstracts.
Kimball, A. W. (1957). Errors of the third kind in statistical consulting. Journal of the American Statistical Association, 52(278), 133–142. https://doi.org/10.1080/01621459.1957.10501374
Type III errors: Philip Stark’s post of Deborah Mayo’s blog
Brower, D. (1949). The problem of quantification in psychological science. Psychological Review, 56(6), 325–333. https://doi.org/10.1037/h0061802
Guttman scales
Wilson, M. (2023). Constructing measures: An item response modeling approach. Taylor & Francis.
Wilson, M., Bathia, S., Morell, L., Gochyyev, P., Koo, B. W., & Smith, R. (2022). Seeking a better balance between efficiency and interpretability: Comparing the likert response format with the Guttman response format. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000462
Bhatti, H.A., Mehta, S., McNeil, R., Wilson, M. (2023). A scientific approach to assessment: Rasch measurement and the four building blocks. In X. Liu & W. Boone (Eds.), Advances in Applications of Rash Measurement in Science Education. Springer Nature.
In preparation for a discussion on Quantifauxcation, a reading of 'Problem-Centering vs. Means-Centering in Science' by Abraham H. Maslow (1946).
Maslow, A. H. (1946). Problem-Centering vs. Means-Centering in Science. Philosophy of Science, 13(4), 326–331. https://doi.org/10.1086/286907