A Social Psychological Model of Scientific Practices: Explaining Research Practices and Outlining the Potential for Successful Reforms

Lee Jussim; Jon A. Krosnick; Sean T. Stevens; Stephanie M. Anglin

doi:10.5334/pb.496

Abstract

A crescendo of incidents have raised concerns about whether scientific practices in psychology may be suboptimal, sometimes leading to the publication, dissemination, and application of unreliable or misinterpreted findings. Psychology has been a leader in identifying possibly suboptimal practices and proposing reforms that might enhance the efficiency of the scientific process and the publication of robust evidence and interpretations. To help shape future efforts, this paper offers a model of the psychological and socio-structural forces and processes that may influence scientists’ practices. The model identifies practices targeted by interventions and reforms, and which practices remain unaddressed. The model also suggests directions for empirical research to assess how best to enhance the effectiveness of psychological inquiry.

References

Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of Personality and Social Psychology, 71(2), 230–244. DOI: 10.1037/0022-3514.71.2.230
Open DOI Search in Google Scholar Back to article
Baumeister, R. F. (2016). Charting the future of social psychology on stormy seas: Winners, losers, and recommendations. Journal of Experimental Social Psychology, 66, 153–158. DOI: 10.1016/j.jesp.2016.02.003
Open DOI Search in Google Scholar Back to article
Bem, D. J. (2011). Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407–425. DOI: 10.1037/a0021524
Open DOI Search in Google Scholar Back to article
Blanton, H., & Ikizer, E. G. (2018). The Bullet-point Bias: How Diluted science communications can impede social progress. In Crawford, J. T., & Jussim, L. (Eds.), The politics of social psychology (pp. 168–190). New York: Psychology Press. DOI: 10.4324/9781315112619-11
Open DOI Search in Google Scholar Back to article
Blaser, M. J. (1996). The Bacteria behind Ulcers. Scientific American, 274(2), 104–107. DOI: 10.1038/scientificamerican0296-104
Open DOI Search in Google Scholar Back to article
Brunner, J., & Schimmack, U. (2017). How replicable is psychology? A comparison of four methods of estimating replicability on the basis of test statistics in original studies. Unpublished manuscript.
Search in Google Scholar Back to article
Chambers, C. (2017). The Seven Deadly Sins of Psychology: A Manifesto for Reforming the Culture of Scientific Practice. Princeton University Press. DOI: 10.1515/9781400884940
Open DOI Search in Google Scholar Back to article
Chopik, W. J., Bremner, R. H., Defever, A. M., & Keller, V. N. (2018). How (and Whether) to Teach Undergraduates About the Replication Crisis in Psychological Science. Teaching of Psychology, 45(2), 158–163. DOI: 10.1177/0098628318762900
Open DOI Search in Google Scholar Back to article
Cialdini, R. B., & Goldstein, N. (2004). Social influence: Compliance and conformity. Annual Review of Psychology, 55, 591–621. DOI: 10.1146/annurev.psych.55.090902.142015
Open DOI Search in Google Scholar Back to article
Corker, K. (2019). Strengths and weaknesses of meta-analysis. Unpublished manuscript. DOI: 10.31234/osf.io/6gcnm
Open DOI Search in Google Scholar Back to article
Edwards, M. A., & Roy, S. (2017). Academic research in the 21^st century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition. Environmental Engineering Science, 34, 51–61. DOI: 10.1089/ees.2016.0223
Open DOI Search in Google Scholar Back to article
Fiedler, K. (2017). What constitutes strong psychological theory? The (neglected) role of diagnosticity and a priori theorizing. Perspectives on Psychological Science, 12, 46–61. DOI: 10.1177/1745691616654458
Open DOI Search in Google Scholar Back to article
Fiedler, K., Messner, C., & Bluemke, M. (2006). Unresolved problems with the “I”, the “A”, and the “T”: A logical and psychometric critique of the Implicit Association Test (IAT). European Review of Social Psychology, 17(1), 74–147. DOI: 10.1080/10463280600681248
Open DOI Search in Google Scholar Back to article
Finnigan, K. M., & Corker, K. S. (2016). Do performance avoidance goals moderate the effect of different types of stereotype threat on women’s math performance? Journal of Research in Personality, 63, 36–43. DOI: 10.1016/j.jrp.2016.05.009
Open DOI Search in Google Scholar Back to article
Flore, P. C., Mulder, J., & Wicherts, J. M. (2019). The influence of gender stereotype threat on mathematics test scores of Dutch high school students: a registered report. Comprehensive Results in Social Psychology, 1–35. DOI: 10.1080/23743603.2018.1559647
Open DOI Search in Google Scholar Back to article
Fraley, R. C., & Vazire, S. (2014). The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical Power. PLoS ONE 9(10): e109019. DOI: 10.1371/journal.pone.0109019
Open DOI Search in Google Scholar Back to article
Funder, D. C., Levin, J. M., Mackie, D. M., Morf, C. C., Sansone, C., Vazire, S., & West, S. G. (2014). Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice. Personality and Social Psychology Review, 18, 3–12. DOI: 10.1177/1088868313507536
Open DOI Search in Google Scholar Back to article
Gelman, A. (2016). What’s happened here is the winds have changed. Retrieved on 3/15/18 from: http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/.
Search in Google Scholar Back to article
Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science.” Science, 351, 1037-a. DOI: 10.1126/science.aad7243
Open DOI Search in Google Scholar Back to article
Giner-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562–571. DOI: 10.1177/1745691612457576
Open DOI Search in Google Scholar Back to article
Gould, S. J. (1981). Evolution as fact and theory. Retrieved on 2/1/18 from: http://www.stephenjaygould.org/ctrl/gould_fact-and-theory.html.
Search in Google Scholar Back to article
Hanselman, P., Rozek, C. S., Grigg, J., & Borman, G. D. (2017). New Evidence on Self-affirmation Effects and Theorized Sources of Heterogeneity from Large-Scale Replications. Journal of Educational Psychology, 109(3), 405–424. DOI: 10.1037/edu0000141
Open DOI Search in Google Scholar Back to article
Hunter, J. (2012). Post-publication peer review: opening up scientific conversation. Frontiers in Computational Neuroscience, 6, 1–2. DOI: 10.3389/fncom.2012.00063
Open DOI Search in Google Scholar Back to article
Inbar, Y., & Lammers, J. (2012). Political diversity in social and personality psychology. Perspectives on Psychological Science, 7, 496–503. DOI: 10.1177/1745691612448792
Open DOI Search in Google Scholar Back to article
Ioannidis, J. P. (2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7(6), 645–654. DOI: 10.1177/1745691612464056
Open DOI Search in Google Scholar Back to article
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 25, 524–532. DOI: 10.1177/0956797611430953
Open DOI Search in Google Scholar Back to article
Jussim, L. (2012). Social perception and social reality: Why accuracy dominates bias and self-fulfilling prophecy. New York: Oxford University Press. DOI: 10.1093/acprof:oso/9780195366600.001.0001
Open DOI Search in Google Scholar Back to article
Jussim, L., Crawford, J. T., Anglin, S. M., Stevens, S. M., & Duarte, J. L. (2016). Interpretations and methods: Towards a more effectively self-correcting social psychology. Journal of Experimental Social Psychology, 66, 116–133. DOI: 10.1016/j.jesp.2015.10.003
Open DOI Search in Google Scholar Back to article
Kahan, D. M., Peters, E., Wittlin, M., Slovic, P., Ouelette, L. L., Braman, D., & Mandel, G. (2012). The polarizing impact of science literacy and numeracy on perceived climate change risks. Nature: Climate Change, 2, 732–735. DOI: 10.1038/nclimate1547
Open DOI Search in Google Scholar Back to article
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217. DOI: 10.1207/s15327957pspr0203_4
Open DOI Search in Google Scholar Back to article
Kidwell, M. C., et al. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PloS Biology, 1–5. DOI: 10.1371/journal.pbio.1002456
Open DOI Search in Google Scholar Back to article
Lane, K. A., Banaji, M. R., Nosek, B. A., & Greenwald, A. G. (2007). Understanding and using the implicit association test: IV. What we know (so far) about the Method. In Wittenbrink, B., & Schwarz, N. (Eds.), Implicit Methods (pp. 59–101). New York: Guilford Press.
Search in Google Scholar Back to article
Lerner, J. S., & Tetlock, P. E. (1999). Accounting for the effects of accountability. Psychological Bulletin, 125, 255–275. DOI: 10.1037/0033-2909.125.2.255
Open DOI Search in Google Scholar Back to article
Lilienfeld, S. O. (2017). Microaggressions: Strong Claims, Inadequate Evidence. Perspectives on Psychological Science, 12(1), 138–169. DOI: 10.1177/1745691616659391
Open DOI Search in Google Scholar Back to article
Loeb, A. (2014). The benefits of diversity. Nature: Physics, 10, 16–17. DOI: 10.1038/nphys3089
Open DOI Search in Google Scholar Back to article
Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. N. W. Storer (Ed.). Chicago, IL: The University of Chicago Press.
Search in Google Scholar Back to article
Mitchell, G. (2018). Jumping to conclusions: Advocacy and application of psychological research. In Crawford, J. T., & Jussim, L. (Eds.), The politics of social psychology. New York: Psychology Press. DOI: 10.1093/oso/9780190687328.003.0016
Open DOI Search in Google Scholar Back to article
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behavior, 1. DOI: 10.1038/s41562-016-0021
Open DOI Search in Google Scholar Back to article
Nelson, L., Simmons, J., & Simonsohn, U. (2018). Psychology’s renaissance. Annual Review of Psychology, 69, 511–534. DOI: 10.1146/annurev-psych-122216-011836
Open DOI Search in Google Scholar Back to article
Nosek, B., Spies, J. R., & Motyl, M. (2012). Scientific Utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631. DOI: 10.1177/1745691612459058
Open DOI Search in Google Scholar Back to article
Nuitjen, M. B., Borghuis, J., Veldkamp, C. L. S., Dominguez-Alvarez, L., van Assen, M. A. L. M., & Wicherts, J. M. (2017). Journal data sharing polices and statistical reporting inconsistencies in psychology. Collabra, 3, 1–22. DOI: 10.1525/collabra.102
Open DOI Search in Google Scholar Back to article
Odutayo, A., Emdin, C. A., Hsiao, A. J., Shakir, M., Copsey, B., Dutton, S., Hopewell, S., et al. (2017). Association between trial registration and positive study findings: cross sectional study (Epidemiological Study of Randomized Trials—ESORT). BMJ, 356, j917. DOI: 10.1136/bmj.j917
Open DOI Search in Google Scholar Back to article
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 943. DOI: 10.1126/science.aac4716
Open DOI Search in Google Scholar Back to article
Popper, K. (1959). The logic of scientific discovery. New York, NY: Routledge. DOI: 10.1063/1.3060577
Open DOI Search in Google Scholar Back to article
Roberts, B. (2019). Its déjà vu all over again. Retrieved on March 17, 2019 from: https://pigee.wordpress.com/2019/03/11/its-deja-vu-all-over-again/.
Search in Google Scholar Back to article
Schimmack, U. (2014). Quantifying statistical research integrity: The Replicability-Index. Unpublished manuscript.
Search in Google Scholar Back to article
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. DOI: 10.1177/0956797611417632
Open DOI Search in Google Scholar Back to article
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2012). A 21 word solution. Dialogues. DOI: 10.2139/ssrn.2160588
Open DOI Search in Google Scholar Back to article
Simmons, J. P., & Simonsohn, U. (2017). Power Posing: P-Curving the Evidence. Psychological Science, 28(5), 687–693. DOI: 10.1177/0956797616658563
Open DOI Search in Google Scholar Back to article
Spellman, B. A. (2015). A Short (Personal) Future History of Revolution 2.0. Perspectives On Psychological Science, 10, 886–899. DOI: 10.1177/1745691615609918
Open DOI Search in Google Scholar Back to article
Sumner, P., Vivian-Griffiths, S., Boivin, J., Williams, A., Venetis, C. A., Davies, A., Chambers, C. D., et al. (2014). The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ, 349, g7015. DOI: 10.1136/bmj.g7015
Open DOI Search in Google Scholar Back to article
Sumner, P., Vivian-Griffiths, S., Bolvin, J., Williams, A., Bott, L., Adams, R., Chambers, C. D., et al. (2016). Exaggerations and caveats in press releases and health-related science news. PloS One, 11. e0168217. DOI: 10.1371/journal.pone.0168217
Open DOI Search in Google Scholar Back to article
van Elk, M., Matzke, D., Gronau, Q. F., Guan, M., Vandekerckhove, J., & Wagenmakers, E. J. (2015). Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming. Frontiers in Psychology, 6. DOI: 10.3389/fpsyg.2015.01365
Open DOI Search in Google Scholar Back to article
Vazire, S. (2017). Quality uncertainty erodes trust in science. Collabra: Psychology, 3. DOI: 10.1525/collabra.74
Open DOI Search in Google Scholar Back to article
Vosgerau, J., Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2018). Internal Meta-Analysis Makes False-Positives Easier To Produce and Harder To Correct (SSRN Scholarly Paper No. ID 3271372). Rochester, NY: Social Science Research Network. Retrieved from https://papers.ssrn.com/abstract=3271372. DOI: 10.2139/ssrn.3271372
Open DOI Search in Google Scholar Back to article
Wagenmakers, E. J., & Gronau, Q. (2017). Two pitfalls of pre-registration: The case of ego depletion. Retrieved on 3/16/18 from https://www.bayesianspectacles.org/two-pitfalls-of-preregistration-the-case-of-ego-depletion/.
Search in Google Scholar Back to article
Warren, M. (2018). First analysis of ‘pre-registered’ studies shows sharp rise in null findings. Nature News, October 24. DOI: 10.1038/d41586-018-07118-1
Open DOI Search in Google Scholar Back to article
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS One, 6, e26828. DOI: 10.1371/journal.pone.0026828
Open DOI Search in Google Scholar Back to article
Yap, S. C. Y., Wortman, J., Anusic, I., Baker, S. G., Scherer, L. D., Donnellan, M. B., & Lucas, R. E. (2017). The effect of mood on judgments of subjective well-being: Nine tests of the judgment model. Journal of Personality and Social Psychology, 113(6), 939–961. DOI: 10.1037/pspp0000115
Open DOI Search in Google Scholar Back to article

A Social Psychological Model of Scientific Practices: Explaining Research Practices and Outlining the Potential for Successful Reforms

Abstract

Paradigm

My account