References
- 1Albers, C., & Lakens, D. (2018). When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias. Journal of Experimental Social Psychology, 74, 187–195. DOI: 10.1016/j.jesp.2017.09.004
- 2Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017). Sample-size planning for more accurate statistical power: A method adjusting sample effect sizes for publication bias and uncertainty. Psychological Science, 28(11), 1547–1562. DOI: 10.1177/0956797617723724
- 3Baayen, R. H. (2008). Analyzing Linguistic Data: A practical introduction to statistics using R. Cambridge, UK: Cambridge University Press. DOI: 10.1017/CBO9780511801686
- 4Bakker, M., Hartgerink, C. H., Wicherts, J. M., & van der Maas, H. L. (2016). Researchers’ intuitions about power in psychological research. Psychological Science, 27(8), 1069–1077. DOI: 10.1177/0956797616647519
- 5Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E. J., Berk, R., Johnson, V. E., et al. (2018). Redefine statistical significance. Nature Human Behaviour, 2(1), 6–10. DOI: 10.1038/s41562-017-0189-z
- 6Birnbaum, M. H. (2004). Human research and data collection via the Internet. Annual Review of Psychology, 55, 803–832. DOI: 10.1146/annurev.psych.55.090902.141601
- 7Bishop, D. V. M. (2013, June 7). Interpreting unexpected significant results [Blog post]. Retrieved from
http://deevybee.blogspot.com/2013/06/interpreting-unexpected-significant.html - 8Bishop, D. V. M. (2018, July 12). One big study or two small studies? Insights from simulations. Retrieved from
http://deevybee.blogspot.com/2018/07/one-big-study-or-two-small-studies.html - 9Borsboom, D. (2013, November 20). Theoretical amnesia [Blog post]. Retrieved from
http://osc.centerforopenscience.org/2013/11/20/theoretical-amnesia/ - 10Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100(2), 431–449. DOI: 10.1037/a0038047
- 11Brooks, G. P., & Barcikowski, R. S. (2012). The PEAR method for sample sizes in multiple linear regression. Multiple Linear Regression Viewpoints, 38(2), 1–16.
- 12Brysbaert, M., & Stevens, M. (2018). Power Analysis and Effect Size in Mixed Effects Models: A Tutorial. Journal of Cognition, 1(9), 1–20. DOI: 10.5334/joc.10
- 13Callens, M., Tops, W., & Brysbaert, M. (2012). Cognitive Profile of Students Who Enter Higher Education with an Indication of Dyslexia. PLoS ONE, 7(6),
e38081 . DOI: 10.1371/journal.pone.0038081 - 14Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., Wu, H., et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637–644. DOI: 10.1038/s41562-018-0399-z
- 15Campbell, J. I., & Thompson, V. A. (2012). MorePower 6.0 for ANOVA with relational confidence intervals and Bayesian analysis. Behavior Research Methods, 44(4), 1255–1265. DOI: 10.3758/s13428-012-0186-0
- 16Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49, 609–610. DOI: 10.1016/j.cortex.2012.12.016
- 17Cohen, J. (1962). The statistical power of abnormal-social psychological research: a review. The Journal of Abnormal and Social Psychology, 65(3), 145–153. DOI: 10.1037/h0045186
- 18Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
- 19Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. DOI: 10.1037/0033-2909.112.1.155
- 20Colquhoun, D. (2014). An investigation of the false discovery rate and the misinterpretation of p-values. Royal Society Open Science, 1(3),
140216 . DOI: 10.1098/rsos.140216 - 21Cronbach, L. J. (1957). The two disciplines of scientific psychology. American Psychologist, 12(11), 671–684. DOI: 10.1037/h0043943
- 22Cumming, G. (2014). The new statistics why and how. Psychological Science, 25(1), 7–29. DOI: 10.1177/0956797613504966
- 23De Deyne, S., Navarro, D. J., Perfors, A., Brysbaert, M., & Storms, G. (2019). The “Small World of Words” English word association norms for over 12,000 cue words. Behavior Research Methods. Preprint available at
https://link.springer.com/article/10.3758/s13428-018-1115-7 . DOI: 10.3758/s13428-018-1115-7 - 24de Jong, T., Marsman, M., & Wagenmakers, E. J. (2019). A Bayesian Approach to the Correction for Multiplicity. Preprint. DOI: 10.31234/osf.io/s56mk
- 25Depaoli, S., & van de Schoot, R. (2017). Improving transparency and replication in Bayesian statistics: The WAMBS-Checklist. Psychological Methods, 22(2), 240–261. DOI: 10.1037/met0000065
- 26Dienes, Z. (2016). How Bayes factors change scientific practice. Journal of Mathematical Psychology, 72, 78–89. DOI: 10.1016/j.jmp.2015.10.003
- 27Dimitrov, D. M., & Rumrill, P. D.,
Jr , (2003). Pretest-posttest designs and measurement of change. Work, 20(2), 159–165. - 28Dumas-Mallet, E., Button, K. S., Boraud, T., Gonon, F., & Munafò, M. R. (2017). Low statistical power in biomedical science: a review of three human research domains. Royal Society Open Science, 4(2), 160254. DOI: 10.1098/rsos.160254
- 29Duval, S., & Tweedie, R. (2000). Trim and fill: a simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. DOI: 10.1111/j.0006-341X.2000.00455.x
- 30Edwards, M. A., & Roy, S. (2017). Academic research in the 21st century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition. Environmental Engineering Science, 34(1), 51–61. DOI: 10.1089/ees.2016.0223
- 31Egbewale, B. E., Lewis, M., & Sim, J. (2014). Bias, precision and statistical power of analysis of covariance in the analysis of randomized trials with baseline imbalance: a simulation study. BMC Medical Research Methodology, 14(1), 49. DOI: 10.1186/1471-2288-14-49
- 32Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. DOI: 10.1136/bmj.315.7109.629
- 33Etz, A., & Vandekerckhove, J. (2018). Introduction to Bayesian inference for psychology. Psychonomic Bulletin & Review, 25(1), 5–34. DOI: 10.3758/s13423-017-1262-3
- 34Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. DOI: 10.3758/BF03193146
- 35Fletcher, T. D. (2015). Package ‘psychometric’. Available at
https://cran.r-project.org/web/packages/psychometric/psychometric.pdf - 36Fraley, R. C., & Vazire, S. (2014). The N-pact factor: Evaluating the quality of empirical journals with respect to sample size and statistical power. PloS one, 9(10),
e109019 . DOI: 10.1371/journal.pone.0109019 - 37Francis, G. (2012). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19(6), 975–991. DOI: 10.3758/s13423-012-0322-y
- 38Fritz, C. O., Morris, P. E., & Richler, J. J. (2012). Effect size estimates: current use, calculations, and interpretation. Journal of Experimental Psychology: General, 141(1), 2–18. DOI: 10.1037/a0024338
- 39Garcia-Marques, L., Garcia-Marques, T., & Brauer, M. (2014). Buy three but get only two: The smallest effect in a 2 × 2 ANOVA is always uninterpretable. Psychonomic Bulletin & Review, 21(6), 1415–1430. DOI: 10.3758/s13423-014-0640-3
- 40Gigerenzer, G., & Marewski, J. N. (2015). Surrogate science: The idol of a universal method for scientific inference. Journal of Management, 41(2), 421–440. DOI: 10.1177/0149206314547522
- 41Gignac, G. E., & Szodorai, E. T. (2016). Effect size guidelines for individual differences researchers. Personality and Individual Differences, 102, 74–78. DOI: 10.1016/j.paid.2016.06.069
- 42Giner-Sorolla, R. (2018, January 24). Powering your interaction [Blog post]. Retrieved from
https://approachingblog.wordpress.com/2018/01/24/powering-your-interaction-2/ - 43Gosling, S. D., & Mason, W. (2015). Internet research in psychology. Annual Review of Psychology, 66, 877–902. DOI: 10.1146/annurev-psych-010814-015321
- 44Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59(2), 93-104. DOI: 10.1037/0003-066X.59.2.93
- 45Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400–407. DOI: 10.3758/s13428-015-0578-z
- 46Higginson, A. D., & Munafò, M. R. (2016). Current incentives for scientists lead to underpowered studies with erroneous conclusions. PLoS Biology, 14(11),
e2000995 . DOI: 10.1371/journal.pbio.2000995 - 47Hilbig, B. E. (2016). Reaction time effects in lab-versus Web-based research: Experimental evidence. Behavior Research Methods, 48(4), 1718–1724. DOI: 10.3758/s13428-015-0678-9
- 48Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: the pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19–24. DOI: 10.1198/000313001300339897
- 49John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. DOI: 10.1177/0956797611430953
- 50Johnson, V. E., Payne, R. D., Wang, T., Asher, A., & Mandal, S. (2017). On the reproducibility of psychological science. Journal of the American Statistical Association, 112(517), 1–10. DOI: 10.1080/01621459.2016.1240079
- 51Kelley, K., Maxwell, S. E., & Rausch, J. R. (2003). Obtaining power or obtaining precision: Delineating methods of sample-size planning. Evaluation & the Health Professions, 26(3), 258–287. DOI: 10.1177/0163278703255242
- 52Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L. S., Nosek, B., et al. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14(5),
e1002456 . DOI: 10.1371/journal.pbio.1002456 - 53Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B.,
Jr. , Alper, S., Batra, R., et al. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. DOI: 10.1177/2515245918810225 - 54Knofczynski, G. T., & Mundfrom, D. (2008). Sample sizes when using multiple linear regression for prediction. Educational and Psychological Measurement, 68(3), 431–442. DOI: 10.1177/0013164407310131
- 55Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006). Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of General Psychiatry, 63(5), 484–489. DOI: 10.1001/archpsyc.63.5.484
- 56Kruschke, J. K., & Liddell, T. M. (2018). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review, 25(1), 178–206. DOI: 10.3758/s13423-016-1221-4
- 57Kühberger, A., Fritz, A., & Scherndl, T. (2014). Publication bias in psychology: a diagnosis based on the correlation between effect size and sample size. PloS One, 9(9),
e105825 . DOI: 10.1371/journal.pone.0105825 - 58Lachin, J. M. (1981). Introduction to sample size determination and power analysis for clinical trials. Controlled Clinical Trials, 2(2), 93–113. DOI: 10.1016/0197-2456(81)90001-5
- 59Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. DOI: 10.3389/fpsyg.2013.00863
- 60Lakens, D. (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44, 701–710. DOI: 10.1002/ejsp.2023
- 61Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M. A., Argamon, S. E., Zwaan, R. A., et al. (2018). Justify your alpha. Nature Human Behaviour, 2(3), 168–171. DOI: 10.1038/s41562-018-0311-x
- 62Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259–269. DOI: 10.1177/2515245918770963
- 63LeBel, E. P., McCarthy, R., Earp, B. D., Elson, M., & Vanpaemel, W. (in press). A unified framework to quantify the credibility of scientific findings. Advances in Methods and Practices in Psychological Science. Advance publication. DOI: 10.1177/2515245918787489
- 64Lenth, R. V. (2006). Java Applets for Power and Sample Size [Computer software]. Retrieved November, 9, 2018 from
http://www.stat.uiowa.edu/~rlenth/Power - 65Lindsay, D. S. (2015). Replication in psychological science. Psychological Science, 26(12), 1827–1832. DOI: 10.1177/0956797615616374
- 66Lindsay, D. S. (2017). Sharing data and materials in psychological science. Psychological Science, 28(6), 699–702. DOI: 10.1177/0956797617704015
- 67Litman, L., Robinson, J., & Abberbock, T. (2017). TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods, 49(2), 433–442. DOI: 10.3758/s13428-016-0727-z
- 68Loiselle, D., & Ramchandra, R. (2015). A counterview of ‘An investigation of the false discovery rate and the misinterpretation of p-values’ by Colquhoun (2014). Royal Society Open Science, 2(8),
150217 . DOI: 10.1098/rsos.150217 - 69Maxwell, S. E. (2000). Sample size and multiple regression analysis. Psychological Methods, 5(4), 434. DOI: 10.1037/1082-989X.5.4.434
- 70Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychological Methods, 9(2), 147–163. DOI: 10.1037/1082-989X.9.2.147
- 71Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E. J. (2016). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 23(1), 103–123. DOI: 10.3758/s13423-015-0947-8
- 72Morey, R. D., Rouder, J. N., Jamil, T., Urbanek, S., Forner, K., & Ly, A. (2018). Package ‘BayesFactor’. Available at
https://cran.r-project.org/web/packages/BayesFactor/BayesFactor.pdf - 73Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated-measures and independent-groups designs. Psychological Methods, 7(1), 105–125. DOI: 10.1037/1082-989X.7.1.105
- 74Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Du Sert, N. P., Ioannidis, J. P., et al. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 0021. DOI: 10.1038/s41562-016-0021
- 75Murphy, K. R., Myors, B., & Wolach, A. (2014). Statistical power analysis: A simple and general model for traditional and modern hypothesis tests. Routledge. DOI: 10.4324/9781315773155
- 76Nosek, B. A., & Lakens, D. (2014). Registered reports. Social Psychology, 45, 137–141. DOI: 10.1027/1864-9335/a000192
- 77O’Connell, N. S., Dai, L., Jiang, Y., Speiser, J. L., & Ward, R. (2017). Methods for Analysis of Pre-Post Data in Clinical Research: A Comparison of Five Common Methods. Journal of Biometrics & Biostatistics, 8(1). DOI: 10.4172/2155-6180.1000334
- 78Onghena, P., Michiels, B., Jamshidi, L., Moeyaert, M., & Van den Noortgate, W. (2018). One by one: Accumulating evidence by using meta-analytical procedures for single-case experiments. Brain Impairment, 19(1), 33–58. DOI: 10.1017/BrImp.2017.25
- 79Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251):
aac4716 . DOI: 10.1126/science.aac4716 - 80Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188. DOI: 10.1177/0963721414531598
- 81Perugini, M., Gallucci, M., & Costantini, G. (2018). A Practical Primer to Power Analysis for Simple Experimental Designs. International Review of Social Psychology, 31(1): 20. DOI: 10.5334/irsp.181
- 82Pollatsek, A., & Well, A. D. (1995). On the use of counterbalanced designs in cognitive research: A suggestion for a better and more powerful analysis. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(3), 785–794. DOI: 10.1037//0278-7393.21.3.785
- 83Pyc, M. A., & Rawson, K. A. (2010). Why testing improves memory: Mediator effectiveness hypothesis. Science, 330(6002), 335–335. DOI: 10.1126/science.1191465
- 84Rouder, J. N., & Haaf, J. M. (2018). Power, dominance, and constraint: A note on the appeal of different design traditions. Advances in Methods and Practices in Psychological Science, 1(1), 19–26. Preprint. DOI: 10.1177/2515245917745058
- 85Rouder, J. N., Morey, R. D., Verhagen, J., Swagman, A. R., & Wagenmakers, E. J. (2017). Bayesian analysis of factorial designs. Psychological Methods, 22(2), 304. DOI: 10.1037/met0000057
- 86Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225–237. DOI: 10.3758/PBR.16.2.225
- 87Schimmack, U. (2015, September 5). The Replicability of Cognitive Psychology in the OSF-Reproducibility-Project [Blog post]. Retrieved from
https://replicationindex.wordpress.com/2015/09/05/comparison-of-php-curve-predictions-and-outcomes-in-the-osf-reproducibility-project-part-2-cognitive-psychology/ - 88Schönbrodt, F. D., Wagenmakers, E. J., Zehetleitner, M., & Perugini, M. (2017). Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences. Psychological Methods, 22(2), 322–339. DOI: 10.1037/met0000061
- 89Shadish, W. R., Hedges, L. V., & Pustejovsky, J. E. (2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52(2), 123–147. DOI: 10.1016/j.jsp.2013.11.005
- 90Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. DOI: 10.1037/0033-2909.86.2.420
- 91Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. DOI: 10.1177/0956797611417632
- 92Simonsohn, U. (2014, March 12). No-way interactions [Blog post]. Retrieved from
http://datacolada.org/17 . DOI: 10.15200/winn.142559.90552 - 93Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society Open Science, 3(9), 160384. DOI: 10.1098/rsos.160384
- 94Smith, J. D. (2012). Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods, 17(4), 510. DOI: 10.1037/a0029312
- 95Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2018). What meta-analyses reveal about the replicability of psychological research. Psychological Bulletin, 144(12), 1325–1346. DOI: 10.1037/bul0000169
- 96Stevens, M., & Brysbaert, M. (2016). A simple solution for missing observations based on random effects models. Unpublished manuscript available at
http://crr.ugent.be/members/marc-brysbaert . - 97Szucs, D., & Ioannidis, J. P. (2017a). Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biology, 15(3),
e2000797 . DOI: 10.1371/journal.pbio.2000797 - 98Szucs, D., & Ioannidis, J. (2017b). When null hypothesis significance testing is unsuitable for research: A reassessment. Frontiers in Human Neuroscience, 11, 390. DOI: 10.3389/fnhum.2017.00390
- 99Tomczak, M., Tomczak, E., Kleka, P., & Lew, R. (2014). Using power analysis to estimate appropriate sample size. Trends in Sport Sciences, 21(4), 195–206. DOI: 10.1007/978-3-8349-3752-0_5
- 100Trafimow, D., & Myüz, H. A. (in press). The sampling precision of research in five major areas of psychology. Behavior Research Methods. Preprint available at
https://link.springer.com/article/10.3758%2Fs13428-018-1173-x - 101Vankov, I., Bowers, J., & Munafò, M. R. (2014). Article Commentary: On the Persistence of Low Power in Psychological Science. Quarterly Journal of Experimental Psychology, 67(5), 1037–1040. DOI: 10.1080/17470218.2014.885986
- 102Vasilev, M. R., Kirkby, J. A., & Angele, B. (2018). Auditory distraction during reading: A Bayesian meta-analysis of a continuing controversy. Perspectives on Psychological Science, 13(5), 567–597. DOI: 10.1177/1745691617747398
- 103Verhaeghen, P. (2003). Aging and vocabulary score: A meta-analysis. Psychology and aging, 18(2), 332–339. DOI: 10.1037/0882-7974.18.2.332
- 104Wagenmakers, E. J., Love, J., Marsman, M., Jamil, T., Ly, A., Morey, R. D., et al. (2018). Bayesian inference for psychology, Part II: Example applications with JASP. Psychonomic Bulletin and Review, 25(1), 58–76. DOI: 10.3758/s13423-017-1323-7
- 105Wilkinson, L. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54(8), 594–604. DOI: 10.1037/0003-066X.54.8.594
- 106Wilson, T. D., Reinhard, D. A., Westgate, E. C., Gilbert, D. T., Ellerbeck, N., Hahn, C., Shaked, A., et al. (2014). Just think: The challenges of the disengaged mind. Science, 345>(6192), 75–77. DOI: 10.1126/science.1250830
- 107Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K., & Zeelenberg, R. (2018). Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin & Review, 25(5), 1968–1972. DOI: 10.3758/s13423-017-1348-y