Have a personal or library account? Click to login
How to Run Behavioural Experiments Online: Best Practice Suggestions for Cognitive Psychology and Neuroscience Cover

How to Run Behavioural Experiments Online: Best Practice Suggestions for Cognitive Psychology and Neuroscience

By: Nathan Gagné and  Léon Franzen  
Open Access
|Jan 2023

References

  1. Aiello, L. M., Barrat, A., Schifanella, R., Cattuto, C., Markines, B., & Menczer, F. (2012). Friendship prediction and homophily in social media. ACM Transactions on the Web, 6(2). DOI: 10.1145/2180861.2180866
  2. Albers, C., & Lakens, D. (2018). When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias. Journal of Experimental Social Psychology, 74, 187195. DOI: 10.1016/j.jesp.2017.09.004
  3. Allen, M., Poggiali, D., Whitaker, K., Marshall, T. R., & Kievit, R. (2019). Raincloud plots: a multi-platform tool for robust data visualization [version 1; peer review: 2 approved]. Wellcome Open Res, 4(63). DOI: 10.12688/wellcomeopenres.15191.1
  4. Altman, D. G., & Bland, J. M. (1999). Treatment allocation in controlled trials: why randomise? BMJ, 318(7192), 12091209. DOI: 10.1136/bmj.318.7192.1209
  5. Anwyl-Irvine, A. L., Dalmaijer, E. S., Hodges, N., & Evershed, J. K. (2021). Realistic precision and accuracy of online experiment platforms, web browsers, and devices. Behavior Research Methods, 53(4), 14071425. DOI: 10.3758/s13428-020-01501-5
  6. Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K. (2020). Gorilla in our midst: An online behavioral experiment builder. Behavior Research Methods, 52(1), 388407. DOI: 10.3758/s13428-019-01237-x
  7. Artner, R., Verliefde, T., Steegen, S., Gomes, S., Traets, F., Tuerlinckx, F., & Vanpaemel, W. (2020). The reproducibility of statistical results in psychological research: An investigation using unpublished raw data. Psychological Methods, 26(5), 527546. DOI: 10.1037/met0000365
  8. Baker, M., & Penny, D. (2016). Is there a reproducibility crisis in science? Nature, 452454. DOI: 10.1038/d41586-019-00067-3
  9. Bartels, C., Wegrzyn, M., Wiedl, A., Ackermann, V., & Ehrenreich, H. (2010). Practice effects in healthy adults: A longitudinal study on frequent repetitive cognitive testing. BMC Neuroscience, 11. DOI: 10.1186/1471-2202-11-118
  10. Benedict, R. H. B., & Zgaljardic, D. J. (1998). Practice effects during repeated administrations of memory tests with and without alternate forms. Journal of Clinical and Experimental Neuropsychology, 20(3), 339352. DOI: 10.1076/jcen.20.3.339.822
  11. Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s mechanical turk. Political Analysis, 20(3), 351368. DOI: 10.1093/pan/mpr057
  12. Beyer, R. M., Manica, A., & Mora, C. (2021). Shifts in global bat diversity suggest a possible role of climate change in the emergence of SARS-CoV-1 and SARS-CoV-2. Science of the Total Environment, 767, 145413. DOI: 10.1016/j.scitotenv.2021.145413
  13. Bieniek, M. M., Bennett, P. J., Sekuler, A. B., & Rousselet, G. A. (2016). A robust and representative lower bound on object processing speed in humans. European Journal of Neuroscience, 44(2), 18041814. DOI: 10.1111/ejn.13100
  14. Blythe, H. I., Kirkby, J. A., & Liversedge, S. P. (2018). Comments on: “What is developmental dyslexia?” brain sci. 2018, 8, 26. the relationship between eye movements and reading difficulties. Brain Sciences, 8(6). DOI: 10.3390/brainsci8060100
  15. Bridges, D., Pitiot, A., MacAskill, M. R., & Peirce, J. W. (2020). The timing mega-study: Comparing a range of experiment generators, both lab-based and online. PeerJ, 8(e9414). DOI: 10.7717/peerj.9414
  16. Brouillard, M., & Byers-Heinlein, K. (2019a). Recruiting hard-to-find participants using Facebook sponsored posts. DOI: 10.17605/OSF.IO/9BCKN
  17. Brouillard, M., & Byers-Heinlein, K. (2019b). Recruiting infant participants using Facebook sponsored posts. DOI: 10.17605/OSF.IO/9BCKN
  18. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365376. DOI: 10.1038/nrn3475
  19. Calamia, M., Markon, K., & Tranel, D. (2012). Scoring higher the second time around: Meta-analyses of practice effects in neuropsychological assessment. Clinical Neuropsychologist, 26(4), 543570. DOI: 10.1080/13854046.2012.680913
  20. Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen, Y., Forsell, E., Gampa, A., Heikensten, E., Hummer, L., Imai, T., … Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637644. DOI: 10.1038/s41562-018-0399-z
  21. Carter, W. L. (2021). ScreenScale. DOI: 10.17605/OSF.IO/8FHQK
  22. Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29(6), 21562160. DOI: 10.1016/j.chb.2013.05.009
  23. Chandler, J. J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112130. DOI: 10.3758/s13428-013-0365-7
  24. Chandler, J. J., & Paolacci, G. (2017). Lie for a Dime: When Most Prescreening Responses Are Honest but Most Study Participants Are Impostors. Social Psychological and Personality Science, 8(5), 500508. DOI: 10.1177/1948550617698203
  25. Chouinard, B., Scott, K., & Cusack, R. (2019). Using automatic face analysis to score infant behaviour from video collected online. Infant Behavior and Development, 54, 112. DOI: 10.1016/j.infbeh.2018.11.004
  26. Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS ONE, 8(3). DOI: 10.1371/journal.pone.0057410
  27. de Leeuw, J. R., & Motz, B. A. (2016). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. Behavior Research Methods, 48(1), 112. DOI: 10.3758/s13428-015-0567-2
  28. DeBruine, L. M., & Barr, D. J. (2021). Understanding Mixed-Effects Models Through Data Simulation. Advances in Methods and Practices in Psychological Science, 4(1). DOI: 10.1177/2515245920965119
  29. DePuy, V., & Berger, V. W. (2014). Counterbalancing. Wiley StatRef: Statistics Reference Online. DOI: 10.1002/9781118445112.stat06195
  30. Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010). Are your participants gaming the system? Screening mechanical Turk workers. Conference on Human Factors in Computing Systems – Proceedings, 4, 23992402. DOI: 10.1145/1753326.1753688
  31. Efron, B. (1979). Bootstrap methods: another look at the jackknife. Annals of Statistics, 7(1), 126. DOI: 10.1214/aos/1176344552
  32. Ettinger, U., Meyhöfer, I., Steffens, M., Wagner, M., & Koutsouleris, N. (2014). Genetics, Cognition, and Neurobiology of Schizotypal Personality: A Review of the Overlap with Schizophrenia. Frontiers in Psychiatry, 5, 116. DOI: 10.3389/fpsyt.2014.00018
  33. Etz, A., & Vandekerckhove, J. (2016). A Bayesian perspective on the reproducibility project: Psychology. PLoS ONE, 11(2), 112. DOI: 10.1371/journal.pone.0149794
  34. Faul, F., Erdefelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Journal of Materials and Environmental Science, 39(2), 175191. DOI: 10.3758/BF03193146
  35. Field, A. P., & Wilcox, R. R. (2017). Robust statistical methods: A primer for clinical psychology and experimental psychopathology researchers. Behaviour Research and Therapy, 98, 1938. DOI: 10.1016/j.brat.2017.05.013
  36. Forscher, P., Wagenmakers, E.-J., Coles, N. A., Silan, M. A., Dutra, N. B., Basnight-Brown, D., & IJzerman, H. (2021). A Manifesto for Big Team Science.
  37. Foster, E. D., & Deardorff, A. (2017). Open Science Framework (OSF). Journal of the Medical Library Association, 105(2). DOI: 10.5195/jmla.2017.88
  38. Franzen, L. (2018). Neural and visual correlates of perceptual decision making in adult dyslexia (Issue December) [University of Glasgow]. https://theses.gla.ac.uk/71950/
  39. Franzen, L., Delis, I., Sousa, G. De, Kayser, C., & Philiastides, M. G. (2020). Auditory information enhances post-sensory visual evidence during rapid multisensory decision-making. Nature Communications, 11, 5440. DOI: 10.1038/s41467-020-19306-7
  40. Franzen, L., Gagné, N., Johnson, A. P., & Grohmann, B. (2022). Behavioral markers of visuo-spatial working memory load in adult dyslexia. Open Science Framework. DOI: 10.17605/OSF.IO/DYN5T
  41. Franzen, L., Stark, Z., & Johnson, A. P. (2021). Individuals with dyslexia use a different visual sampling strategy to read text. Scientific Reports, 11, 6449. DOI: 10.1038/s41598-021-84945-9
  42. Gallant, J., & Libben, G. (2019). No lab, no problem: Designing lexical comprehension and production experiments using PsychoPy3. The Mental Lexicon, 14(1), 152168. DOI: 10.1075/ml.00002.gal
  43. GazeRecorder. (2021). Gaze flow. https://gazerecorder.com/gazeflow/
  44. Giakoumaki, S. G. (2012). Cognitive and prepulse inhibition deficits in psychometrically high schizotypal subjects in the general population: Relevance to schizophrenia research. Journal of the International Neuropsychological Society, 18(4), 643656. DOI: 10.1017/S135561771200029X
  45. Gillan, C. M., & Daw, N. D. (2016). Taking Psychiatry Research Online. Neuron, 91(1), 1923. DOI: 10.1016/j.neuron.2016.06.002
  46. Gilmore, R. O., Kennedy, J. L., & Adolph, K. E. (2018). Practical Solutions for Sharing Data and Materials From Psychological Research. Advances in Methods and Practices in Psychological Science, 1(1), 121130. DOI: 10.1177/2515245917746500
  47. Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples. Journal of Behavioral Decision Making, 26(3), 213224. DOI: 10.1002/bdm.1753
  48. Gorilla Team. (2021). Gorilla Screen Calibration. https://support.gorilla.sc/support/reference/task-builder-zones#eyetracking
  49. Grootswagers, T. (2020). A primer on running human behavioural experiments online. Behavior Research Methods, 52(6), 22832286. DOI: 10.3758/s13428-020-01395-3
  50. Hauser, D. J., & Schwarz, N. (2016). Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48(1), 400407. DOI: 10.3758/s13428-015-0578-z
  51. Henderson, P. W., & Cote, J. A. (1998). Guidelines for selecting or modifying logos. Journal of Marketing, 62(2), 1430. DOI: 10.1177/002224299806200202
  52. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 6183. DOI: 10.1017/S0140525X0999152X
  53. Holcombe, A. (2020). The reproducibility crisis. https://osf.io/r4wpt
  54. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. DOI: 10.1371/journal.pmed.0020124
  55. Ioannidis, J. P. A. (2008). Why most discovered true associations are inflated. Epidemiology, 19(5), 640648. DOI: 10.1097/EDE.0b013e31818131e7
  56. Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: Detection, prevalence, and prevention. Trends in Cognitive Sciences, 18(5), 235241. DOI: 10.1016/j.tics.2014.02.010
  57. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science, 23(5), 524532. DOI: 10.1177/0956797611430953
  58. Jun, E., Hsieh, G., & Reinecke, K. (2017). Types of motivation affect study selection, attention, and dropouts in online experiments. Proceedings of the ACM on Human-Computer Interaction, 1(CSCW), 115. DOI: 10.1145/3134691
  59. Kingdom, F., & Prins, N. (2016). Psychophysics (2nd ed.). Academic Press.
  60. Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., Jzerman, H. I., Nilsonne, G., Vanpaemel, W., & Frank, M. C. (2018). A practical guide for transparency in psychological science. Collabra: Psychology, 4(1), 115. DOI: 10.1525/collabra.158
  61. Knoeferle, K. M., Woods, A., Käppler, F., & Spence, C. (2010). That Sounds Sweet: Using Cross-Modal Correspondences to Communicate Gustatory Attributes. Psychology & Marketing, 30(6), 461469. DOI: 10.1002/mar
  62. Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A., & Poeppel, D. (2017). Neuroscience Needs Behavior: Correcting a Reductionist Bias. Neuron, 93(3), 480490. DOI: 10.1016/j.neuron.2016.12.041
  63. Kruschke, J. K. (2014). Doing Bayesian Data Analysis. Academic Press.
  64. Kruschke, J. K., & Liddell, T. M. (2018). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin and Review, 25(1), 178206. DOI: 10.3758/s13423-016-1221-4
  65. Kwapil, T. R., & Barrantes-Vidal, N. (2015). Schizotypy: Looking back and moving forward. Schizophrenia Bulletin, 41(2), S366S373. DOI: 10.1093/schbul/sbu186
  66. Labvanced Team. (2022). Labvanced Eye-tracking Guide. Scicovery GmbH. https://www.labvanced.com/docs/guide/eyetracking/
  67. Levay, K. E., Freese, J., & Druckman, J. N. (2016). The Demographic and Political Composition of Mechanical Turk Samples. SAGE Open, 6(1). DOI: 10.1177/2158244016636433
  68. Leys, C., Delacre, M., Mora, Y. L., Lakens, D., & Ley, C. (2019). How to classify, detect, and manage univariate and multivariate outliers, with emphasis on pre-registration. International Review of Social Psychology, 32(1), 110. DOI: 10.5334/irsp.289
  69. Lourenco, S. F., & Tasimi, A. (2020). No Participant Left Behind: Conducting Science During COVID-19. Trends in Cognitive Sciences, 24(8), 583584. DOI: 10.1016/j.tics.2020.05.003
  70. Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in Psychology Research: How Often Do They Really Occur? Perspectives on Psychological Science, 7(6), 537542. DOI: 10.1177/1745691612460688
  71. Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44(1), 123. DOI: 10.3758/s13428-011-0124-6
  72. Masuda, T., Batdorj, B., & Senzaki, S. (2020). Culture and Attention: Future Directions to Expand Research Beyond the Geographical Regions of WEIRD Cultures. Frontiers in Psychology, 11, 1394. DOI: 10.3389/fpsyg.2020.01394
  73. Mathôt, S., & March, J. (2021). Conducting linguistic experiments online with OpenSesame and OSWeb. PsyArXiv. DOI: 10.31234/osf.io/wnryc
  74. Mathôt, S., Schreij, D., & Theeuwes, J. (2012). OpenSesame: An open-source, graphical experiment builder for the social sciences. Behavior Research Methods, 44(2), 314324. DOI: 10.3758/s13428-011-0168-7
  75. Meteyard, L., & Davies, R. A. I. (2020). Best practice guidance for linear mixed-effects models in psychological science. Journal of Memory and Language, 112. DOI: 10.1016/j.jml.2020.104092
  76. Meyers, E. A., Walker, A. C., Fugelsang, J. A., & Koehler, D. J. (2020). Reducing the number of non-naïve participants in Mechanical Turk samples. Methods in Psychology, 3, 100032. DOI: 10.1016/j.metip.2020.100032
  77. Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie Du Sert, N., Simonsohn, U., Wagenmakers, E. J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. DOI: 10.1038/s41562-016-0021
  78. Nelson, M. T., Seal, M. L., Pantelis, C., & Phillips, L. J. (2013). Evidence of a dimensional relationship between schizotypy and schizophrenia: A systematic review. Neuroscience and Biobehavioral Reviews, 37(3), 317327. DOI: 10.1016/j.neubiorev.2013.01.004
  79. Nosek, B. A. (2015). Promoting an open research culture: The TOP guidelines. Science, 348(6242), 14221425. DOI: 10.1126/science.aab2374
  80. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11), 26002606. DOI: 10.1073/pnas.1708274114
  81. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science, 7(6), 615631. DOI: 10.1177/1745691612459058
  82. Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867872. DOI: 10.1016/j.jesp.2009.03.009
  83. Palan, S., & Schitter, C. (2018). Prolific.ac—A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 2227. DOI: 10.1016/j.jbef.2017.12.004
  84. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon mechanical turk. Judgment and Decision Making, 5(5), 411419.
  85. Papoutsaki, A., Daskalova, N., Sangkloy, P., Huang, J., Laskey, J., & Hays, J. (2016). WebGazer: Scalable webcam eye tracking using user interactions. IJCAI International Joint Conference on Artificial Intelligence, 38393845. DOI: 10.1145/2702613.2702627
  86. Parsons, S., Kruijt, A.-W., & Fox, E. (2019). Psychological Science Needs a Standard Practice of Reporting the Reliability of Cognitive-Behavioral Measurements. Advances in Methods and Practices in Psychological Science, 2(4), 378395. DOI: 10.1177/2515245919879695
  87. Peer, E., Rothschild, D., Gordon, A., Evernden, Z., & Damer, E. (2021). Data quality of platforms and panels for online behavioral research. Behavior Research Methods, 54(4), 16431662. DOI: 10.3758/s13428-021-01694-3
  88. Peirce, J., Gray, J. R., Simpson, S., MacAskill, M., Höchenberger, R., Sogo, H., Kastman, E., & Lindeløv, J. K. (2019). PsychoPy2: Experiments in behavior made easy. Behavior Research Methods, 51(1), 195203. DOI: 10.3758/s13428-018-01193-y
  89. Prolific Team. (2022). Prolific’s Attention and Comprehension Check Policy. https://researcher-help.prolific.co/hc/en-gb/articles/360009223553-Prolific-s-Attention-and-Comprehension-Check-Policy
  90. Pronk, T. (2021). Demo Eye Tracking 2. https://gitlab.pavlovia.org/tpronk/demo_eye_tracking2
  91. Pronk, T., Wiers, R. W., Molenkamp, B., & Murre, J. (2020). Mental chronometry in the pocket? Timing accuracy of web applications on touchscreen and keyboard devices. Behavior Research Methods, 52(3), 13711382. DOI: 10.3758/s13428-019-01321-2
  92. Psychtoolbox Team. (2021). Psychtoolbox MeasureDpi. http://psychtoolbox.org/docs/MeasureDpi
  93. Raine, A., Lencz, T., Scerbo, A., & Kim, D. (1994). Disorganized Features of Schizotypal Personality. Schizophrenia Bulletin, 20(1), 191201. DOI: 10.1093/schbul/20.1.191
  94. Rodd, J. (2019). How to maintain data quality when you can’t see your participants. Psychological Science. https://www.psychologicalscience.org/observer/how-to-maintain-data-quality-when-you-cant-see-your-participants
  95. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638641. DOI: 10.1037/0033-2909.86.3.638
  96. Rousselet, G. A., Pernet, C. R., & Wilcox, R. R. (2017). Beyond differences in means: Robust graphical methods to compare two groups in neuroscience. European Journal of Neuroscience, 46(2), 17381748. DOI: 10.1111/ejn.13610
  97. RStudio Team. (2021). RStudio: Integrated development for R.
  98. Saunders, M. N. K., Lewis, P., & Thornhill, A. (2019). “Research Methods for Business Students” Chapter 4: Understanding research philosophy and approaches to theory development. In Research Methods for Business Students (8th ed., pp. 128171). Pearson Education. www.pearson.com/uk
  99. Sauter, M., Draschkow, D., & Mack, W. (2020). Building, hosting and recruiting: A brief introduction to running behavioral experiments online. Brain Sciences, 10(4), 111. DOI: 10.3390/brainsci10040251
  100. Sauter, M., Stefani, M., & Mack, W. (2022). Equal Quality for Online and Lab Data: A Direct Comparison from Two Dual-Task Paradigms. Open Psychology, 4(1), 4759. DOI: 10.1515/psych-2022-0003
  101. Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using mechanical turk to study clinical populations. Clinical Psychological Science, 1(2), 213220. DOI: 10.1177/2167702612469015
  102. Sharpe, D., & Poets, S. (2020). Meta-analysis as a response to the replication crisis. Canadian Psychology, 61(4), 377387. DOI: 10.1037/cap0000215
  103. Sheskin, M., Scott, K., Mills, C. M., Bergelson, E., Bonawitz, E., Spelke, E. S., Fei-Fei, L., Keil, F. C., Gweon, H., Tenenbaum, J. B., Jara-Ettinger, J., Adolph, K. E., Rhodes, M., Frank, M. C., Mehr, S. A., & Schulz, L. (2020). Online Developmental Science to Foster Innovation, Access, and Impact. Trends in Cognitive Sciences, 24(9), 675678. DOI: 10.1016/j.tics.2020.06.004
  104. Siddi, S., Petretto, D. R., & Preti, A. (2017). Neuropsychological correlates of schizotypy: a systematic review and meta-analysis of cross-sectional studies. Cognitive Neuropsychiatry, 22(3), 186212. DOI: 10.1080/13546805.2017.1299702
  105. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 13591366. DOI: 10.1177/0956797611417632
  106. Smythe, I., & Everatt, J. (2001). Adult Dyslexia Checklist. http://www.itcarlow.ie/public/userfiles/files/Adult-Checklist.pdf
  107. Sprouse, J. (2011). A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43(1), 155167. DOI: 10.3758/s13428-010-0039-7
  108. Stark, Z., Franzen, L., & Johnson, A. P. (2022). Insights from a dyslexia simulation font: Can we simulate reading struggles of individuals with dyslexia? Dyslexia, 28(2), 228243. DOI: 10.1002/dys.1704
  109. Sternberg, S. (1966). High-speed scanning in human memory. Science, 153(3736), 652654. DOI: 10.1126/science.153.3736.652
  110. Stoet, G. (2010). PsyToolkit: A software package for programming psychological experiments using Linux. Behavior Research Methods, 42(4), 10961104. DOI: 10.3758/BRM.42.4.1096
  111. Stoet, G. (2017). PsyToolkit: A Novel Web-Based Method for Running Online Questionnaires and Reaction-Time Experiments. Teaching of Psychology, 44(1), 2431. DOI: 10.1177/0098628316677643
  112. Uittenhove, K., Jeanneret, S., & Vergauwe, E. (2022). From lab-based to web-based behavioural research: Who you test is more important than how you test. PsyArXiv, February 16. DOI: 10.31234/osf.io/uy4kb
  113. van den Akker, O. R., Weston, S., Campbell, L., Chopik, B., Damian, R., Davis-Kean, P., Hall, A., Kosie, J., Kruse, E., Olsen, J., Ritchie, S., Valentine, K., Van ’t Veer, A., & Bakker, M. (2021). Preregistration of secondary data analysis: A template and tutorial. Meta-Psychology, 5. DOI: 10.15626/MP.2020.2625
  114. van Stolk-Cooke, K., Brown, A., Maheux, A., Parent, J., Forehand, R., & Price, M. (2018). Crowdsourcing Trauma: Psychopathology in a Trauma-Exposed Sample Recruited via Mechanical Turk. Journal of Traumatic Stress, 31(4), 549557. DOI: 10.1002/jts.22303
  115. Vogel, E. K., Woodman, G. F., & Luck, S. J. (2001). Storage of features, conjunctions, and objects in visual working memory. Journal of Experimental Psychology: Human Perception and Performance, 27(1), 92114. DOI: 10.1037/0096-1523.27.1.92
  116. Walters, K., Christakis, D. A., & Wright, D. R. (2018). Are Mechanical Turk worker samples representative of health status and health behaviors in the U.S.? PLoS ONE, 13(6), e0198835. DOI: 10.1371/journal.pone.0198835
  117. Warnke, A. (1999). Reading and spelling disorders: Clinical features and causes. European Child & Adolescent Psychiatry, 8(S3), S002S012. DOI: 10.1007/PL00010689
  118. Wilcox, R. R., & Rousselet, G. A. (2018). A Guide to Robust Statistical Methods in Neuroscience. Current Protocols in Neuroscience, 82(1), 842. DOI: 10.1002/cpns.41
  119. Woike, J. K. (2019). Upon Repeated Reflection: Consequences of Frequent Exposure to the Cognitive Reflection Test for Mechanical Turk Participants. Frontiers in Psychology, 10, 2646. DOI: 10.3389/fpsyg.2019.02646
  120. Woods, A. T., Spence, C., Butcher, N., & Deroy, O. (2013). Fast lemons and sour boulders: Testing crossmodal correspondences using an internet-based testing methodology. I-Perception, 4(6), 365379. DOI: 10.1068/i0586
  121. Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015). Conducting perception research over the internet: a tutorial review. PeerJ, 3, e1058. DOI: 10.7717/peerj.1058
  122. Yetano, A., & Royo, S. (2017). Keeping Citizens Engaged: A Comparison Between Online and Offline Participants. Administration and Society, 49(3), 394422. DOI: 10.1177/0095399715581625
DOI: https://doi.org/10.5334/spo.34 | Journal eISSN: 2752-5341
Language: English
Submitted on: Jan 23, 2022
|
Accepted on: Dec 23, 2022
|
Published on: Jan 4, 2023
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
Publication frequency: 1 issue per year

© 2023 Nathan Gagné, Léon Franzen, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.