Qualitative Individual Differences are Useful, but Reliability Should be Assessed and Not Assumed

Craig Hedge

doi:10.5334/joc.169

Abstract

Rouder and Haaf (2021) propose that studying qualitative individual differences would be a useful tool for researchers. I agree with their central message. I use this commentary to highlight examples from the literature where similar questions have been asked, and how researchers have addressed them with existing tools. I also observe that while the hierarchical Bayesian framework is a useful tool for studying individual differences, it does not relieve us of the requirement to evaluate the forms of reliability that are critical to our research questions.

References

Brown, V. M., Chen, J., Gillan, C. M., & Price, R. B. (2020). Improving the Reliability of Computational Analyses: Model-Based Planning and Its Relationship With Compulsivity. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging. DOI: 10.1016/j.bpsc.2019.12.019
Open DOI Search in Google Scholar Back to article
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of Generalizability: a Liberalization of Reliability Theory. British Journal of Statistical Psychology, 16(2), 137–163. DOI: 10.1111/j.2044-8317.1963.tb00206.x
Open DOI Search in Google Scholar Back to article
Haaf, J. M., & Rouder, J. N. (2017). Developing Constraint in Bayesian Mixed Models. Psychological Methods, 22(4), 779–798. DOI: 10.1037/met0000156
Open DOI Search in Google Scholar Back to article
Hedge, C., Powell, G., & Sumner, P. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50(3), 1166–1186. DOI: 10.3758/s13428-017-0935-1
Open DOI Search in Google Scholar Back to article
Hedge, C., Vivian-Griffiths, S., Powell, G., Bompas, A., & Sumner, P. (2019). Slow and steady? Strategic adjustments in response caution are moderately reliable and correlate across tasks. Consciousness and Cognition, 75. DOI: 10.1016/j.concog.2019.102797
Open DOI Search in Google Scholar Back to article
Novick, M. R. (1966). The axioms and principal results of classical test theory. Journal of Mathematical Psychology, 3, 1–18. DOI: 10.1016/0022-2496(66)90002-2
Open DOI Search in Google Scholar Back to article
Parsons, S., Kruijt, A.-W., & Fox, E. (2019). Psychological Science Needs a Standard Practice of Reporting the Reliability of Cognitive-Behavioral Measurements. Advances in Methods and Practices in Psychological Science, 2(4), 378–395. DOI: 10.1177/2515245919879695
Open DOI Search in Google Scholar Back to article
Rae, B., Heathcote, A., Donkin, C., Averell, L., & Brown, S. (2014). The hare and the tortoise: Emphasizing speed can change the evidence used to make decisions. Journal of Experimental Psychology: Learning Memory and Cognition, 40(5), 1226–1243. DOI: 10.1037/a0036801
Open DOI Search in Google Scholar Back to article
Rouder, J. N., & Haaf, J. M. (2019). A psychometrics of individual differences in experimental tasks. Psychonomic Bulletin and Review, 26(2), 452–467. DOI: 10.3758/s13423-018-1558-y
Open DOI Search in Google Scholar Back to article
Rouder, J. N., & Haaf, J. M. (2021). Are There Reliable Qualitative Individual Difference in Cognition? Journal of Cognition, 4(1): 46, 1–16. DOI: 10.31234/osf.io/3ezmw
Open DOI Search in Google Scholar Back to article
Tidwell, J. W., Dougherty, M. R., Chrabaszcz, J. R., Thomas, R. P., & Mendoza, J. L. (2014). What counts as evidence for working memory training? Problems with correlated gains and dichotomization. Psychonomic Bulletin and Review. DOI: 10.3758/s13423-013-0560-7
Open DOI Search in Google Scholar Back to article
Watts, F. N., McKenna, F. P., Sharrock, R., & Trezise, L. (1986). Colour naming of phobia‐related words. British Journal of Psychology, 77(1), 97–108. DOI: 10.1111/j.2044-8295.1986.tb01985.x
Open DOI Search in Google Scholar Back to article
Wiecki, T. V., Sofer, I., & Frank, M. J. (2013). HDDM: Hierarchical Bayesian estimation of the Drift-Diffusion Model in Python. Frontiers in Neuroinformatics, 7. DOI: 10.3389/fninf.2013.00014
Open DOI Search in Google Scholar Back to article
Wilkinson, L. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54(8), 594–604. DOI: 10.1037/0003-066X.54.8.594
Open DOI Search in Google Scholar Back to article

Qualitative Individual Differences are Useful, but Reliability Should be Assessed and Not Assumed

Abstract

Paradigm

My account