Conducting Power Analyses to Determine Sample Sizes in Quantitative Research: A Primer for Technology Education Researchers Using Common Statistical Tests
By: Jeffery Buckley
References
- Aczel, B., Szaszi, B., Nilsonne, G., Holzmeister, F., Kosa, L., & Wagenmakers, E.-J. (2022). The Multi100 project. OSF Preprints.
https://osf.io/https://osf.io/7snkz - Ankiewicz, P. (2016).
Perceptions and attitudes of pupils towards technology . In M. de Vries (Ed.), Handbook of Technology Education (pp. 581–595). Springer. 10.1007/978-3-319-44687-5_43 - Bartholomew, S., & Jones, M. (2021). A systematized review of research with adaptive comparative judgment (ACJ) in higher education. International Journal of Technology and Design Education. 10.1007/s10798-020-09642-6
- Bartholomew, S., Strimel, G., & Yoshikawa, E. (2019). Using adaptive comparative judgment for student formative feedback and learning during a middle school design project. International Journal of Technology and Design Education, 29(2), 363–385. 10.1007/s10798-018-9442-7
- Bartholomew, S., & Yoshikawa-Ruesch, E. (2018).
A systematic review of research around adaptive comparative judgement (ACJ) in K-16 education . In J. Wells (Ed.), CTETE – Research Monograph Series (Vol. 1, pp. 6–28). Council on Technology and Engineering Teacher Education. - Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M., Kirchler, M., Iwanir, R., Mumford, J. A., Adcock, R. A., Avesani, P., Baczkowski, B. M., Bajracharya, A., Bakst, L., Ball, S., Barilari, M., Bault, N., Beaton, D., Beitner, J., … Schonberg, T. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582(7810), 84–88. 10.1038/s41586-020-2314-9
- Buckley, J. (2023). Considering the credibility of technology education research: A discussion on empirical insights and possible next steps. Proceedings of the International PATT40 Conference.
- Buckley, J., Adams, L., Aribilola, I., Arshad, I., Azeem, M., Bracken, L., Breheny, C., Buckley, C., Chimello, I., Fagan, A., Fitzpatrick, D. P., Garza Herrera, D., Gomes, G. D., Grassick, S., Halligan, E., Hirway, A., Hyland, T., Imtiaz, M. B., Khan, M. B., … Zhang, L. (2022). An assessment of the transparency of contemporary technology education research employing interview-based methodologies. International Journal of Technology and Design Education, 32(4), 1963–1982. 10.1007/s10798-021-09695-1
- Buckley, J., Araujo, J. A., Aribilola, I., Arshad, I., Azeem, M., Buckley, C., Fagan, A., Fitzpatrick, D. P., Garza Herrera, D. A., Hyland, T., Imtiaz, M. B., Khan, M. B., Lanzagorta Garcia, E., Moharana, B., Mohd Sufian, M. S. Z., Osterwald, K. M., Phelan, J., Platonava, A., Reid, C., … Zainol, I. (2023). How transparent are quantitative studies in contemporary technology education research? Instrument development and analysis. International Journal of Technology and Design Education. 10.1007/s10798-023-09827-9
- Buckley, J., Canty, D., & Seery, N. (2022). An exploration into the criteria used in assessing design activities with adaptive comparative judgment in technology education. Irish Educational Studies, 41(2), 313–331. 10.1080/03323315.2020.1814838
- Buckley, J., Hyland, T., & Seery, N. (2021). Examining the replicability of contemporary technology education research. Techne Series: Research in Sloyd Education and Craft Sciences, 28(2), 1–9.
- Buckley, J., Hyland, T., & Seery, N. (2023). Estimating the replicability of technology education research. International Journal of Technology and Design Education, 33(4), 1243–1264. 10.1007/s10798-022-09787-6
- Buckley, J., Seery, N., & Kimbell, R. (2022). A review of the valid methodological use of adaptive comparative judgment in technology education research. Frontiers in Education, 7(787926), 1–6. 10.3389/feduc.2022.787926
- Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science. 10.1126/science.aaf0918
- Champely, S. (2020). pwr: Basic Functions for Power Analysis (R package version 1.3–0) [Computer software].
https://CRAN.R-project.org/package=pwr - Cochran, W. (1977). Sampling techniques (3rd Ed.). John Wiley & Sons.
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Lawrence Erlbaum Associates.
- Cohen, J. (1992). Statistical power analysis. Current Directions in Psychological Science, 1(3), 98–101. 10.1111/1467–;8721.ep10768783
- DeBruine, L. (2021). faux: Simulation for Factorial Designs (R package version 1.1.0) [R].
https://debruine.github.io/faux/ - Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Investigating the replicability of preclinical cancer biology. eLife, 10,
e71601 . 10.7554/eLife.71601 - Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. 10.3758/bf03193146
- Francis, J. J., Johnston, M., Robertson, C., Glidewell, L., Entwistle, V., Eccles, M. P., & Grimshaw, J. M. (2010). What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychology & Health, 25(10), 1229–1245. 10.1080/08870440903194015
- Funder, D. C., & Ozer, D. J. (2019). Evaluating Effect Size in Psychological Research: Sense and Nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168. 10.1177/2515245919847202
- Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine Publishing.
- Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough?: An experiment with data saturation and variability. Field Methods, 18(1), 59–82. 10.1177/1525822X05279903
- Hartell, E., & Buckley, J. (2021).
Comparative judgement: An overview . In A. Marcus Quinn & T. Hourigan (Eds.), Handbook for Online Learning Contexts: Digital, Mobile and Open (pp. 289–307). Springer International Publishing. 10.1007/978-3-030-67349-9_20 - Hoogeveen, S., Sarafoglou, A., Aczel, B., Aditya, Y., Alayan, A. J., Allen, P. J., Altay, S., Alzahawi, S., Amir, Y., Anthony, F.-V., Kwame Appiah, O., Atkinson, Q. D., Baimel, A., Balkaya-Ince, M., Balsamo, M., Banker, S., Bartoš, F., Becerra, M., Beffara, B., … Wagenmakers, E.-J. (2023). A many-analysts approach to the relation between religiosity and well-being. Religion, Brain & Behavior, 13(3), 237–283. 10.1080/2153599X.2022.2070255
- Jak, S., Jorgensen, T. D., Verdam, M. G. E., Oort, F. J., & Elffers, L. (2021). Analytical power calculations for structural equation modeling: A tutorial and Shiny app. Behavior Research Methods, 53(4), 1385–1406. 10.3758/s13428-020-01479-0
- Kimbell, R., Martin, G., Wharfe, W., Wheeler, T., Perry, D., Miller, S., Shepard, T., Hall, P., & Potter, J. (2005). E-scape portfolio assessment: Phase 1 report. Goldsmiths, University of London.
http://research.gold.ac.uk/1527/ - Kimbell, R., Wheeler, T., Miller, S., & Pollitt, A. (2007). E-scape portfolio assessment: Phase 2 report. Goldsmiths, University of London.
https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/606018/0107_RichardKimball_et_al_e-scape2report.pdf - Kimbell, R., Wheeler, T., Stables, K., Shepard, T., Martin, F., Davies, D., Pollitt, A., & Whitehouse, G. (2009). E-scape portfolio assessment: Phase 3 report. Goldsmiths, University of London.
- Lafit, G., Adolf, J. K., Dejonckheere, E., Myin-Germeys, I., Viechtbauer, W., & Ceulemans, E. (2021). Selection of the Number of Participants in Intensive Longitudinal Studies: A User-Friendly Shiny App and Tutorial for Performing Power Analysis in Multilevel Regression Models That Account for Temporal Dependencies. Advances in Methods and Practices in Psychological Science, 4(1),
2515245920978738 . 10.1177/2515245920978738 - Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4,
863 . 10.3389/fpsyg.2013.00863 - Lakens, D. (2019). The value of preregistration for psychological science: A conceptual analysis. Japanese Psychological Review, 62(3), 221–230. 10.24602/sjpr.62.3_221
- Lakens, D. (2021). Sample size justification. PsyArXiv. 10.31234/osf.io/9d3yf
- Lakens, D., & Caldwell, A. R. (2021). Simulation-Based Power Analysis for Factorial Analysis of Variance Designs. Advances in Methods and Practices in Psychological Science, 4(1),
2515245920951503 . 10.1177/2515245920951503 - Lakens, D., & Evers, E. R. K. (2014). Sailing From the Seas of Chaos Into the Corridor of Stability: Practical Recommendations to Increase the Informational Value of Studies. Perspectives on Psychological Science, 9(3), 278–292. 10.1177/1745691614528520
- Lakens, D., Pahlke, F., & Wassmer, G. (2023). Group Sequential Designs: A Tutorial. 10.31234/osf.io/x4azm
- Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259–269. 10.1177/2515245918770963
- Low, J. (2019). A pragmatic definition of the concept of theoretical saturation. Sociological Focus, 52(2), 131–139. 10.1080/00380237.2018.1544514
- Makowski, D., Ben-Shachar, M. S., Patil, I., & Lüdecke, D. (2020). Methods and Algorithms for Correlation Analysis in R. Journal of Open Source Software, 5(51),
2306 . 10.21105/joss.02306 - Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), 943. 10.1126/science.aac4716
- Qin, X. (2023). Sample size and power calculations for causal mediation analysis: A Tutorial and Shiny App. Behavior Research Methods. 10.3758/s13428-023-02118-0
- Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003). One Hundred Years of Social Psychology Quantitatively Described. Review of General Psychology, 7(4), 331–363. 10.1037/1089-2680.7.4.331
- Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., … Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. 10.1177/2515245917747646
- Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712. 10.1177/1745691616658637
DOI: https://doi.org/10.21061/jte.v35i2.a.5 | Journal eISSN: 1045-1064
Language: English
Page range: 81 - 109
Submitted on: Jan 5, 2024
Accepted on: Apr 5, 2024
Published on: Jun 25, 2024
Published by: Virginia Tech
In partnership with: Paradigm Publishing Services
Keywords:
© 2024 Jeffery Buckley, published by Virginia Tech
This work is licensed under the Creative Commons Attribution 4.0 License.