Making Barriers to Learning in MOOCs Visible. A Factor Analytical Approach

Maartje Henderikx; Karel Kreijns; Kate M. Xu; Marco Kalz

doi:10.5944/openpraxis.13.2.124

Making Barriers to Learning in MOOCs Visible. A Factor Analytical Approach

Open Praxis

Volume 13 (2021): Issue 2

By: Maartje Henderikx , Karel Kreijns , Kate M. Xu and Marco Kalz

Open Access

|Oct 2021

Abstract

Learners in MOOCs often experience challenges that can be identified as barriers to learning. These barriers may be MOOC- or not MOOC-related. By knowing about potential barriers learners would be better prepared and more likely to handle and overcome them. Therefore, the aim of this study was to advance insight and knowledge about barriers to learning in MOOCs. Assessment and reassessment of the data using exploratory factor analysis provided a good model fit for a 6-factor structure. This was confirmed by a confirmatory factor analysis. Further classification of the factors revealed that barriers experienced by learners were predominantly non-MOOC related. To get insight into the barriers learners experience, it was suggested to convert the identified factor structure into a diagnostic instrument (dashboard) powered by learner self-report. This dashboard then provides information about barriers learners experience and can be valuable for making (re) design decisions and for developing learner supporting tools and interventions.

References

Adamopoulos, P. (2013). What Makes a Great MOOC? An Interdisciplinary Analysis of Student Retention in Online Courses. In 34th International Conference on Information Systems: ICIS 2013. United States: Association for Information Systems. http://pages.stern.nyu.edu/~padamopo/What%20makes%20a%20great%20MOOC.pdf
Search in Google Scholar Back to article
Antonaci, A., Klemke, R., Dirkx, K., & Specht, M. (2019). May the plan be with you! A usability study of the stimulated planning game element embedded in a MOOC platform. International Journal of Serious Games, 6(1), 49–70. DOI: 10.17083/ijsg.v6i1.239
Open DOI Search in Google Scholar Back to article
Balfour, S. P. (2013). Assessing Writing in MOOCs: Automated Essay Scoring and Calibrated Peer Review. Research & Practice in Assessment, 8, 40–48. https://eric.ed.gov/?id=EJ1062843
Search in Google Scholar Back to article
Barnes, C. (2013). MOOCs: The challenges for academic librarians. Australian Academic & Research Libraries, 44(3), 163–175. DOI: 10.1080/00048623.2013.821048
Open DOI Search in Google Scholar Back to article
Barnett, S. D., Hickling, E. J., & Sheppard, S. (2018). The impact of gender on the factor structure of PTSD symptoms among active duty United States military personnel. European Journal of Trauma & Dissociation, 2(3), 117–124. DOI: 10.1016/j.ejtd.2018.01.002
Open DOI Search in Google Scholar Back to article
Belanger, Y., & Thornton, J. (2013). Bioelectricity: A quantitative approach. Durham, NC. http://dukespace.lib.duke.edu/dspace/bitstream/handle/10161/6216/Duke_Bioelectricity_MOOC_Fall2012.pdf?sequence=1
Search in Google Scholar Back to article
Boyatt, R., Joy, M., Rocks, C., & Sinclair, J. (2013). What (Use) is a MOOC? In L. Uden, Y. Tao, H. Yang & I. Ting (Eds.), Springer proceedings in complexity: 2nd International Workshop on Learning Technology for Education in Cloud (pp. 133–145). DOI: 10.1007/978-94-007-7308-0_15
Open DOI Search in Google Scholar Back to article
Bryant, F. B., & Yarnold, P. R. (1995). Principal-components analysis and exploratory and confirmatory factor analysis. In L. G. Grimm & P. R. Yarnold (Eds.), Reading and understanding multivariate statistics (p. 99–136). American Psychological Association. https://psycnet.apa.org/record/1995-97110-004
Search in Google Scholar Back to article
Byrne, B. M. (2005). Factor analytic models: Viewing the structure of an assessment instrument from three perspectives. Journal of personality assessment, 85(1), 17–32. DOI: 10.1207/s15327752jpa8501_02
Open DOI Search in Google Scholar Back to article
Byrne, B. M. (2012). Structural equation modeling with Mplus: Basic concepts, applications, and programming. New York: Routledge. DOI: 10.4324/9780203807644
Open DOI Search in Google Scholar Back to article
Carr, S. (2000). As distance education comes of age, the challenge is keeping the students. The Chronicle of Higher Education, 4, A39–A41. https://eric.ed.gov/?id=EJ601725
Search in Google Scholar Back to article
Comrey, A. L., & Lee, H. B. (2013). A first course in factor analysis (2nd ed.). Psychology Press. DOI: 10.4324/9781315827506
Open DOI Search in Google Scholar Back to article
Drake, K. E., & Egan, V. (2017). Investigating gender differences in the factor structure of the Gudjonsson Compliance Scale. Legal and Criminological Psychology, 22(1), 88–98. DOI: 10.1111/lcrp.12081
Open DOI Search in Google Scholar Back to article
Duffy, M. C., Lajoie, S. P., Pekrun, R., & Lachapelle, K. (2018). Emotions in medical education: Examining the validity of the Medical Emotion Scale (MES) across authentic medical learning environments. Learning and Instruction. DOI: 10.1016/j.learninstruc.2018.07.001
Open DOI Search in Google Scholar Back to article
Fan, X., Thompson, B., & Wang, L. (1999). Effects of sample size, estimation methods, and model specification on structural equation modeling fit indexes. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 56–83. DOI: 10.1080/10705519909540119
Open DOI Search in Google Scholar Back to article
Fishbein, M., & Ajzen, I. (2010). Predicting and changing behaviour: The reasoned action approach. Psychology Press (Taylor & Francis). DOI: 10.4324/9780203838020
Open DOI Search in Google Scholar Back to article
Gamage, D., Fernando, S., & Perera, I. (2015, August). Quality of MOOCs: A review of literature on effectiveness and quality aspects. In Ubi-Media Computing (UMEDIA), 2015 8th International Conference (pp. 224–229). IEEE. DOI: 10.1109/UMEDIA.2015.7297459
Open DOI Search in Google Scholar Back to article
Greene, J. A., Oswald, C. A., & Pomerantz, J. (2015). Predictors of Retention and Achievement in a Massive Open Online Course. American Educational Research Journal, 52(5), 925–955. DOI: 10.3102/0002831215584621
Open DOI Search in Google Scholar Back to article
Grover, S., Franz, P., Schneider, E., & Pea, R. (2013, June). The MOOC as distributed intelligence: Dimensions of a framework & evaluation of MOOCs. In Proceedings CSCL, 2, 42–5. https://repository.isls.org/bitstream/1/1940/1/42-45.pdf
Search in Google Scholar Back to article
Harrington, D. (2009). Confirmatory factor analysis. Oxford New York Press. DOI: 10.1093/acprof:oso/9780195339888.001.0001
Open DOI Search in Google Scholar Back to article
Hastie, T., Tibshirani, R., & Friedman, J. (2001). The elements of statistical learning: Data mining, inference, and prediction. Springer. DOI: 10.1007/BF02985802
Open DOI Search in Google Scholar Back to article
Hayduk, L. A. (2014). Shame for disrespecting evidence: The personal consequences of insufficient respect for structural equation model testing. BMC Medical Research Methodology, 14, 124. DOI: 10.1186/1471-2288-14-124
Open DOI Search in Google Scholar Back to article
Henderikx, M. A. (2019). Mind The Gap: Unravelling learner success and behaviour in Massive Open Online Courses [doctoral dissertation]. Open Universiteit Netherlands. https://research.ou.nl/ws/files/11849471/DissertationMaartjeHenderikx.pdf
Search in Google Scholar Back to article
Henderikx, M. A., Kreijns, K., & Kalz, M. (2017). Refining success and dropout in massive open online courses based on the intention–behavior gap. Distance Education, 38(3), 353–368. DOI: 10.1080/01587919.2017.1369006
Open DOI Search in Google Scholar Back to article
Henderikx, M., Kreijns, K., & Kalz M. (2018a). A classification of barriers that influence intention achievement in MOOCs. In V. Pammer-Schindler, M. Pérez-Sanagustín, H. Drachsler, R. Elferink & M. Scheffel (Eds.), Lifelong technology-enhanced learning. EC-TEL 2018. LNCS, 11082, 3–15. DOI: 10.1007/978-3-319-98572-5_1
Open DOI Search in Google Scholar Back to article
Henderikx, M. A., Kreijns, C., & Kalz, M. (2018b). Intention – behavior dynamics in MOOCs Learning. What happens to good intentions along the way? In 2018 Learning With MOOCS (LWMOOCS): Proceedings of the Fifth Learning with MOOCs Conference (pp. 110–112). IEEE. DOI: 10.1109/LWMOOCS.2018.8534595
Open DOI Search in Google Scholar Back to article
Henderikx, M., Kreijns, K., Castaño Muñoz, J., & Kalz, M. (2019). Factors influencing the pursuit of personal learning goals in MOOCs. Distance Education, 40(2), 187–204. DOI: 10.1080/01587919.2019.1600364
Open DOI Search in Google Scholar Back to article
Ho, A. D., Reich, J., Nesterko, S., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses. HarvardX Working Paper No. 1. http://harvardx.harvard.edu/multiple-course-report. DOI: 10.2139/ssrn.2381263
Open DOI Search in Google Scholar Back to article
Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A survey study. Computers & Education, 98, 157–168. DOI: 10.1016/j.compedu.2016.03.016
Open DOI Search in Google Scholar Back to article
Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods (EJBRM), 6(1), 53–60. https://academic-publishing.org/index.php/ejbrm/article/view/1224
Search in Google Scholar Back to article
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. DOI: 10.1080/10705519909540118
Open DOI Search in Google Scholar Back to article
Idrees, M., Hafeez, M., & Kim, J. Y. (2017). Workers’ age and the impact of psychological factors on the perception of safety at construction sites. Sustainability, 9(5), 745. https://www.mdpi.com/2071-1050/9/5/745/htm. DOI: 10.3390/su9050745
Open DOI Search in Google Scholar Back to article
Jansen, R. S., van Leeuwen, A., Janssen, J., Conijn, R., & Kester, L. (2020). Supporting learners’ self-regulated learning in Massive Open Online Courses. Computers & Education, 146, 103771. DOI: 10.1016/j.compedu.2019.103771
Open DOI Search in Google Scholar Back to article
Kenny, D. A., Kaniskan, B., & McCoach, D. B. (2015). The performance of RMSEA in models with small degrees of freedom. Sociological Methods & Research, 44(3), 486–507. DOI: 10.1016/j.compedu.2019.103771
Open DOI Search in Google Scholar Back to article
Khalil, H., & Ebner, M. (2013). Interaction Possibilities in MOOCs – How Do They Actually Happen? International Conference on Higher Education Development (pp. 1–24). Egypt: Mansoura University. https://www.scribd.com/document/134249470/Interaction-Possibilities-in-MOOCs-How-Do-They-Actually-Happen. DOI: 10.1177/0049124114543236
Open DOI Search in Google Scholar Back to article
Khalil, H., & Ebner, M. (2014). MOOCs Completion Rates and Possible Methods to Improve Retention – A Literature Review. In World Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 1236–1244). Chesapeak, VA: AACE. https://www.learntechlib.org/p/147656
Search in Google Scholar Back to article
Koller, D., Ng, A., Do, C., & Chen, Z. (2013). Retention and intention in massive open online courses: In depth. Educause Review Online. http://er.educause.edu/articles/2013/6/retention-and-intention-in-massive-open-online-courses-in-depth
Search in Google Scholar Back to article
Mackness, J., Mak, S., & Williams, R. (2010). The ideals and reality of participating in a MOOC. In Proceedings of the 7th international conference on networked learning 2010 (pp. 266–275). University of Lancaster. https://researchportal.port.ac.uk/portal/en/publications/the-ideals-and-reality-of-participating-in-a-mooc(067e281e-6637-423f-86a5-ff4d2d687af1).html
Search in Google Scholar Back to article
Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103(3), 391–410. DOI: 10.1037/0033-2909.103.3.391
Open DOI Search in Google Scholar Back to article
Marsh, H. W., Hau, K. T., & Grayson, D. A. (2005). Goodness of fit evaluation in structural equation modeling. In A. Maydeu-Olivares & J. J. McArdle (Eds.), Contemporary psychometrics. A festschrift to Roderick P. McDonald (pp. 225–340). n. https://psycnet.apa.org/record/2005-04585-010
Search in Google Scholar Back to article
Marsh, H. W., Hau, K. T., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) Findings. Structural Equation Modeling: A Multidisciplinary Journal, 11(3), 320–341. DOI: 10.1207/s15328007sem1103_2
Open DOI Search in Google Scholar Back to article
McAuley, A., Stewart, B., Siemens, G., & Cormier, D. (2010). The MOOC model for digital practice. https://oerknowledgecloud.org/sites/oerknowledgecloud.org/files/MOOC_Final.pdf
Search in Google Scholar Back to article
McNeish, D., An, J., & Hancock, G. R. (2018). The thorny relation between measurement quality and fit index cutoffs in latent variable models. Journal of personality assessment, 100(1), 43–52. DOI: 10.1080/00223891.2017.1281286
Open DOI Search in Google Scholar Back to article
Misopoulos, F., Argyropoulou, M., & Tzavara, D. (2018). Exploring the factors affecting student academic performance in online programs: a literature review. In A. Khare & D. Hurst (Eds.), On the line (pp. 235–249). DOI: 10.1007/978-3-319-62776-2_18
Open DOI Search in Google Scholar Back to article
Muilenburg, L. Y., & Berge, Z. L. (2005). Student barriers to online learning: A factor analytic study. Distance Education, 26(1), 29–48. DOI: 10.1080/01587910500081269
Open DOI Search in Google Scholar Back to article
Muthén, L. K., & Muthén, B. O. (1998–2014). MPlus user’s guide (8th ed.). Los Angeles, CA. Retrieved from https://www.statmodel.com/download/usersguide/MplusUserGuideVer_8.pdf
Search in Google Scholar Back to article
Ng, S.-M. (2013). Validation of the 10-item Chinese perceived stress scale in elderly service workers: one-factor versus two-factor structure. BMC Psychology, 9. https://link.springer.com/article/10.1186/2050-7283-1-9. DOI: 10.1186/2050-7283-1-9
Open DOI Search in Google Scholar Back to article
Onah, D. F., Sinclair, J., & Boyatt, R. (2014). Dropout rates of massive open online courses: behavioural patterns. In International conference on education and new learning technologies. EDULEARN14 proceedings, 1, 5825–5834. Barcelona. http://wrap.warwick.ac.uk/65543/
Search in Google Scholar Back to article
Osborne, J. W., Costello, A. B., & Kellow, J. T. (2008). Best practices in exploratory factor analysis. Best practices in quantitative methods (pp. 86–99). DOI: 10.4135/9781412995627.d8
Open DOI Search in Google Scholar Back to article
Preacher, K. J. (2006). Testing complex correlational hypotheses with structural equation models. Structural Equation Modeling, 13(4), 520–543. DOI: 10.1207/s15328007sem1304_2
Open DOI Search in Google Scholar Back to article
Preacher, K. J., Zhang, G., Kim, C., & Mels, G. (2013). Choosing the optimal number of factors in exploratory factor analysis: A model selection perspective. Multivariate Behavioral Research, 48, 28–56. DOI: 10.1080/00273171.2012.710386
Open DOI Search in Google Scholar Back to article
Prudon, P. (2014). Confirmatory factor analysis: a brief introduction and critique. Tilgængelig på. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.476.6207&rep=rep1&type=pdf
Search in Google Scholar Back to article
Reich, J. (2014). MOOC completion and retention in the context of student intent. Educause Review Online. http://er.educause.edu/articles/2014/12/mooc-completion-and-retention-in-the-context-of-student-intent
Search in Google Scholar Back to article
Reich, J., & Ruipérez-Valiente, J. A. (2019). The MOOC pivot. Science, 363(6423), 130–131. DOI: 10.1126/science.aav7958
Open DOI Search in Google Scholar Back to article
Rhemtulla, M., Brosseau-Liard, P. E., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological methods, 17(3), 354–373. DOI: 10.1037/a0029315
Open DOI Search in Google Scholar Back to article
Saleh, A., & Bista, K. (2017). Examining factors impacting online survey response rates in educational research: Perceptions of graduate students. Journal of MultiDisciplinary Evaluation, 13(29), 63–74. https://files.eric.ed.gov/fulltext/ED596616.pdf
Search in Google Scholar Back to article
Shapiro, H. B., Lee, C. H., Roth, N. E. W., Li, K., Çetinkaya-Rundel, M., & Canelas, D. A. (2017). Understanding the massive open online course (MOOC) student experience: An examination of attitudes, motivations, and barriers. Computers & Education, 110, 35–50. DOI: 10.1016/j.compedu.2017.03.003
Open DOI Search in Google Scholar Back to article
Schmitt, T. A. (2011). Current methodological considerations in exploratory and confirmatory factor analysis. Journal of Psychoeducational Assessment, 29, 304–321. DOI: 10.1177/0734282911406653
Open DOI Search in Google Scholar Back to article
Schmitt, T. A., Sass, D. A., Chappelle, W., & Thompson, W. (2018). Selecting the best factor structure and moving measurement validation forward: An illustration. Journal of personality assessment, 100(4), 345–362. DOI: 10.1080/00223891.2018.1449116
Open DOI Search in Google Scholar Back to article
Shin, N. (2003). Transactional presence as a critical predictor of success in distance learning. Distance Education, 24(1), 69–68. DOI: 10.1080/01587910303048
Open DOI Search in Google Scholar Back to article
Tabachnick, B. G., & Fidell, L. S. (2013). Using Multivariate Statistics (6th Ed., New International Edition). Pearson Education Limited.
Search in Google Scholar Back to article
Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296. DOI: 10.1007/s11165-016-9602-2
Open DOI Search in Google Scholar Back to article
Urushihata, T., Kinugasa, T., Soma, Y., & Miyoshi, H. (2010). Aging effects on the structure underlying balance abilities tests. Journal of the Japanese Physical Therapy Association, 13(1), 1–8. DOI: 10.1298/jjpta.13.1
Open DOI Search in Google Scholar Back to article
Wegener, D. T., & Fabrigar, L. R. (2000). Analysis and design for nonexperimental data addressing casual and noncausal hypotheses. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 412–450). Cambridge University Press. DOI: 10.1017/CBO9780511996481.024
Open DOI Search in Google Scholar Back to article
Yu, C.-Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. Los Angeles: University of California. https://pdfs.semanticscholar.org/7a22/ae22553f78582fc61c6cab4567d36998293b.pdf
Search in Google Scholar Back to article

Articles in this issue

DOI: https://doi.org/10.5944/openpraxis.13.2.124 | Journal eISSN: 2304-070X

Journal RSS Feed

Language: English

Page range: 143 - 159

Submitted on: Oct 18, 2020

Accepted on: May 31, 2021

Published on: Oct 20, 2021

Published by: International Council for Open and Distance Education (ICDE)

In partnership with: Paradigm Publishing Services

Keywords:

barriers to learning,

online learning,

MOOCs,

open education,

factor analysis

© 2021 Maartje Henderikx, Karel Kreijns, Kate M. Xu, Marco Kalz, published by International Council for Open and Distance Education (ICDE)
This work is licensed under the Creative Commons Attribution 4.0 License.

Volume 13 (2021): Issue 2