Figure 1
Flow chart of the literature search process.
Table 1
Table of articles included in this meta-analysis.
| CODE | AUTHOR | TOTAL N | AGE | DURATION | FOCUS | TARGET LANGUAGE | APPLICATION | DESIGN | ORIGIN COUNTRY | TYPE | SOURCE |
|---|---|---|---|---|---|---|---|---|---|---|---|
| A1 | Lan & Lin, 2016 | 34 | 17–23 | 4 weeks | Communication | Chinese | Mobile Seamless System (MOSE) | Between-subjects, Mixed Methods | Taiwan | Journal | jstor.org/stable/10.2307/jeductechsoci.19.3.335 |
| A2 | Rachels & Rockinson-Szapkiw, 2017 | 167 | 3rdand 4thgraders | 12 weeks | Vocabulary & Grammar | Spanish | Duolingo | Quasi-Experimental | United States | Journal | https://doi.org/10.1080/09588221.2017.1382536 |
| A3 | Sandberg, Maris, & Geus, 2011 | 75 | 8–10 | 1 day | Vocabulary | English | MEL Application | Quasi-Experimental | Netherlands | Journal | https://doi.org/10.1016/j.compedu.2011.01.015 |
| A4 | Basoglu & Akdemir, 2010 | 60 | 17–24 | 6 weeks | Vocabulary | English | ETACO mobile flashcards | Between-Subject, Mixed Methods | Turkey | Journal | ERIC Number: EJ898010 https://eric.ed.gov/?id=EJ898010 |
| A5 | Cavus & Ibrahim, 2017 | 37 | 10–13 | 4 weeks | Listening, Vocabulary, Comprehension Pronunciation | English | Near East University Children’s Story Teller (NEUCST) | Between-Subjects | Turkey | Journal | https://doi.org/10.1111/bjet.12427 |
| A6 | Azabdaftari & Mozaheb, 2012 | 80 | 21 | 7 weeks | Vocabulary | English | Spaced Repetition System (SRS) | Between-Subject, Mixed Methods | Iran | Journal | ERIC Number: EJ1064983 https://eric.ed.gov/?id=EJ1064983 |
| A7 | Ibrahim, Chee & Yahaya, 2017 | 53 | Primary school | 4 weeks | Reading | Chinese | Learn Chinese Mandarin App | Quasi-Experimental | Malaysia | Journal | https://doi.org/10.1504/IJMLO.2017.10005992 |
| A8 | Ozer & Kilic, 2018 | 63 | 18–22 | 6 weeks | Grammar | English | Variety of mobile apps | Not Explicit, Between-Subjects | Turkey | Journal | https://doi.org/10.29333/ejmste/90992 |
| A9 | Kilickaya & Krajka, 2010 | 38 | 17–19 | 5 weeks | Vocabulary | English | WordChamp | Between-Subjects | Turkey | Journal | ERIC Number: EJ898003 https://eric.ed.gov/?id=EJ898003 |
| A10 | Wu, 2014 | 50 | 20–23 | 1 semester | Vocabulary | English | Word Learning | Not Explicit, Between-Subjects | China | Grey | https://doi.org/10.1016/j.sbspro.2014.07.409 |
| A11 | Wu, 2015a | 70 | 20–23 | 8 weeks | Vocabulary | English | Word Learning-CET6 | Not Explicit, Between-Subjects | China | Journal | https://doi.org/10.1016/j.compedu.2015.02.013 |
| A12 | Wu, 2015b | 199 | 20–22 | 1 semester | Vocabulary | English | Word Learning-CET4 | Not Explicit, Between-Subjects | China | Journal | https://doi.org/10.1371/journal.pone.0128762 |
| A13 | Fageeh, 2013 | 58 | University students | 1 semester | Vocabulary | English | Android Online Dictionary | Not Explicit, Between-Subjects | Saudi Arabia | Journal | web.a.ebscohost.com |
| A14 | Rahimi & Miri, 2014 | 34 | University students | 16 sessions | Vocabulary | English | Longman Mobile Dictionary | Not Explicit, Between-Subjects | Iran | Grey | https://doi.org/10.1016/j.sbspro.2014.03.567 |
| A15 | Kondo et al., 2012 | 88 | University students | April – July semester | Listening & Reading | English | TOEIC | Not Explicit, Between-Subjects, Mixed Methods | Japan | Journal | https://doi.org/10.1017/S0958344012000055, |
| A16 | Wang, 2016 | 196 | University students | 1 semester | Reading | English | Learn English Audio & Video | Between-Subjects | Taiwan | Journal | https://doi.org/10.1080/10494820.2015.1131170 |
| A17 | Wang & Shih, 2015 | 93 | 19 | 1 semester | Vocabulary | English | The Most Important 2000 TOEIC Words | Not Explicit, Between-Subjects | Taiwan | Journal | https://doi.org/10.1504/IJMC.2015.070060 |
| A28 | Hwang, Chen, Shadiev, Huang & Chen, 2014 | 59 | 6thgrade | 3 classes per week for 1.5 months | Writing | English | Situated Writing System | Quasi-Experimental | Taiwan | Journal | https://doi.org/10.1080/09588221.2012.733711 |
| A19 | Shadiev, Hwang, Liu, 2018 | 53 | 13–14 | 3 weeks | Grammar, Writing, Reading | English | Mobile Multimedia Learning System (MMLS) | Quasi-Experimental | Taiwan | Journal | https://doi.org/10.1007/s11423-018-9590-1 |
| A20 | Wu, Sung, Huang, Yang, Yang, 2011 | 113 | University students | 7 weeks | Reading | English | Ubiquitous English-Reading Learning System | Not Explicit, Between-Subjects | Taiwan | Journal | www.jstor.org/stable/10.2307/jeductechsoci.14.4.164 |
| A21 | Hsieh, Wang, Su, Lee, 2012 | 60 | University students | 4 months | Reading & Vocabulary | English | Fuzzy, Logic-Based Personalized Learning System | Between-Subjects | Taiwan | Journal | www.jstor.org/stable/10.2307/jeductechsoci.15.1.273 |
| A22 | Zhang, 2016 | 120 | University students | 10 weeks | Listening Comprehension | English | Keke English & Easy IELTS | Between-Subjects | China | Grey | https://doi.org/10.12783/dtssehs/icaem2016/4290 |
| A23 | Lee, 2014 | 120 | 15–21 | 20 classes | Vocabulary | English | Teacher-Created Application | Not Explicit, Between-Subjects | Taiwan | Grey | https://doi.org/10.1109/MCSoC.2014.24 |
Figure 2
Risk of bias across studies.
Note: Summary table of the risk of bias in all included studies overall and across each of the five domains: overall bias (high risk), selection of the reported result (some concerns), measurement of the outcome (a mix of low bias and some concerns, but mostly high risk), missing outcome data (predominantly low risk), deviations from intended interventions (largely some concerns), randomization process (mostly high risk). The bias domain is seen on the y-axis, and the score out of 100 is illustrated on the x-axis.
Figure 3
Funnel plot of all included studies.
Figure 4
Forest plot of all studies and overall effects.
Note: Forest plot of all studies included in the meta-analysis. Individual effect sizes of all studies included in this meta-analysis with their respective weight in the analysis and confidence intervals represented by the triangles. Larger triangles indicate more weight in the random effects model. Red triangles denote the outliers identified in the analysis. The x-axis represents the standardized mean difference effect size (Hedges’ g) and the y-axis is each individual study with its 95% confidence interval. The overall effect sizes are represented by the red and blue circles at the bottom. The 95% PI denotes the 95% prediction intervals for the overall effect of all studies (blue) and for the overall effect with outliers removed (red), indexed by the thick red and blue lines.
Table 2
Summary of findings table.
| INTERVENTION (MALL-APPLICATION VS. CONTROL) | N | G | 95% CI | TAU2 | I2 | Q | 95% CI |
|---|---|---|---|---|---|---|---|
| Overall | 23 | 1.08 | 0.66–1.51 | 0.67 | 79.3% | 106.10*** | <0.0001 |
| Overall, outliers removed | 19 | 0.88 | 0.62–1.14 | 0.08 | 45% | 32.72* | 0.018 |
| INTERVENTION (MALL-APPLICATION VS. CONTROL) | N | G | 95% CI | QBETWEEN-GROUPS | P-VALUE | ||
| School Level | |||||||
| University | 14 | 0.87 | 0.51–1.23 | 0.16 | 56.3% | ||
| High School | 1 | 0.83 | 0.26–1.93 | NA | NA | 1.44 | 0.84 |
| Middle School | 2 | 1.13 | –4.39–6.66 | 0.18 | 39.1% | ||
| Elementary School | 1 | 0.37 | –0.59–1.33 | NA | NA | ||
| Learning focus | |||||||
| Vocabulary | 10 | 0.87 | 0.41–1.33 | 0.17 | 55.1% | ||
| Reading | 1 | 0.80 | 0.51–1.09 | NA | NA | ||
| Writing | 1 | 0.87 | 0.32–1.41 | NA | NA | 7.99 | 0.24 |
| Grammar | 1 | 1.43 | 0.88–1.99 | NA | NA | ||
| Communication | 1 | –0.10 | –1.48–1.28 | NA | NA | ||
| Listening | 1 | 1.34 | 0.60–2.08 | NA | NA | ||
| Mix of language skills | 4 | 0.65 | –0.38–1.67 | 0.06 | 28.6% | ||
| Intervention Duration | |||||||
| 1 day | 1 | 0.37 | –0.59–1.33 | NA | NA | ||
| 2–6 weeks | 6 | 0.98 | 0.27–1.70 | 0.17 | 38.6% | ||
| 7–10 weeks | 4 | 0.99 | 0.45–1.54 | 0.05 | 35.6% | 2.34 | 0.67 |
| 1 semester | 6 | 0.89 | 0.14–1.65 | 0.23 | 58.9% | ||
| 10–16 sessions (undefined weeks) | 2 | 0.46 | –5.97–6.89 | 0.35 | 67.8% | ||
| App Type | |||||||
| Pre-existing App | 10 | 0.98 | 0.68–1.28 | 0.05 | 33.8% | 0.63 | 0.43 |
| Developed App | 9 | 0.76 | 0.20–1.32 | 0.19 | 52.5% | ||
| Learning Principles | |||||||
| Retrieval Practice | 12 | 0.95 | 0.56–1.34 | 0.13 | 51.1% | 2.57 | 0.28 |
| No-Retrieval Practice | 6 | 0.73 | 0.35–1.11 | 0.006 | 28.1% | ||
| Feedback | 9 | 0.89 | 0.58–1.20 | 0.02 | 26.6% | 0.01 | 0.91 |
| No-Feedback | 10 | 0.86 | 0.37–1.35 | 0.22 | 58.7% | ||
| Multimodal | 14 | 0.87 | 0.53–1.22 | 0.12 | 51.8% | 0.03 | 0.87 |
| No-Multimodal | 5 | 0.91 | 0.33–1.49 | 0.07 | 23.1% | ||
[i] Note: “*” = p < 0.01, “**” = p < 0.001, “***” = p < 0.0001. Qbetween-groups is the denotes the Q statistic for between-groups comparison, NA denotes unavailable values due to small number of studies.