When Leaving is Persisting
Studying Patterns of Persistence in an Online Game-Based Learning Environment for Mathematics
DOI:
https://doi.org/10.18608/jla.2023.8219Keywords:
persistence, micro-persistence, game-based learning, log analysis, mathematics education, research paperAbstract
We report on a large-scale, log-based study of the associations between persistence and success in an online game-based learning environment for elementary school mathematics. While working with applets, learners can rerun a task after completing it or can halt before completing and rerun it again; both of these mechanisms may improve the score. We analyzed about 3.1 million applet runs by N=44,323 1st–6th-grade students to have a nuanced understanding of persistence patterns, by identifying sequences of consecutive single applet runs (SoCSARs). Overall, we analyzed 2,249,647 SoCSARs and identified six patterns, based on halting and rerunning tasks, and their completion: 1) Single Complete, 2) Single Incomplete, 3) Some Incomplete and Single Complete, 4) Multiple Incomplete and No Complete, 5) Multiple Complete and No Incomplete, and 6) Multiple Complete and Some Incomplete. Expectedly, we found a positive correlation between SoCSAR length and success. Some patterns demonstrate low to medium positive associations with success, while others demonstrate low to medium negative associations. Furthermore, the associations between the type of persistence and success vary by grade level. We discuss these complex relationships and suggest metacognitive and motivational factors that may explain why some patterns are productive and others are not.
References
Abdul Jabbar, A. I., & Felicia, P. (2015). Gameplay engagement and learning in game-based learning: A systemic review. Review of Educational Research, 85(4), 740–779. https://doi.org/10.3102/0034654315577210
Aker, L. B. (2016). A meta-analysis of middle school science engagement [Unpublished doctoral dissertation]. Seattle Pacific University. https://digitalcommons.spu.edu/soe_etd/10
Auvinen, T., Hakulinen, L., & Malmi, L. (2015). Increasing students’ awareness of their behavior in online learning environments with visualizations and achievement badges. IEEE Transactions on Learning Technologies, 8(3), 261–273. https://doi.org/10.1109/TLT.2015.2441718
Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., & Koedinger, K. (2008a). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185–224. https://www.learntechlib.org/p/24328
Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2004a). Detecting student misuse of intelligent tutoring systems. Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS 2004), 30 August–3 September 2004, Maceió, Alagoas, Brazil (pp. 531–540). Springer. https://doi.org/10.1007/978-3-540-30139-4_50
Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004b). Off-task behavior in the cognitive tutor classroom: When students “game the system.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ04), 24–29 April 2004, Vienna, Austria (pp. 383–390). ACM Press. https://doi.org/10.1145/985692.985741
Baker, R. S., Hershkovitz, A., Rossi, L. M., Goldstein, A. B., & Gowda, S. M. (2013). Predicting robust learning with the visual form of the moment-by-moment learning curve. Journal of the Learning Sciences, 22(4), 639–666. https://doi.org/10.1080/10508406.2013.836653
Baker, R. S. J. d., Corbett, A. T., Roll, I., & Koedinger, K. R. (2008b). Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction, 18(3), 287–314. https://doi.org/10.1007/s11257-007-9045-6
Basu, S., Rutstein, D., Xu, Y., & Shear, L. (2020). A principled approach to designing a computational thinking practices assessment for early grades. Proceedings of the 51st ACM Technical Symposium on Computer Science Education (SIGCSE ʼ20), 11–14 March 2020, Portland, OR, USA (pp. 912–918). ACM Press. https://doi.org/10.1145/3328778.3366849
Bećirović, S., & Remzija Hurić-Bećirović, H. (2017). The role of age in students’ motivation and achievement in learning English as second language. The Journal of Linguistic and Intercultural Education, 10(1), 23–35. https://doi.org/10.29302/jolie.2017.10.1.2
Ben-Zadok, G., Leiba, M., & Nachmias, R. (2010). Comparison of online learning behaviors in school vs. at home in terms of age and gender based on log file analysis. Interdisciplinary Journal of e-Skills and Lifelong Learning, 6, 305–322. https://doi.org/10.28945/1317
Buckley, P., & Doyle, E. (2016). Gamification and student motivation. Interactive Learning Environments, 24(6), 1162–1175. https://doi.org/10.1080/10494820.2014.964263
Chung, C.-Y., & Hsiao, I.-H. (2020). Investigating patterns of study persistence on self-assessment platform of programming problem-solving. Proceedings of the 51st ACM Technical Symposium on Computer Science Education (SIGCSE ʼ20), 11–14 March 2020, Portland, OR, USA (pp. 162–168). ACM Press. https://doi.org/10.1145/3328778.3366827
Cocea, M., Hershkovitz, A., & Baker, R. S. J. d. (2009). The impact of off-task and gaming behaviors on learning: Immediate or aggregate? In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Proceedings of the 14th International Conference on Artificial Intelligence in Education (AIED ʼ09) 6–10 July 2009, Brighton, UK (pp. 507–514). IOS Press. https://doi.org/10.3233/978–1–60750–028–5–507
Cooley, E., Beaird, J., & Ayres, R. (1994). Academic persistence and attributional style in fifth graders. Psychology in the Schools, 31(2), 156–163. https://doi.org/10.1002/1520-6807(199404)31:2%3c156::AID-PITS2310310210%3e3.0.CO;2-S
Dalipi, F., Imran, A. S., & Kastrati, Z. (2018). MOOC dropout prediction using machine learning techniques: Review and research challenges. Proceedings of the 2018 IEEE Global Engineering Education Conference (EDUCON 2018), 17–20 April 2018, Santa Cruz de Tenerife, Spain (pp. 1007–1014). IEEE Computer Society. https://doi.org/10.1109/EDUCON.2018.8363340
del Valle, R., & Duffy, T. M. (2009). Online learning: Learner characteristics and their approaches to managing learning. Instructional Science, 37(2), 129–149. https://doi.org/10.1007/s11251-007-9039-0
Delen, E., & Liew, J. (2016). The use of interactive environments to promote self-regulation in online learning: A literature review. European Journal of Contemporary Education, 15(1), 24–33. https://doi.org/10.13187/ejced.2016.15.24
Fang, Y., Xu, Y. J., Nye, B., Graesser, A., Pavlik, P., & Hu, X. (2017). Online learning persistence and academic achievement. In X. Hu, T. Barnes, A. Hershkovitz, & L. Paquette (Eds.), Proceedings of the 10th International Conference on Educational Data Mining (EDM2017), 25–28 June 2017, Wuhan, China (pp. 312–317). International Educational Data Mining Society. http://educationaldatamining.org/EDM2017/proc_files/papers/paper_114.pdf
Hart, C. (2012). Factors associated with student persistence in an online program of study: A review of the literature. Journal of Interactive Online Learning, 11(1), 19–42. https://www.ncolr.org/jiol/issues/pdf/11.1.2.pdf
Hershkovitz, A., Tabach, M., & Cohen, A. (2021). Online activity and achievements in elementary school mathematics: A large-scale exploration. Journal of Educational Computing Research, 60(1), 258–278. https://doi.org/10.1177/07356331211027822
Higgins, K., Huscroft-D’Angelo, J., & Crawford, L. (2019). Effects of technology in mathematics on achievement, motivation, and attitude: A meta-analysis. Journal of Educational Computing Research, 57(2), 283–319. https://doi.org/10.1177/0735633117748416
Israel-Fishelson, R., & Hershkovitz, A. (2020). Shooting for the stars: Micro-persistence of students in game-based learning environments. In D. Glick, A. Cohen, & C. Chang (Eds.), Early warning systems and targeted interventions for student success in online courses (pp. 239–258). IGI Global. https://doi.org/10.4018/978-1-7998-5074-8.ch012
Israel-Fishelson, R., & Hershkovitz, A. (2021). Micro-persistence and difficulty in a game-based learning environment for computational thinking acquisition. Journal of Computer Assisted Learning, 37(3), 839–850. https://doi.org/10.1111/jcal.12527
Jung, Y., & Lee, J. (2018). Learning engagement and persistence in massive open online courses (MOOCs). Computers & Education, 122, 9–22. https://doi.org/10.1016/J.COMPEDU.2018.02.013
Kovanović, V., Gašević, D., Dawson, S., Joksimović, S., & Baker, R. (2016). Does time-on-task estimation matter? Implications on validity of learning analytics findings. Journal of Learning Analytics, 2(3), 81–110. https://doi.org/10.18608/jla.2015.23.6
Langlands, E., & Moralles-Trujillo, M. (2023). Learning from errors: An empirical study on the impact of gamification on SQL query formulation. Proceedings of the 2023 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE ’20), 7–12 July 2023, Turku, Finland (pp. 341–347). ACM Press. https://doi.org/10.1145/3587102.3588821
Malkiewich, L. J., Lee, A., Slater, S., Xing, C., & Chase, C. C. (2016). No lives left: How common game features can undermine persistence, challenge-seeking, and learning to program. In C.-K. Looi, J. L. Polman, U. Cress, & P. Reimann (Eds.), Transforming Learning, Empowering Learners: Proceedings of the 12th International Conference of the Learning Sciences (ICLS ’16), 20–24 June 2016, Singapore (2 volumes, pp. 186–193). International Society of the Learning Sciences. https://repository.isls.org/bitstream/1/115/1/26.pdf
Metcalfe, J. (2017). Learning from errors. Annual Review of Psychology, 68, 465–489. https://doi.org/10.1146/annurev-psych-010416-044022
Motz, B., Quick, J., Schroeder, N., Zook, J., & Gunkel, M. (2019). The validity and utility of activity logs as a measure of student engagement. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (pp. 300–309). ACM Press. https://doi.org/10.1145/3303772.3303789
Multon, K. D., Brown, S. D., & Lent, R. W. (1991). Relation of self-efficacy beliefs to academic outcomes: A meta-analytic investigation. Journal of Counseling Psychology, 38(1), 30–38. https://doi.org/10.1037/0022-0167.38.1.30
Nguyen, Q. (2020). Rethinking time-on-task estimation with outlier detection accounting for individual, time, and task differences. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 376–381). ACM Press. https://doi.org/10.1145/3375462.3375538
Richardson, J. C., & Newby, T. (2006). The role of students’ cognitive engagement in online learning. International Journal of Distance Education, 20(1), 23–37. https://doi.org/10.1207/s15389286ajde2001_3
Schwabe, F., McElvany, N., & Trendtel, M. (2015). The school age gender gap in reading achievement: Examining the influences of item format and intrinsic reading motivation. Reading Research Quarterly, 50(2), 219–232. https://doi.org/10.1002/RRQ.92
Sümer, M., & Aydın, C. H. (2018). Gamification in open and distance learning: A systematic review. In J. M. Spector, B. B. Lockee, & M. D. Childress (Eds.), Learning, design, and technology: An international compendium of theory, research, practice, and policy (pp. 1–16). Springer. https://doi.org/10.1007/978-3-319-17727-4_115-1
Sun, L., Kangas, M., & Ruokamo, H. (2023). Game-based features in intelligent game-based learning environments: A systematic literature review. Interactive Learning Environments. Advance online publication. https://doi.org/10.1080/10494820.2023.2179638
Tran, Y. (2019). Computational thinking equity in elementary classrooms: What third-grade students know and can do. Journal of Educational Computing Research, 57(1), 3–31. https://doi.org/10.1177/0735633117743918
Vollmeyer, R., & Rheinberg, F. (2000). Does motivation affect performance via persistence? Learning and Instruction, 10(4), 293–309. https://doi.org/10.1016/S0959-4752(99)00031-6
Vu, T., Magis-Weinberg, L., Jansen, B. R. J., van Atteveldt, N., Janssen, T. W. P., Lee, N. C., van der Maas, H. L. J., Raijmakers, M. E. J., Sachisthal, M. S. M., & Meeter, M. (2022). Motivation-achievement cycles in learning: A literature review and research agenda. Educational Psychology Review, 34(1), 39–71. https://doi.org/10.1007/s10648-021-09616-7
Zapata-Cáceres, M., & Martín-Barroso, E. (2021). Applying game learning analytics to a voluntary video game: Intrinsic motivation, persistence, and rewards in learning to program at an early age. IEEE Access, 9, 123588–123602. https://doi.org/10.1109/ACCESS.2021.3110475
Zhang, Q., & Yu, Z. (2022). Meta-analysis on investigating and comparing the effects on learning achievement and motivation for gamification and game-based learning. Education Research International, 2022, 1519880. https://doi.org/10.1155/2022/1519880
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
