Understanding Student Navigation Patterns in Game-Based Learning

Min Liu; Ying Cai; Songhee  Han; Peixia  Shao

doi:10.18608/jla.2022.7637

Authors

Min Liu The University of Texas at Austin https://orcid.org/0000-0003-0735-1148
Ying Cai The University of Texas at Austin https://orcid.org/0000-0002-3611-630X
Songhee Han The University of Texas at Austin https://orcid.org/0000-0002-8984-7560
Peixia Shao The University of Texas at Austin

DOI:

https://doi.org/10.18608/jla.2022.7637

Keywords:

learning analytics, game-based learning, log data, middle school students, science learning, research paper

Abstract

Research on learning analytics (LA) has focused mostly at the university level. LA research in the K–12 setting is needed. This study aimed to understand 4,115 middle school students’ learning paths based on their behavioural patterns and the relationship with performance levels when they used a digital learning game as their science curriculum. The findings showed significant positive relationships between various tool uses and performance measures and varied tool use patterns at different problem-solving phases by high- and low-performing students. The results indicated that students who used tools appropriately and wisely, given the phase they were at, were more likely to succeed. The findings offered an insightful glimpse of learners’ navigation patterns in relation to their performance and provided much-needed empirical evidence to support using analytics for game-based learning in K–12 education. The findings also revealed that log data cannot explain all learners’ actions. Implications for both research and practice are discussed.

References

Alonso-Fernández, C., Calvo-Morata, A., Freire, M., Martinez-Ortiz, I., & Fernández-Manjón, B. (2019). Applications of data science to game learning analytics data: A systematic literature review. Computers & Education, 141. https://doi.org/10.1016/j.compedu.2019.103612

Baker, R. S., & Siemens, G. (2014). Educational data mining and learning analytics. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences, 2nd ed. (pp. 253–274). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.016

Bastian, M., Heymann, S., & Jacomy, M. (2009). Gephi: An open source software for exploring and manipulating networks. Proceedings of the 3rd International AAAI Conference on Weblogs and Social Media (ICWSM ’09) 17–20 May 2009, San Jose, CA, USA (pp. 361–362). AAAI Press. https://ojs.aaai.org/index.php/ICWSM/article/view/13937/13786

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x

Berland, M., Martin, T., Benton, T., Petrick Smith, C., & Davis, D. (2013). Using learning analytics to understand the learning pathways of novice programmers. Journal of the Learning Sciences, 22(4), 564–599. https://doi.org/10.1080/10508406.2013.836655

Blikstein, P., Worsley, M., Piech, C., Sahami, M., Cooper, S., & Koller, D. (2014). Programming pluralism: Using learning analytics to detect patterns in the learning of computer programming. Journal of the Learning Sciences, 23(4), 561–599. https://doi.org/10.1080/10508406.2014.954750

Boyle, E. A., MacArthur, E. W., Connolly, T. M., Hainey, T., Manea, M., Kärki, A., & Van Rosmalen, P. (2014). A narrative literature review of games, animations and simulations to teach research methods and statistics. Computers & Education, 74, 1–14. https://doi.org/10.1016/j.compedu.2014.01.004

Chaudy, Y., Connolly, T., & Hainey, T. (2014). Learning analytics in serious games: A review of the literature. Proceedings of European Conference in the Applications of Enabling Technologies (ECAET 2014), 20–21 November 2014, Glasgow, UK. https://research-portal.uws.ac.uk/en/publications/learning-analytics-in-serious-games-a-review-of-the-literature

Cheng, M. T., Rosenheck, L., Lin, C. Y., & Klopfer, E. (2017). Analyzing gameplay data to inform feedback loops in the Radix Endeavor. Computers & Education, 111, 60–73. https://doi.org/10.1016/j.compedu.2017.03.015

Cheng, M. T., Lin, Y. W., & She, H. C. (2015). Learning through playing Virtual Age: Exploring the interactions among student concept learning, gaming performance, in-game behaviors, and the use of in-game characters. Computers & Education, 86, 18–29. https://doi.org/10.1016/j.compedu.2015.03.007

Choi, Y., Na, Y., Yoon, Y., Shin, J., Bae, C., Suh, H., Kim, B., & Heo, J. (2020). Choose your own question: Encouraging self-personalization in learning path construction. https://doi.org/10.48550/arXiv.2005.03818

Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. S. (2016). Digital games, design, and learning: A systematic review and meta-analysis. Review of Educational Research, 86(1), 79–122. https://doi.org/10.3102/0034654315582065

Dabney, A., & Storey, J. D. (2004). QVALUE: The Manual Version 1.0. University of Washington, Department of Biostatistics. http://www.bioconductor.org/packages/2.13/bioc/vignettes/qvalue/inst/doc/manual.pdf

Diedenhofen, B., & Musch, J. (2015). cocor: A comprehensive solution for the statistical comparison of correlations. PloS One, 10(4), e0121945. https://doi.org/10.1371/journal.pone.0121945

Durand, G., Belacel, N., & LaPlante, F. (2013). Graph theory based model for learning path recommendation. Information Sciences, 251, 10–21. https://doi.org/10.1016/j.ins.2013.04.017

Feng, X., & Yamada, M. (2021). An analytical approach for detecting and explaining the learning path patterns of an informal learning game. Educational Technology & Society, 24(1), 176–190. https://www.jstor.org/stable/26977866

Freire, M., Serrano-Laguna, Á., Manero, B., Martínez-Ortiz, I., Moreno-Ger, P., Fernández-Manjón, B. (2016). Game learning analytics: Learning analytics for serious games. In M. Spector, B. Lockee, & M. Childress (Eds.), Learning, design, and technology (pp. 1–29). Springer Nature Switzerland AG. https://doi.org/10.1007/978-3-319-17727-4_21-1

Fu, J., Zapata, D., & Mavronikolas, E. (2014). Statistical methods for assessments in simulations and serious games. ETS Research Report Series, 2014(2), 1–17. https://doi.org/10.1002/ets2.12011

Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441–467. https://doi.org/10.1177/1046878102238607

Gauthier, A., Corrin, M., & Jenkinson, J. (2015). Exploring the influence of game design on learning and voluntary use in an online vascular anatomy study aid. Computers & Education, 87, 24–34. https://doi.org/10.1016/j.compedu.2015.03.017

Gee, J. P. (2008). Learning and games. In K. Salen (Ed.), The ecology of games: Connecting youth, games, and learning (pp. 21–40). MIT Press. https://doi.org/10.1080/13691180802552890

Gomez, M. J., Ruipérez-Valiente, J. A., Martínez, P. A., & Kim, Y. J. (2021). Applying learning analytics to detect sequences of actions and common errors in a geometry game. Sensors, 21(4), Article 1025. https://doi.org/10.3390/s21041025

Gomez, M. J., Ruipérez-Valiente, J. A., Martinez, P. A., & Kim, Y. J. (2020). Exploring the affordances of sequence mining in educational games. Proceedings of the 8th International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM ’20), 21–23 October, Salamanca, Spain (pp. 648–654). ACM Press. https://doi.org/10.1145/3434780.3436562

Gursoy, M. E., Inan, A., Nergiz, M. E., & Saygin, Y. (2017). Privacy-preserving learning analytics: Challenges and techniques. IEEE Transactions on Learning Technologies, 10(1), 68–81. https://doi.org/10.1109/TLT.2016.2607747

Hauge, J. B., Boyle, E., Mayer, I., Nadolski, R., Riedel, J. C., Moreno-Ger, P., Bellotti, F., Lim, T., & Ritchie, J. (2014). Study design and data gathering guide for serious games’ evaluation. In T. Connolly, T. Hainey, E. Boyle, G. Baxter, & P. Moreno-Ger (Eds.), Psychology, pedagogy, and assessment in serious games (pp. 394–419). IGI Global. http://doi:10.4018/978-1-4666-4773-2.ch018

Hittner, J. B., May, K., & Silver, N. C. (2003). A Monte Carlo evaluation of tests for comparing dependent correlations. The Journal of General Psychology, 130(2), 149–168. https://doi.org/10.1080/00221300309601282

Horn, B., Hoover, A. K., Barnes, J., Folajimi, Y., Smith, G., & Harteveld, C. (2016). Opening the black box of play: Strategy analysis of an educational game. Proceedings of the Annual Symposium on Computer–Human Interaction in Play (CHI PLAY '16), 16–19 October 2016, Austin, Texas, USA (pp. 142–153). ACM Press. https://doi.org/10.1145/2967934.2968109

Hwang, G. J., Chu, H. C., & Yin, C. (2017). Objectives, methodologies and research issues of learning analytics. Interactive Learning Environments, 25(2), 143–146. https://doi.org/10.1080/10494820.2017.1287338

Ifenthaler, D. (2015). Learning analytics. In J. M. Spector (Ed.), The SAGE encyclopedia of educational technology (Vol. 2, pp. 447–451). Sage.

Ifenthaler, D. (2017). Designing effective digital learning environments: Toward learning analytics design. Technology, Knowledge and Learning, 22(3), 401–404. https://doi.org/10.1007/s10758-017-9333-0

Ifenthaler, D., Mah, D. K., & Yau, Y. K. (2019). Utilising learning analytics for study success: Reflections on current empirical findings. In D. Ifenthaler, D. K. Mah, & Y. K. Yau (Eds.), Utilizing learning analytics to support study success (pp. 27–36). Springer. https://doi.org/10.1007/978-3-319-64792-0_2

Karumbaiah, S., Baker, R. S., Barany, A., & Shute, V. (2019). Using epistemic networks with automated codes to understand why players quit levels in a learning game. In B. Eagan, M. Misfeldt, & A. Siebert-Evenstone (Eds.), Advances in quantitative ethnography (Vol. 1112, pp. 106–116). Springer. https://doi.org/10.1007/978-3-030-33232-7_9

Kang, J., Liu, M., & Qu, W. (2017). Using gameplay data to examine learning behavior patterns in a serious game. Computers in Human Behavior, 72, 757–770. https:// doi.org/10.1016/j.chb.2016.09.062

Kang, J., Liu, S., & Liu, M. (2017). Tracking students’ activities in serious games. In F. Q. Lai., & J. D. Lehman (Eds.). Learning and knowledge analytics in open education (pp. 125–137). Springer. https://doi.org/10.1007/978-3-319-38956-1_10

Kim, Y. J., & Ifenthaler, D. (2019). Game-based assessment: The past ten years and moving forward. In D. Ifenthaler & Y. Kim (Eds.), Game-based assessment revisited: Advances in game-based learning (pp. 3–11). Springer. https://doi.org/10.1007/978-3-030-15569-8_1

Kopeinik, S., Nussbaumer, A., Bedek, M., & Albert, D. (2012). Using CbKST for learning path recommendation in game-based learning. Proceedings of the 20th International Conference on Computers in Education (ICCE 2012), 26–30 November 2012, Singapore (pp. 26–30). Asia-Pacific Society for Computers in Education. https://www.researchgate.net/publication/274256220_Using_CbKST_for_Learning_Path_Recommendation_in_Game-based_Learning

Krath, J., Schürmann, L., & von Korflesch, H. F. O. (2021). Revealing the theoretical basis of gamification: A systematic review and analysis of theory in research on gamification, serious games and game-based learning. Computers in Human Behavior, 125, Article 106963. https://doi.org/10.1016/ j.chb.2021.106963

L’Heureux, A., Grolinger, K., Elyamany, H. F., & Capretz, M. A. M. (2017). Machine learning with big data: Challenges and approaches. IEEE Access, 5, 7776–7797. https://doi.org/10.1109/ACCESS.2017.2696365

Law, E. L. C., & Lárusdóttir, M. K. (2015). Whose experience do we care about? Analysis of the fitness of scrum and kanban to user experience. International Journal of Human–Computer Interaction, 31(9), 584–602. https://doi.org/10.1080/10447318.2015.1065693

Lee, J., & Liu, M. (2017). Design of fantasy and their effect on learning and engagement. In R. Z. Zheng & M. K. Gardner (Eds.), Handbook of research on serious games for educational applications (pp. 200–219). IGI Global. https://doi.org/10.4018/978-1-5225-0513-6.ch009

Li, F. Y., Hwang, G. J., Chen, P. Y., & Lin, Y. J. (2021). Effects of a concept mapping-based two-tier test strategy on students’ digital game-based learning performances and behavioural patterns. Computers & Education, 173, 104293. https://doi.org/10.1016/j.compedu.2021.104293

Linek, S., Öttl, G., & Albert, D. (2010). Non-invasive data tracking in educational games: Combination of logfiles and natural language processing. In L. Gómez Chova, D. Martí Belenguer, & I. Candel Torres (Eds.), Proceedings of the 4th International Technology, Education and Development Conference (INTED2010), 8–10 March 2010, Valencia, Spain (pp. 148–149). International Association of Technology, Education and Development. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.972.1032&rep=rep1&type=pdf

Liu, M., & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning environment. Educational Technology Research and Development, 53(1), 5–21. https://doi.org/10.1007/BF02504854

Liu, M., Horton, L., Corliss, S. B., Svinicki, M. D., Bogard, T., Kim, J., & Chang, M. (2009). Students’ problem solving as mediated by their cognitive tool use: A study of tool use patterns. Journal of Educational Computing Research, 40(1), 111–139. https://doi.org/10.2190/EC.40.1.e

Liu, M., Li, C., & Pan, Z. (2022). Creating an interactive dashboard to support middle school teachers’ implementation of a technology-supported problem-based learning program. International Journal of Designs for Learning, 13(1). 1–18. https://doi.org/10.14434/ijdl.v13i1.31243

Liu, M., Horton, L., Olmanson, J., & Toprac, P. (2011). A study of learning and motivation in a new media enriched environment for middle school science. Educational Technology Research and Development, 59(2), 249–266. https://doi.org/10.1007/s11423-011-9192-7

Liu, M., Kang, J., Liu, S., Zou, W., & Hodson, J. (2017). Learning analytics as an assessment tool in serious games: A review of literature. In M. Ma & A. Oikonomou (Eds.), Serious games and edutainment applications. Springer. https://doi.org/10.1007/978-3-319-51645-5_24

Liu, M., Lee, J., Kang, J., & Liu, S. (2016). What we can learn from the data: A multiple-case study examining behavior patterns by students with different characteristics in using a serious game. Technology, Knowledge and Learning, 21(1), 33–57. https://doi.org/10.1007/s10758-015-9263-7

Liu, M., Li, C., Pan, Z., & Pan, X. (2019). Mining big data to help make informed decisions for designing effective digital educational games. Interactive Learning Environments, 1–21. https://doi:10.1080/10494820.2019.1639061

Loh, C. S., Sheng, Y., & Ifenthaler, D. (2015). Serious games analytics: Theoretical framework. In C. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Advances in game-based learning (pp. 3–29). Springer. https://doi.org/10.1007/978-3-319-05834-4_1

Loh, C. S., & Sheng, Y. (2014). Maximum similarity index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games. Computers in Human Behavior, 39, 322–330. https://doi.org/10.1016/j.chb.2014.07.022

Loh, C. S., & Sheng, Y. (2015). Measuring the (dis-)similarity between expert and novice behaviors as serious games analytics. Education and Information Technologies, 20(1), 5–19. https://doi.org/10.1007/s10639-013-9263-y

Malone, T. W., & Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, learning and instruction (vol. 3, pp. 223–253). Lawrence Erlbaum. https://ocw.metu.edu.tr/pluginfile.php/2340/mod_resource/content/0/ceit706/week3/MakingLearningFun-ATaxonomyOfIntrinsicMotivationsForLearning.pdf

Mao, L., Liu, O. L., Roohr, K., Belur, V., Mulholland, M., Lee, H. S., & Pallant, A. (2018). Validation of automated scoring for a formative assessment that employs scientific argumentation. Educational Assessment, 23(2), 121–138. https://doi.org/10.1080/10627197.2018.1427570

Martin, T., Aghababyan, A., Pfaffman, J., Olsen, J., Baker, S., Janisiewicz, P., Phillips, R., & Smith, C. P. (2013). Nanogenetic learning analytics: Illuminating student learning pathways in an online fraction game. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 165–169). ACM Press. https://doi.org/10.1145/2460296.2460328

Mislevy, R. J. (2011). Evidence-centered design for simulation-based assessment (No. 800). National Center for Research on Evaluation, Standards, and Student Testing (CRESST). https://files.eric.ed.gov/fulltext/ED522835.pdf

Oren, M., Pedersen, S., & Butler-Purry, K. L. (2021). Teaching digital circuit design with a 3-D video game: The impact of using in-game tools on students’ performance. IEEE Transactions on Education, 64(1), 24–31. https://doi.org/10.1109/TE.2020.3000955

Owen, V. E., Roy, M.-H., Thai, K. P., Burnett, V., Jacobs, D., Keylor, E., & Baker, R. S. (2019). Detecting wheel-spinning and productive persistence in educational games. In C. F. Lynch, A. Merceron, M. Desmarais, & R. Nkambou (Eds.), Proceedings of the 12th International Conference on Educational Data Mining (EDM2019), 2–5 July 2019, Montréal, Quebec, Canada (pp. 378–383). International Educational Data Mining Society. https://files.eric.ed.gov/fulltext/ED599202.pdf

Park, J., Kim, S., Kim, A., & Mun, Y. Y. (2019). Learning to be better at the game: Performance vs. completion contingent reward for game-based learning. Computers & Education, 139, 1–15. https://doi.org/10.1016/j.compedu.2019.04.016

Qian, M., & Clark, K. R. (2016). Game-based learning and 21st century skills: A review of recent research. Computers in Human Behavior, 63, 50–58. https://doi.org/10.1016/j.chb.2016.05.023

Rapp, A. (2017). Designing interactive systems through a game lens: An ethnographic approach. Computers in Human Behavior, 71, 455–468. https://doi.org/10.1016/j.chb.2015.02.048

Rosenheck, L., Cheng, M. T., Lin, C. Y., & Klopfer, E. (2021). Approaches to illuminate content-specific gameplay decisions using open-ended game data. Educational Technology Research and Development, 69(2), 1135–1154. https://doi.org/10.1007/s11423-021-09989-0

Rowe, E., Asbell-Clarke, J., Baker, R. S., Eagle, M., Hicks, A. G., Barnes, T. M., Brown, R. A., & Edwards, T. (2017). Assessing implicit science learning in digital games. Computers in Human Behavior, 76, 617–630. https://doi.org/10.1016/j.chb.2017.03.043

Ruipérez-Valiente, J. A., Rosenheck, L., & Kim, Y. J. (2019). What does exploration look like? Painting a picture of learning pathways using learning analytics. In D. Ifenthaler & Y. Kim (Eds.), Game-based assessment revisited: Advances in game-based learning (pp. 281–300). Springer. https://doi.org/10.1007/978-3-030-15569-8_14

Seng, W. Y., & Yatim, M. H. M. (2014). Computer game as learning and teaching tool for object oriented programming in higher education institution. Procedia-Social and Behavioral Sciences, 123, 215–224. https://doi.org/10.1016/j.sbspro.2014.01.1417

Shou, Z., Lu, X., Wu, Z., Yuan, H., Zhang, H., & Lai, J. (2020). On learning path planning algorithm based on collaborative analysis of learning behavior. IEEE Access, 8, 119863–119879. https://doi.org/10.1109/ACCESS.2020.3005793

Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. https://doi.org/10.1177/0002764213498851

Snow, E. L., Allen, L. K., Jacovina, M. E., & McNamara, D. S. (2015). Does agency matter? Exploring the impact of controlled behaviors within a game-based environment. Computers & Education, 82, 378–392. https://doi.org/10.1016/j.compedu.2014.12.011

Squire, K. D. (2008). Video game–based learning: An emerging paradigm for instruction. Performance Improvement Quarterly, 21(2), 7–36. https://doi.org/10.1002/piq.20020

Su, C. (2017). Designing and developing a novel hybrid adaptive learning path recommendation system (ALPRS) for gamification mathematics geometry course. Eurasia Journal of Mathematics, Science and Technology Education, 13(6), 2275–2298. https://doi.org/10.12973/eurasia.2017.01225a

Sun, Z., Lin, C. H., Lv, K., & Song, J. (2021). Knowledge-construction behaviors in a mobile learning environment: A lag-sequential analysis of group differences. Educational Technology Research and Development, 69(2), 533–551. https://doi.org/10.1007/s11423-021-09938-x

Sutcliffe, A., & Hart, J. (2017). Analyzing the role of interactivity in user experience. International Journal of Human–Computer Interaction, 33(3), 229–240. https://doi.org/10.1080/10447318.2016.1239797

van Leeuwen, A., Knoop-van Campen, C. A. N., Molenaar, I., & Rummel, N. (2021). How teacher characteristics relate to how teachers use dashboards: Results from two case studies in K–12. Journal of Learning Analytics, 8(2), 6–21. https://doi.org/10.18608/jla.2021.7325

Vista, A., Awwal, N., & Care, E. (2016). Sequential actions as markers of behavioural and cognitive processes: Extracting empirical pathways from data streams of complex tasks. Computers & Education, 92, 15–36. https://doi.org/10.1016/j.compedu.2015.10.009

Williams, J., & Rosenbaum, S. (2004). Learning paths: Increase profits by reducing the time it takes employees to get up-to-speed. John Wiley & Sons.

Wilson, K. A., Bedwell, W. L., Lazzara, E. H., Salas, E., Burke, C. S., Estock, J. L., & Conkey, C. (2009). Relationships between game attributes and learning outcomes: Review and research proposals. Simulation & Gaming, 40(2), 217–266. https://doi.org/10.1177/1046878108321866

Wrzesien, M., & Raya, M. A. (2010). Learning in serious virtual worlds: Evaluation of learning effectiveness and appeal to students in the E-Junior project. Computers & Education, 55(1), 178–187. https://doi.org/10.1016/j.compedu.2010.01.003

Zhu, H., Tian, F., Wu, K., Shah, N., Chen, Y., Ni, Y., Zhang, X., Chao, K., & Zheng, Q. (2018). A multi-constraint learning path recommendation algorithm based on knowledge map. Knowledge-Based Systems, 143, 102–114. https://doi.org/10.1016/j.knosys.2017.12.011

Understanding Student Navigation Patterns in Game-Based Learning

Authors

DOI:

Keywords:

Abstract

References

Downloads

Published

How to Cite

Issue

Section

License