Serious Game Analytics by Design: Feature Generation and Selection Using Game Telemetry and Game Metrics
Toward Predictive Model Construction
DOI:
https://doi.org/10.18608/jla.2023.7681Keywords:
serious games, learning analytics, game-based learning environment, logging system, trace data, educational data mining, performance measurement, research paperAbstract
The construction of prediction models reflecting players’ learning performance in serious games currently faces various challenges for learning analytics. In this study, we design, implement, and field test a learning analytics system for a serious game, advancing the field by explicitly showing which in-game features correspond to differences in learner performance. We then deploy and test a system that provides instructors with clear signals regarding student learning and progress in the game, which instructors could depend upon for interventions. Within the study, we examined, coded, and filtered a substantial gameplay corpus, determining expertise in the game. Mission HydroSci (MHS) is a serious game that teaches middle-school students water science. Using our logging system, designed and implemented along with game design and development, we captured around 60 in-game features from the gameplay of 373 students who completed Unit 3 of MHS in its first field test. We tested eight hypotheses during the field test and presented this paper’s results to participating teachers. Our findings reveal several features with statistical significance that will be critical for creating a validated prediction model. We discuss how this work will help future research establish a framework for designing analytics systems for serious games and advancing gaming design and analytics theory.
References
All, A., Nuñez Castellar, E. P., & Van Looy, J. (2015). Towards a conceptual framework for assessing the effectiveness of digital game-based learning. Computers & Education, 88, 29–37. https://doi.org/10.1016/j.compedu.2015.04.012
Alvarez, J., Plantec, J.-Y., Vermeulen, M., & Kolski, C. (2017). RDU Model dedicated to evaluate needed counsels for Serious Game projects. Computers & Education, 114, 38–56. https://doi.org/10.1016/j.compedu.2017.06.007
Barab, S. A., Gresalfi, M., & Ingram-Goble, A. (2010). Transformational play: Using games to position person, content, and context. Educational Researcher, 39(7), 525–536. https://doi.org/10.3102/0013189X10386593
Barzilai, S., & Blau, I. (2014). Scaffolding game-based learning: Impact on learning achievements, perceived learning, and game experiences. Computers & Education, 70, 65–79. https://doi.org/10.1016/j.compedu.2013.08.003
Boyle, E. A., Hainey, T., Connolly, T. M., Gray, G., Earp, J., Ott, M., Lim, T., Ninaus, M., Ribeiro, C., & Pereira, J. (2016). An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Computers & Education, 94, 178–192. https://doi.org/10.1016/j.compedu.2015.11.003
Buckingham Shum, S., & Deakin Crick, R. (2016). Learning analytics for 21st century competencies. Journal of Learning Analytics, 3(2), 6–21. https://doi.org/10.18608/jla.2016.32.2
Caballero-Hernández, J. A., Palomo-Duarte, M., & Dodero, J. M. (2017). Skill assessment in learning experiences based on serious games: A systematic mapping study. Computers & Education, 113, 42–60. https://doi.org/10.1016/j.compedu.2017.05.008
Carvalho, M. B., Bellotti, F., Berta, R., De Gloria, A., Sedano, C. I., Hauge, J. B., Hu, J., & Rauterberg, M. (2015). An activity theory-based model for serious games analysis and conceptual design. Computers & Education, 87, 166–181. https://doi.org/10.1016/j.compedu.2015.03.023
Chaves, R. O., von Wangenheim, C. G., Furtado, J. C. C., Oliveira, S. R. B., Santos, A., & Favero, E. L. (2015). Experimental evaluation of a serious game for teaching software process modeling. IEEE Transactions on Education, 58(4), 289–296. https://doi.org/10.1109/TE.2015.2411573
Chittaro, L., & Buttussi, F. (2015). Assessing knowledge retention of an immersive serious game vs. a traditional education method in aviation safety. IEEE Transactions on Visualization and Computer Graphics, 21(4), 529–538. https://doi.org/10.1109/TVCG.2015.2391853
Clarke-Midura, J., Rosenheck, L., & Groff, J. (2015). Designing games for learning and assessment: The radix endeavor. In Proceedings of the 14th International Conference on Interaction Design and Children (IDC 2015), 21–24 June 2015, Boston, MA, USA (pp. 343–346). ACM. https://doi.org/10.1145/2771839.2771912
Cutumisu, M., Blair, K. P., Chin, D. B., & Schwartz, D. L. (2015). Posterlet: A game-based assessment of children’s choices to seek feedback and to revise. Journal of Learning Analytics, 2(1), 49–71. https://doi.org/10.18608/jla.2015.21.4
Fernández, C. A. (2016). Gaming learning analytics for serious games [eprint], Universidad Complutense Madrid. https://eprints.ucm.es/38711
Fridenson-Hayo, S., Berggren, S., Lassalle, A., Tal, S., Pigat, D., Meir-Goren, N., O’Reilly, H., Ben-Zur, S., Bölte, S., Baron-Cohen, S., & Golan, O. (2017). “Emotiplay”: A serious game for learning about emotions in children with autism: Results of a cross-cultural evaluation. European Child & Adolescent Psychiatry, 26(8), 979–992. https://doi.org/10.1007/s00787-017-0968-0
Fulmore, Y. (2015). Video games and the customization of learning: Interactive narratives as a promising design framework for crafting inclusive educational environments. In Proceedings of the Third Conference on GenderIT (GenderIT 2015), 24 April 2015, Philadelphia, PA, USA (pp. 32–39). ACM. https://doi.org/10.1145/2807565.2807710
Gauthier, A., & Jenkinson, J. (2018). Designing productively negative experiences with serious game mechanics: Qualitative analysis of game-play and game design in a randomized trial. Computers & Education, 127, 66–89. https://doi.org/10.1016/j.compedu.2018.08.017
Goggins, S. P., Gallagher, M., Laffey, J., & Amelung, C. (2010). Social intelligence in completely online groups—Toward social prosthetics from log data analysis and transformation. In IEEE Second International Conference on Social Computing (SocialCom 2010), 20–22 August 2010, Minneapolis, MN, USA (pp. 500–507). IEEE. https://doi.org/10.1109/SocialCom.2010.79
Goggins, S. P., Galyen, K., & Laffey, J. (2010). Network analysis of trace data for the support of group work: Activity patterns in a completely online course. In Proceedings of the 16th ACM International Conference on Supporting Group Work (GROUP 2010), 7–10 November 2010, Sanibel Island, FL, USA (pp. 107–116). https://doi.org/10.1145/1880071.1880089
Goggins, S., Galyen, K., Petakovic, E., & Laffey, J. (2016). Connecting performance to social structure and pedagogy as a pathway to scaling learning analytics in MOOCs: An exploratory study. Journal of Computer Assisted Learning, 32(3), 244–266. https://doi.org/10.1111/jcal.12129
Graves, I., McDonald, N., & Goggins, S. P. (2016). Sifting signal from noise: A new perspective on the meaning of tweets about the “big game.” New Media & Society, 18(2), 293–312. https://doi.org/10.1177/1461444814541783
Hauge, J. B., Berta, R., Fiucci, G., Manjón, B. F., Padrón-Nápoles, C., Westra, W., & Nadolski, R. (2014). Implications of learning analytics for serious game design. In IEEE 14th International Conference on Advanced Learning Technologies (ICALT 2014), 7–9 July 2014, Athens, Greece (pp. 230–232). IEEE. https://doi.org/10.1109/ICALT.2014.73
Kang, J., Liu, M., & Qu, W. (2017). Using gameplay data to examine learning behavior patterns in a serious game. Computers in Human Behavior, 72, 757–770. https://doi.org/10.1016/j.chb.2016.09.062
Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379–425. http://www.jstor.org/stable/27739887
Kiili, K., Moeller, K., & Ninaus, M. (2018). Evaluating the effectiveness of a game-based rational number training—In-game metrics as learning indicators. Computers & Education, 120, 13–28. https://doi.org/10.1016/j.compedu.2018.01.012
Laffey, J., Griffin, J., Sigoloff, J., Sadler, T., Goggins, S., Womack, A., Wulff, E., & Lander, S. (n.d.). Mission HydroSci: Meeting learning standards through gameplay [Unpublished manuscript]. College of Education & Human Development, University of Missouri. https://files.eric.ed.gov/fulltext/ED600153.pdf
Liu, M., Lee, J., Kang, J., & Liu, S. (2016). What we can learn from the data: A multiple-case study examining behavior patterns by students with different characteristics in using a serious game. Technology, Knowledge and Learning, 21(1), 33–57. https://doi.org/10.1007/s10758-015-9263-7
Loh, C. S., & Li, I.-H. (2015). Predicting the competency improvement for serious games analytics: Action-sequences, game grids, PLS DA and JMP [Paper presentation]. Discovery Summit 2015, San Diego, CA, USA. https://doi.org/10.13140/RG.2.1.3997.4889
Loh, C. S., Li, I.-H., & Sheng, Y. (2016). Comparison of similarity measures to differentiate players’ actions and decisionmaking profiles in serious games analytics. Computers in Human Behavior, 64, 562–574. https://doi.org/10.1016/j.chb.2016.07.024
Loh, C. S., & Sheng, Y. (2014). Maximum similarity index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games. Computers in Human Behavior, 39, 322–330. https://doi.org/10.1016/j.chb.2014.07.022
Loh, C. S., Sheng, Y., & Li, I.-H. (2015). Predicting expert–novice performance as serious games analytics with objectiveoriented and navigational action sequences. Computers in Human Behavior, 49, 147–155. https://doi.org/10.1016/j.chb.2015.02.053
Martin, C. K., Nacu, D., & Pinkard, N. (2016). Revealing opportunities for 21st century learning: An approach to interpreting user trace log data. Journal of Learning Analytics, 3(2), 37–87. https://doi.org/10.18608/jla.2016.32.4
Palmquist, S. D., & Crowley, K. (2007). Studying dinosaur learning on an island of expertise. In R. Goldman, R. Pea, B. Barron, & S. J. Derry (Eds.), Video research in the learning sciences (pp. 271–286). Routledge. https://doi.org/10.4324/9780203877258
Renninger, K. A. (2012). Interest and its development. In R. M. Ryan (Ed.), The Oxford Handbook of Human Motivation (pp. 167–187). Oxford University Press.
Renninger, K. (2010). Working with and cultivating the development of interest, self-efficacy, and self-regulation. In D. D. Preiss & R. J. Sternberg (Eds.), Innovations in educational psychology: Perspectives on learning, teaching and human development (pp. 107–138). Springer. https://psycnet.apa.org/record/2010-06004-005
Renninger, K., Cai, M., Lewis, M., Adams, M., & Ernst, K. (2011). Motivation and learning in an online, unmoderated, mathematics workshop for teachers. Educational Technology Research and Development, 59(2), 229–247. https://doi.org/10.1007/s11423-011-9195-4
Serrano-Laguna, A., Martínez-Ortiz, I., Haag, J., Regan, D., Johnson, A., & Fernandez-Manjon, B. (2017). Applying standards to systematize learning analytics in serious games. Computer Standards & Interfaces, 50, 116–123. https://doi.org/10.1016/j.csi.2016.09.014
Shute, V., & Ventura, M. (2013). Stealth assessment: Measuring and supporting learning in video games. The MIT Press. https://doi.org/10.7551/mitpress/9589.001.0001
Towne, T. J., Boot, W. R., & Ericsson, K. A. (2016). Understanding the structure of skill through a detailed analysis of individuals’ performance on the Space Fortress game. Acta Psychologica, 169, 27–37. https://doi.org/10.1016/j.actpsy.2016.05.006
van Leeuwen, A. (2015). Learning analytics to support teachers during synchronous CSCL: Balancing between overview and overload. Journal of Learning Analytics, 2(2), 138–162. https://doi.org/10.18608/jla.2015.22.11
Ventura, M., & Shute, V. (2013). The validity of a game-based assessment of persistence. Computers in Human Behavior, 29(6), 2568–2572. https://doi.org/10.1016/j.chb.2013.06.033
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
