Developing a Stealth Assessment System Using a Continuous Conjunctive Model
DOI:
https://doi.org/10.18608/jla.2022.7639Keywords:
digital game-based learning, learning analytics, CDMS, CCM, adaptive expertise, mathematics, research paperAbstract
Integrating learning analytics in digital game-based learning has gained popularity in recent decades. The interactive nature of educational games creates an ideal environment for learning analytics data collection. However, past research has limited success in producing accessible and effective assessments using game learning analytics. In this study, a mathematics educational game called The Nomads was designed and developed to train learners’ adaptive expertise in rational number arithmetic. Players’ game log data were captured and fitted to a cognitive diagnostic model (CDM) — CCM (continuous conjunctive model). CCM lends itself well to the complex and dynamic nature of game learning analytics. Unlike traditional CDMs, CCM generates parameters at an attribute level and offers more parsimonious diagnoses using continuous variables. The findings suggest that learners’ attribute mastery improved during the gameplay and that learners benefit from using the scaffolds for three of the attributes instructed by the game. This study presents the application of a powerful new tool for game learning analytics. Future studies can benefit from more generalized analytics models and more specified learning attributes and game tasks.
References
Adamo-Villani, N., Haley-Hermiz, T., & Cutler, R. (2013). Using a serious game approach to teach ‘operator precedence’ to introductory programming students. Proceedings of the 17th International Conference on Information Visualisation (IV 2013), 16–18 July, London, UK (pp. 523–526). IEEE Computer Society. https://doi.org/10.1109/IV.2013.70
Alonso-Fernandez, C., Calvo, A., Freire, M., Martinez-Ortiz, I., & Fernandez-Manjon, B. (2017). Systematizing game learning analytics for serious games. Proceedings of the 2017 IEEE Global Engineering Education Conference (EDUCON 2017), 25–28 April 2017, Athens, Greece (pp. 1111–1118). IEEE Computer Society. https://doi.org/10.1109/EDUCON.2017.7942988
Alonso-Fernández, C., Cano, A. R., Calvo-Morata, A., Freire, M., Martínez-Ortiz, I., & Fernández-Manjón, B. (2019). Lessons learned applying learning analytics to assess serious games. Computers in Human Behavior, 99, 301–309. https://doi.org/10.1016/j.chb.2019.05.036
Alrehaili, E. A., & Al Osman, H. (2022). A virtual reality role-playing serious game for experiential learning. Interactive Learning Environments, 30(5), 922–935. https://doi.org/10.1080/10494820.2019.1703008
Baroody, A. J. (2003). The development of adaptive expertise and flexibility: The integration of conceptual and procedural knowledge. In A. J. Baroody & A. Dowker (Eds.), The development of arithmetic concepts and skills: Constructing adaptive expertise (pp. 1–33). Lawrence Erlbaum.
Beavis, C. (2015). Multimodal literacy, digital games and curriculum. In T. Lowrie & R. Jorgensen (Eds.), Digital games and mathematics learning (pp. 109–122). Springer. https://doi.org/10.1007/978-94-017-9517-3_7
Berkovsky, S., Coombe, M., Freyne, J., Bhandari, D., & Baghaei, N. (2010). Physical activity motivating games: Virtual rewards for real activity. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ10), 10–15 April, Atlanta, GA, USA (pp. 243–252). ACM Press. https://doi.org/10.1145/1753326.1753362
Blöte, A. W., Klein, A. S., & Beishuizen, M. (2000). Mental computation and conceptual understanding. Learning and Instruction, 10, 221–247. https://doi.org/10.1016/S0959-4752(99)00028-6
Brandt, D. S. (1997). Constructivism: Teaching for understanding of the internet. Communications of the ACM, 40(10), 112–117. https://doi.org/10.1145/262793.262814
Bressler, D. M., & Bodzin, A. M. (2013). A mixed methods assessment of students’ flow experiences during a mobile augmented reality science game. Journal of Computer Assisted Learning, 29(6), 505–517. https://doi.org/10.1111/jcal.12008
Brezovszky, B., McMullen, J., Veermans, K., Hannula-Sormunen, M. M., Rodríguez-Aflecht, G., Pongsakdi, N., Laakkonen, E., & Lehtinen, E. (2019). Effects of a mathematics game-based learning environment on primary school students’ adaptive number knowledge. Computers & Education, 128, 63–74. https://doi.org/10.1016/j.compedu.2018.09.011
Chen, C. H., Liu, G. Z., & Hwang, G. J. (2016). Interaction between gaming and multistage guiding strategies on students’ field trip mobile learning performance and motivation. British Journal of Educational Technology, 47(6), 1032–1050. https://doi.org/10.1111/bjet.12270
Chen, P. Y., Hwang, G. J., Yeh, S. Y., Chen, Y. T., Chen, T. W., & Chien, C. H. (2021). Three decades of game-based learning in science and mathematics education: An integrated bibliometric analysis and systematic review. Journal of Computers in Education, 9, 455–476. https://doi.org/10.1007/s40692-021-00210-y
Corbett, A. T., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253–278. https://doi.org/10.1007/BF01099821
de la Torre, J. (2011). The generalized DINA Model framework. Psychometrika 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7
de la Torre, J., & Chiu, C. Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273. https://doi.org/10.1007/s11336-015-9467-8
Dede, C. (2012). Interweaving assessments into immersive authentic simulations: Design strategies for diagnostic and instructional insights. [Paper presentation]. Invitational Research Symposium on Technology Enhanced Assessments, 7–8 May 2012. https://www.ets.org/Media/Research/pdf/session4-dede-paper-tea2012.pdf
Devlin, K. (2011). Mathematics education for a new era: Video games as a medium for learning. CRC Press.
El-Nasr, M. S., Desurvire, H., Aghabeigi, B., & Drachen, A. (2013). Game analytics for game user research, Part 1: A workshop review and case study. IEEE Computer Graphics and Applications, 33(2), 6–11. https://doi.org/10.1109/MCG.2013.26
Freire, M., Serrano-Laguna, Á., Iglesias, B. M., Martínez-Ortiz, I., Moreno-Ger, P., & Fernández-Manjón, B. (2016). Game learning analytics: Learning analytics for serious games. In M. J. Spector, B. B. Lockee, & M. D. Childress (Eds.), Learning, design, and technology (pp. 1–29). Springer. https://doi.org/10.1007/978-3-319-17727-4_21-1
Friedman, N., & Goldszmidt, M. (1996). Discretizing continuous attributes while learning Bayesian networks. Proceedings of the 13th International Conference on Machine Learning (ICML ’96), 3–6 July, Bari, Italy (pp. 157–165). Morgan Kaufmann Publishers. https://dl.acm.org/doi/10.5555/3091696.3091716
Gaschler, R., Vaterrodt, B., Frensch, P. A., Eichler, A., & Haider, H. (2013). Spontaneous usage of different shortcuts based on the commutativity principle. PLoS One, 8(9), e74972. https://doi.org/10.1371/journal.pone.0074972
Gee, J. P. (2004). What video games have to teach us about learning and literacy. Palgrave Macmillan.
Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301–321. https://doi.org/10.1111/j.1745-3984.1989.tb00336.x
Hatano, G., & Oura, Y. (2003). Reconceptualizing school learning using insight from expertise research. Educational Researcher, 32(8), 26–29. https://doi.org/10.3102/0013189X032008026
Hauge, J. B., Berta, R., Fiucci, G., Manjón, B. F., Padrón-Nápoles, C., Westra, W., & Nadolski, R. (2014). Implications of learning analytics for serious game design. Proceedings of the 14th IEEE International Conference on Advanced Learning Technologies (ICALT 2014), 7–10 July 2014, Athens, Greece (pp. 230–232). IEEE Computer Society. https://dl.acm.org/doi/10.1109/ICALT.2014.73
Heckenberg, S. G., Herbert, R. D., & Webber, R. (2004). Visualisation of the minority game using a mod. Proceedings of the 2004 Australasian Symposium on Information Visualisation (APVis ’04) 23–24 January 2004, Christchurch, New Zealand (Vol. 35, pp. 157–163). Australian Computer Society. https://crpit.scem.westernsydney.edu.au/confpapers/CRPITV35Heckenberg.pdf
Heckerman, D. (2008). A tutorial on learning with Bayesian networks. In D. E. Holmes & L. C. Jain (Eds.), Innovations in Bayesian networks (pp. 33–82). Springer. https://doi.org/10.1007/978-3-540-85066-3_3
Hong, H., Wang, C., Lim, Y. S., & Douglas, J. (2015). Efficient models for cognitive diagnosis with continuous and mixed-type latent variables. Applied Psychological Measurement, 39(1), 31–43. https://doi.org/10.1177/0146621614524981
Jensen, F. V., & Nielsen, T. D. (2007). Bayesian networks and decision graphs (Vol. 2). Springer. https://doi.org/10.1007/978-0-387-68282-2
Junker, B. W., & Sijtsma, K. (2001). Nonparametric item response theory in action: An overview of the special issue. Applied Psychological Measurement, 25, 258–272. https://doi.org/10.1177/01466210122032028
Ifenthaler, D. (2015). Learning analytics. In J. M. Spector (Ed.), The SAGE encyclopedia of educational technology. SAGE Publications.
Ifenthaler, D., & Widanapathirana, C. (2014). Development and validation of a learning analytics framework: Two case studies using support vector machines. Technology, Knowledge, and Learning, 19(1–2), 221–240. https://doi.org/10.1007/s10758-014-9226-4
Ke, F., & Shute, V. (2015). Design of game-based stealth assessment and learning support. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement (pp. 301–318). Springer. https://doi.org/10.1007/978-3-319-05834-4_13
Kickmeier-Rus, M. D. (2018). Predicting learning performance in serious games. In S. Göbel, A. Garcia-Agundez, T. Tregel, M. Ma, J. Baalsrud Hauge, M. Oliveira, T. Marsh, & P. Caserman (Eds.), Serious games (pp. 133–144). Springer. https://doi.org/10.1007/978-3-030-02762-9_14
Kosmas, P., Ioannou, A., & Retalis, S. (2018). Moving bodies to moving minds: A study of the use of motion-based games in special education. TechTrends, 62(6), 594–601. https://doi.org/10.1007/s11528-018-0294-5
Ku, O., Chen, S. Y., Wu, D. H., Lao, A. C., & Chan, T. W. (2014). The effects of game-based learning on mathematical confidence and performance: High ability vs. low ability. Journal of Educational Technology & Society, 17(3), 65–78. https://eric.ed.gov/?id=EJ1038973
Levy, R. (2014). Dynamic Bayesian network modeling of game based diagnostic assessments. CRESST Report 837. University of California, National Center for Research on Evaluation, Standards, and Student Testing. https://doi.org/10.1080/00273171.2019.1590794
Liu, M., Kang, J., Liu, S., Zou, W., & Hodson, J. (2017). Learning analytics as an assessment tool in serious games: A review of literature. In M. Ma & A. Oikonomou (Eds.), Serious games and edutainment applications (pp. 537–563). Springer. https://doi.org/10.1007/978-3-319-51645-5_24
Loh, C. S., Sheng, Y., & Ifenthaler, D. (Eds.). (2015). Serious games analytics: Methodologies for performance measurement, assessment, and improvement. Springer. https://doi.org/10.1007/978-3-319-05834-4
López-Martínez, Á., Santiago-Ramajo, S., Caracuel, A., Valls-Serrano, C., Hornos, M. J., & Rodríguez-Fórtiz, M. J. (2011). Game of gifts purchase: Computer-based training of executive functions for the elderly. 2011 IEEE 1st International Conference on Serious Games and Applications for Health (SeGAH 2011), 16–18 November, Braga, Portugal (pp. 1–8). IEEE Computer Society.
Lopez-Morteo, G., & Lopez, G. (2007). Computer support for learning mathematics: A learning environment based on recreational learning objects. Computers & Education, 48(4), 618–641. https://doi.org/10.1016/j.compedu.2005.04.014
Lowrie, T., & Jorgensen, R. (2015). Digital games and learning: What’s new is already old? In T. Lowrie & R. Jorgensen (Eds.), Digital games and mathematics learning (pp. 1–9). Springer. https://doi.org/10.1007/978-94-017-9517-3_1
Lucas, P. (2004). Bayesian analysis, pattern analysis, and data mining in health care. Current Opinion in Critical Care, 10(5), 399–403. https://doi.org/10.1097/01.ccx.0000141546.74590.d6
Ma, W., & de la Torre, J. (2020). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14), 1–26. https://doi.org/10.18637/jss.v093.i14
Mavridis, A., Katmada, A., & Tsiatsos, T. (2017). Impact of online flexible games on students’ attitude towards mathematics. Educational Technology Research and Development, 65(6), 1451–1470. https://doi.org/10.1007/s11423-017-9522-5
McMullen, J., Brezovszky, B., Hannula-Sormunen, M. M., Veermans, K., Rodríguez-Aflecht, G., Pongsakdi, N., & Lehtinen, E. (2017). Adaptive number knowledge and its relation to arithmetic and pre-algebra knowledge. Learning and Instruction, 49, 178–187. https://doi.org/10.1016/j.learninstruc.2017.02.001
McMullen, J., Brezovszky, B., Rodríguez-Aflecht, G., Pongsakdi, N., Hannula-Sormunen, M. M., & Lehtinen, E. (2016). Adaptive number knowledge: Exploring the foundations of adaptivity with whole-number arithmetic. Learning and Individual Differences, 47, 172–181. https://doi.org/10.1016/j.lindif.2016.02.007
McMullen, J., Kanerva, K., Lehtinen, E., Hannula-Sormunen, M. M., & Kiuru, N. (2019). Adaptive number knowledge in secondary school students: Profiles and antecedents. Journal of Numerical Cognition, 5(3), 283–300. https://doi.org/10.5964/jnc.v5i3.201
National Council of Teachers of Mathematics. (2014). Principles to actions: Ensuring mathematical success for all. https://www.nctm.org/PtA/
National Governors Association. (2010). Common core state standards. National Governors Association Center for Best Practices & Council of Chief State School Officers. Washington, DC.
Nunes, T., Dorneles, B. V., Lin, P. J., & Rathgeb-Schnierer, E. (2016). Teaching and learning about whole numbers in primary school. Springer. https://doi.org/10.1007/978-3-319-45113-8_1
Ohri, A. (2021). Bayesian belief networks: An introduction in 6 easy points. Jigsaw Academy. https://www.jigsawacademy.com/blogs/data-science/bayesian-belief-network
Owen, V. E., & Baker, R. S. (2019). Learning analytics for games. In J. L. Plass, R. E. Mayer, & B. D. Homer (Eds.), Handbook of game-based learning (pp. 513–535). MIT Press.
Pareto, L., Haake, M., Lindström, P., Sjödén, B., & Gulz, A. (2012). A teachable-agent-based game affording collaboration and competition: Evaluating math comprehension and motivation. Educational Technology Research and Development, 60(5), 723–751. https://doi.org/10.1007/s11423-012-9246-5
Piu, A., Fregola, C., & Santoro, A. (2015). Development and analysis of a design model for geometry-based simulation games. Procedia: Social and Behavioral Sciences, 186, 293–304. https://doi.org/10.1016/j.sbspro.2015.04.206
Qudrat-Ullah, H. (2010). Perceptions of the effectiveness of system dynamics-based interactive learning environments: An empirical study. Computers & Education, 55(3), 1277–1286. https://doi.org/10.1016/j.compedu.2010.05.025
Rhyne, T. M. (2000). Computer games’ influence on scientific and information visualization. Computer, 33(12), 154–159. https://doi.org/10.1109/2.889099
Rittle-Johnson, B., & Star, J. R. (2009). Compared with what? The effects of different comparisons on conceptual knowledge and procedural flexibility for equation solving. Journal of Educational Psychology, 101(3), 529–544. https://doi.org/10.1037/a0014224
Rowe, E., Asbell-Clarke, J., Bardar, E., Almeda, M. V., Baker, R. S., Scruggs, R., & Gasca, S. (2020). Advancing research in game-based learning assessment: Tools and methods for measuring implicit learning. In E. Kennedy & Y. Qian (Eds.), Advancing educational research with emerging technology (pp. 99–123). IGI Global. https://doi.org/10.4018/978-1-7998-1173-2.ch006
Serrano-Laguna, Á., Martínez-Ortiz, I., Haag, J., Regan, D., Johnson, A., & Fernández-Manjón, B. (2017). Applying standards to systematize learning analytics in serious games. Computer Standards & Interfaces, 50, 116–123. https://doi.org/10.1016/j.csi.2016.09.014
Serrano-Laguna, Á., Torrente, J., Moreno-Ger, P., & Fernández-Manjón, B. (2014). Application of learning analytics in educational videogames. Entertainment Computing, 5(4), 313–322. https://doi.org/10.1016/j.entcom.2014.02.003
Shute, V. J., & Kim, Y. J. (2014). Formative and stealth assessment. In J. Spector, M. Merrill, J. Elen, & M. Bishop (Eds.), Handbook of research on educational communications and technology (pp. 311–321). Springer. https://doi.org/10.1007/978-1-4614-3185-5_25
Shute, V., Rahimi, S., Smith, G., Ke, F., Almond, R., Dai, C. P., Kuba, R., Liu, Z., Yang, X., & Sun, C. (2021). Maximizing learning without sacrificing the fun: Stealth assessment, adaptivity and learning supports in educational games. Journal of Computer Assisted Learning, 37(1), 127–141. https://doi.org/10.1111/jcal.12473
Shute, V., & Ventura, M. (2013). Stealth assessment: Measuring and supporting learning in video games. The MIT Press.
Shute, V. J., Wang, L., Greiff, S., Zhao, W., & Moore, G. (2016). Measuring problem solving skills via stealth assessment in an engaging video game. Computers in Human Behavior, 63, 106–117. https://doi.org/10.1016/j.chb.2016.05.047
Siegler, R. S., & Lemaire, P. (1997). Older and younger adults’ strategy choices in multiplication: Testing predictions of ASCM using the choice/no-choice method. Journal of Experimental Psychology: Gênerai, 126(1), 71–92. https://doi.org/10.1037/0096-3445.126.1.71
Smith, S. P., Blackmore, K., & Nesbitt, K. (2015). A meta-analysis of data collection in serious games research. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement (pp. 31–55). Springer. https://doi.org/10.1007/978-3-319-05834-4_2
Star, J. R., & Seifert, C. (2006). The development of flexibility in equation solving. Contemporary Educational Psychology, 31, 280–300. https://doi.org/10.1016/j.cedpsych.2005.08.001
Steiner, C. M., Kickmeier-Rus, M. D., & Albert, D. (2015). Making sense of game-based user data: Learning analytics in applied games. [Paper presentation]. International Association for Development of the Information Society, Las Palmas de Gran Canaria, Spain. https://files.eric.ed.gov/fulltext/ED562478.pdf
Tatsuoka, K. K. (1983): Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x.
Tatsuoka, K. K. (1990). Toward an integration of item-response theory and cognitive error diagnoses. In N. Frederiksen, R. L. Glaser, A. M. Lesgold, & M. G. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition. Lawrence Erlbaum Associates.
Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305. https://doi.org/10.1037/1082-989X.11.3.287
Uusitalo, L. (2007). Advantages and challenges of Bayesian networks in environmental modelling. Ecological Modelling, 203(3–4), 312–318. https://doi.org/10.1016/j.ecolmodel.2006.11.033
Verkijika, S. F., & De Wet, L. (2015). Using a brain-computer interface (BCI) in reducing math anxiety: Evidence from South Africa. Computers & Education, 81, 113–122. https://doi.org/10.1016/j.compedu.2014.10.002
Verschaffel, L., Luwel, K., Torbeyns, J., & Van Dooren, W. (2009). Conceptualizing, investigating, and enhancing adaptive expertise in elementary mathematics education. European Journal of Psychology of Education, 24, 335–359. https://doi.org/10.1007/BF03174765
von Davier, M., & Lee, Y.-S. (Eds.). (2019). Handbook of diagnostic classification models: Models and Model Extensions, Applications, Software Packages. Springer. https://doi.org/10.1007/978-3-030-05584-4
Wang, S., Yang, Y., Culpepper, S. A., & Douglas, J. A. (2018). Tracking skill acquisition with cognitive diagnosis models: A higher-order, hidden Markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57–87. https://doi.org/10.3102/1076998617719727
Westera, W., Nadolski, R. J., Hummel, H. G., & Wopereis, I. G. (2008). Serious games for higher education: A framework for reducing design complexity. Journal of Computer Assisted Learning, 24(5), 420–432. https://doi.org/10.1111/j.1365-2729.2008.00279.x
Whitton, N. (2014). Digital games and learning: Research and theory. Routledge.
Wiemeyer, J., Kickmeier-Rus, M., & Steiner, C. M. (2016). Performance assessment in serious games. In S. Göbel, A. Garcia-Agundez, T. Tregel, M. Ma, J. Baalsrud Hauge, M. Oliveira, T. Marsh, & P. Caserman (Eds.), Serious games (pp. 273–302). Springer. https://link.springer.com/chapter/10.1007/978-3-319-40612-1_10
Wilcox, R. R., & Erceg-Hurn, D. M. (2012). Comparing two dependent groups via quantiles. Journal of Applied Statistics, 39(12), 2655–2664. https://doi.org/10.1080/02664763.2012.724665
Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). Woodcock-Johnson III NU complete. Riverside Publishing. https://www.hmhco.com/~/media/sites/home/hmh-assessments/clinical/woodcock-johnson/pdf/wjiii/wjiii_asb8.pdf?la=en
Zin, N. A. M., Jaafar, A., & Yue, W. S. (2009). Digital game-based learning (DGBL) model and development methodology for teaching history. WSEAS Transactions on Computers, 8(2), 322–333. http://www.wseas.us/e-library/transactions/computers/2009/28-786.pdf
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
