Game-Based Learning Analytics for Supporting Adolescents’ Reflection
DOI:
https://doi.org/10.18608/jla.2021.7371Keywords:
reflection, game-learning analytics, adolescents, problem solving, knowledge acquisitionAbstract
Reflection is critical for adolescents’ problem solving and learning in game-based learning environments (GBLEs). Yet challenges exist in the literature because most studies lack a theoretical perspective and clear operational definition to inform how and when reflection should be scaffolded during game-based learning. In this paper, we address these issues by studying the quantity and quality of 120 adolescents’ written reflections and their relation to their learning and problem solving with Crystal Island, a GBLE. Specifically, we (1) define reflection and how it relates to skill and knowledge acquisition; (2) review studies examining reflection and its relation to problem solving and learning with emerging technologies; and (3) provide direction for building reflection prompts into GBLEs that are aligned with the learning goals built into the learning session (e.g., learn about microbiology versus successfully solve a problem) to maximize adolescents’ reflection, learning, and performance. Overall, our findings emphasize how important it is to examine not only the quantity of reflection but also the depth of written reflection as it relates to specific learning goals. We discuss the implications of using game-learning analytics to guide instructional decision making in the classroom.
References
Ak, O., & Kutlu, B. (2017). Comparing 2D and 3D game-based learning environments in terms of learning gains and student perceptions. British Journal of Educational Technology, 48(1), 129–144. https://doi.org/10.1111/bjet.12346
Alzaid, M., & Hsiao, I.-H. (2018). Effectiveness of reflection on programming problem solving self-assessments. In Proceedings of the ieee frontiers in education conference (pp. 1–5). Piscataway, NJ, USA: IEEE. https://doi.org/10.1109/FIE.2018.8659245
Azevedo, R., Feyzi-Behnagh, R., Duffy, M., Harley, J., & Trevors, G. (2012). Metacognition and self-regulated learning in student-centered learning environments. In D. Jonassen & S. Land (Eds.), Theoretical Foundations of Learning Environments (pp. 171–197). New York, NY, USA: Routledge.
Azevedo, R., Taub, M., & Mudrick, N. (2018). Understanding and reasoning about real-time cognitive, affective, and metacognitive processes to foster self-regulation with advanced learning technologies. In D. H. Schunk & J. A. Greene (Eds.), Handbook of Self-Regulation of Learning and Performance (pp. 254–270). New York, NY, USA: Routledge.
Bannert, M., & Reimann, P. (2012). Supporting self-regulated hypermedia learning through prompts. Instructional Science, 40(1), 193–211. https://doi.org/10.1007/s11251-011-9167-4
Biswas, G., Segedy, J. R., & Bunchongchit, K. (2016). From design to implementation to practice a learning by teaching system: Betty’s Brain. International Journal of Artificial Intelligence in Education, 26(1), 350–364. https://doi.org/10.1007/s40593-015-0057-9
Brown, A., Bransford, J., Ferrara, R., & Campione, J. (1983). Learning, Remembering, and Understanding (Tech. Rep. No. 244). New York, NY, USA: Bolt, Beranek and Newman, Inc. and Center for the Study of Reading. Retrieved from https://files.eric.ed.gov/fulltext/ED217401.pdf
Carpenter, D., Cloude, E. B., Rowe, J., Azevedo, R., & Lester, J. (2021). Investigating student reflection during game-based learning in middle grades science. In Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK 2021), 12–16 April 2021, online (pp. 280–291). New York, NY, USA: ACM.
Carpenter, D., Geden, M., Rowe, J., Azevedo, R., & Lester, J. (2020). Automated analysis of middle school students’ written reflections during game-based learning. In I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Mill´an (Eds.), Proceedings of the International Conference on Artificial Intelligence in Education (AIED 2020), 6–10 July 2020, Utrecht, Netherlands (pp. 67–78). Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-030-52237-7 6
Clark, D. B., Tanner-Smith, E. E., & Killingsworth, S. S. (2016). Digital games, design, and learning: A systematic review and meta-analysis. Review of Educational Research, 86(1), 79–122. https://doi.org/10.3102/0034654315582065
Cloude, E. B., Dever, D. A., Wiedbusch, M. D., & Azevedo, R. (2020). Quantifying scientific thinking using multichannel data with Crystal Island: Implications for individualized game-learning analytics. Frontiers in Education, 5, 572546. https://doi.org/10.3389/feduc.2020.572546
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. Retrieved from https://doi.org/10.1177/001316446002000104
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
Dewey, J. (1923). Democracy and Education: An Introduction to the Philosophy of Education. Macmillan.
Dewey, J. (1933). How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process. Lexington, MA, USA: D. C. Heath and Co.
Fiorella, L., & Mayer, R. E. (2012). Paper-based aids for learning with a computer-based game. Journal of Educational Psychology, 104(4), 1074. https://doi.org/10.1037/a0028088
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906–911. https://doi.org/10.1037/0003-066X.34.10.906
Geden, M., Emerson, A., Carpenter, D., Rowe, J., Azevedo, R., & Lester, J. (2020). Predictive student modeling in game-based learning environments with word embedding representations of reflection. International Journal of Artificial Intelligence in Education, 7(3), 1–23. https://doi.org/10.1007/s40593-020-00220-4
Grubbs, F. E. (1969). Procedures for detecting outlying observations in samples. Technometrics, 11(1), 1–21. https://doi.org/10.1080/00401706.1969.10490657
Izu, C., & Alexander, B. (2018). Using unstructured practice plus reflection to develop programming/problem-solving fluency. In Proceedings of the 20th Australasian Computing Education Conference (ACE 2018), 30 January–2 February 2018, Brisbane, Australia (pp. 25–34). New York, NY, USA: ACM. https://doi.org/10.1145/3160489.3160496
Johnson, C. I., & Mayer, R. E. (2010). Applying the self-explanation principle to multimedia learning in a computer-based game-like environment. Computers in Human Behavior, 26(6), 1246–1252. https://doi.org/10.1016/j.chb.2010.03.025
Kovanović, V., Joksimović, S., Mirriahi, N., Blaine, E., Gašević, D., Siemens, G., & Dawson, S. (2018). Understand students’ self-reflections through learning analytics. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 7–9 March 2018, Sydney, Australia (pp. 389–398). New York, NY, USA: ACM. https://doi.org/10.1145/3170358.3170374
Luo, T., & Baaki, J. (2019). Scaffolding problem-solving and instructional design processes: Engaging students in reflection-in-action and external representations in three online courses. In M. Boboc & S. Koc¸ (Eds.), Student-Centered Virtual Learning Environments in Higher Education (pp. 40–69). Hershey, PA, USA: IGI Global. https://doi.org/10.4018/978-1-5225-5769-2.ch003
Mangaroska, K., Sharma, K., Gasevic, D., & Giannakos, M. (2020). Multimodal learning analytics to inform learning design: Lessons learned from computing education. Journal of Learning Analytics, 7(3), 31–60. https://doi.org/10.18608/jla.2020.73.7
Mayer, R. E. (2014). Computer games for learning: An evidence-based approach. Cambridge, MA, USA: MIT Press. McAlpine, L., Weston, C., Beauchamp, C., Wiseman, C., & Beauchamp, J. (1999). Building a metacognitive model of reflection. Higher Education, 37(2), 105–131. https://doi.org/10.1023/A:1003548425626
McQuiggan, S. W., Goth, J., Ha, E., Rowe, J. P., & Lester, J. C. (2008). Student note-taking in narrative-centered learning environments: Individual differences and learning effects. In Proceedings of the International Conference on Intelligent Tutoring Systems (ITS 2008), 23–27 June 2008, Montreal, QC, Canada (pp. 510–519). New York, NY, USA: Springer. Retrieved from https://doi.org/10.1007/978-3-540-69132-754
Moreno, R., & Mayer, R. E. (2005). Role of guidance, reflection, and interactivity in an agent-based multimedia game. Journal of Educational Psychology, 97(1), 117–128. Retrieved from https://doi.org/10.1037/0022-0663.97.1.117
Moshman, D. (2011). Adolescent Rationality and Development: Cognition, Morality, and Identity. New York, NY, USA: Taylor & Francis.
Moshman, D. (2013). Adolescent rationality. In R. M. Lerner & J. B. Benson (Eds.), Embodiment and Epigenesis: Theoretical and Methodological Issues in Understanding the Role of Biology within the Relational Developmental System (Vol. 45, pp. 155–183). Amsterdam, Netherlands: Elsevier. Retrieved from https://doi.org/10.1016/B978-0-12-397946-9.00007-5
NASEM. (2018). How People Learn II: Learners, Contexts, and Cultures. Washington, DC, USA: The National Academies Press. Retrieved from https://www.doi.org/10.17226/24783
NRC. (2012). Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. Washington, DC, USA: The National Academies Press. https://doi.org/10.17226/13398
OECD. (2016). PISA 2015 Results (Volume I): Excellence and Equity in Education. Paris, France: OECD Publishing. Retrieved from https://www.doi.org/10.1787/9789264266490-en
Patel, N., Baker, S. G., & Scherer, L. D. (2019). Evaluating the cognitive reflection test as a measure of intuition/reflection, numeracy, and insight problem solving, and the implications for understanding real-world judgments and beliefs. Journal of Experimental Psychology: General, 148(12), 2129–2153. https://doi.org/10.1037/xge0000592
Perez-Colado, I. J., Perez-Colado, V. M., Freire-Moran, M., Martinez-Ortiz, I., & Fernandez-Manjon, B. (2017). Integrating learning analytics into a game authoring tool. In Proceedings of the International Conference on Web-Based Learning (ICWL 2017), 20–22 September 2017, Cape Town, South Africa (pp. 51–61). New York, NY, USA: Springer. https://doi.org/10.1007/978-3-319-66733-1_6
Pintrich, P. R. (2002). The role of metacognitive knowledge in learning, teaching, and assessing. Theory Into Practice, 41(4), 219–225. https://doi.org/10.1207/s15430421tip4104_3
Plass, J. L., Homer, B. D., & Kinzer, C. K. (2015). Foundations of game-based learning. Educational Psychologist, 50(4), 258–283. https://doi.org/10.1080/00461520.2015.1122533
Plass, J. L., Mayer, R. E., & Homer, B. D. (2020). Handbook of Game-Based Learning. Cambridge, MA, USA: MIT Press. Qian, M., & Clark, K. R. (2016). Game-based learning and 21st century skills: A review of recent research. Computers in Human Behavior, 63, 50–58. https://doi.org/10.1016/j.chb.2016.05.023
R Core Team. (2019). R: A Language and Environment for Statistical Computing [Computer software manual]. Vienna, Austria. Retrieved from https://www.R-project.org/
Roll, I., & Winne, P. H. (2015). Understanding, evaluating, and supporting self-regulated learning using learning analytics. Journal of Learning Analytics, 2(1), 7–12. https://doi.org/10.18608/jla.2015.21.2
Rosenthal, D. (2000). Introspection and self-interpretation. Philosophical Topics, 28(2), 201–233. Retrieved from https://www.jstor.org/stable/43154687
Rowe, J. P., Shores, L. R., Mott, B. W., & Lester, J. C. (2011). Integrating learning, problem solving, and engagement in narrative-centered learning environments. International Journal of Artificial Intelligence in Education, 21(1-2), 115–133. https://doi.org/10.3233/JAI-2011-019
Scheffel, M., Drachsler, H., Stoyanov, S., & Specht, M. (2014). Quality indicators for learning analytics. Journal of Educational Technology & Society, 17(4), 117–132. Retrieved from https://www.jstor.org/stable/jeductechsoci.17.4.117
Shavelson, R. J., & Stern, P. (1981). Research on teachers’ pedagogical thoughts, judgments, decisions, and behavior. Review of Educational Research, 51(4), 455–498. https://doi.org/10.3102/00346543051004455
Siadaty, M., Gasevic, D., & Hatala, M. (2016). Trace-based micro-analytic measurement of self-regulated learning processes. Journal of Learning Analytics, 3(1), 183–214. Retrieved from https://doi.org/10.18608/jla.2016.31.11
Tarricone, P. (2011). The Taxonomy of Metacognition. London, England, UK: Psychology Press. Retrieved from https://doi.org/10.4324/9780203830529
Taub, M., Azevedo, R., Bradbury, A. E., & Mudrick, N. V. (2020). Self-regulation and reflection during game-based learning. In J. L. Plass, R. E. Mayer, & B. D. Homer (Eds.), Handbook of Game-Based Learning (pp. 239–262). Cambridge, MA, USA: MIT Press.
Taub, M., Sawyer, R., Smith, A., Rowe, J., Azevedo, R., & Lester, J. (2020). The agency effect: The impact of student agency on learning, emotions, and problem-solving behaviors in a game-based learning environment. Computers & Education, 147, 103781. https://doi.org/10.1016/j.compedu.2019.103781
Ullmann, T. D. (2019). Automated analysis of reflection in writing: Validating machine learning approaches. International Journal of Artificial Intelligence in Education, 29(2), 217–257. https://doi.org/10.1007/s40593-019-00174-2
Van Rossum, G., & Drake, F. L. (2011). The Python Language Reference Manual [Computer software manual]. Network Theory Ltd.
Venables, W. N., & Ripley, B. D. (2002). Modern Applied Statistics with S (Fourth ed.). New York, NY, USA: Springer. Retrieved from https://www.doi.org/10.1007/978-0-387-21706-2
Viberg, O., Khalil, M., & Baars, M. (2020). Self-regulated learning and learning analytics in online learning environments: A review of empirical research. In Proceedings of the 10th international conference on learning analytics and knowledge (pp. 524–533). New York, NY, USA: ACM. Retrieved from https://doi.org/10.1145/3375462.3375483
Vrugte, J. t., Jong, T. d., Wouters, P., Vandercruysse, S., Elen, J., & van Oostendorp, H. (2015). When a game supports prevocational math education but integrated reflection does not. Journal of Computer Assisted Learning, 31(5), 462–480. https://doi.org/10.1111/jcal.12104
Wang, H.-H., Chen, H.-T., Lin, H.-S., & Hong, Z.-R. (2017). The effects of college students’ positive thinking, learning motivation and self-regulation through a self-reflection intervention in Taiwan. Higher Education, Research, & Development, 36(1), 201–216. https://doi.org/10.1080/07294360.2016.1176999
Wickham, H. (2007). Reshaping data with the reshape package. Journal of Statistical Software, 21(12), 1–20. Retrieved from http://www.jstatsoft.org/v21/i12/
Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis. New York, NY, USA: Springer. https://doi.org/10.1007/978-3-319-24277-4
Wickham, H., & Bryan, J. (2019). readxl: Read Excel Files [Computer software manual]. (R package version 1.3.1) Retrieved from https://CRAN.R-project.org/package=readxl
Wickham, H., François, R., Henry, L., & Müller, K. (2020). dplyr: A Grammar of Data Manipulation [Computer software manual]. (R package version 0.8.5) https://doi.org/10.18637/jss.v021.i12
Wiedbusch, M. D., Kite, V., Yang, X., Park, S., Chi, M., Taub, M., & Azevedo, R. (2021). A theoretical and evidence-based conceptual design of MetaDash: An intelligent teacher dashboard to support teachers’ decision making and students’ self-regulated learning. Frontiers in Education, 6, 14. https://doi.org/10.3389/feduc.2021.570229
Winne, P. H. (2017). Learning analytics for self-regulated learning. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), Handbook of Learning Analytics (pp. 241–249). New York, NY, USA: Society for Learning Analytics Research. https://doi.org/10.18608/hla17.021
Winne, P. H., Teng, K., Chang, D., Lin, M. P.-C., Marzouk, Z., Nesbit, J. C., . . . Vytasek, J. (2019). nStudy: Software for learning analytics about processes for self-regulated learning. Journal of Learning Analytics, 6(2), 95–106. Retrieved from https://doi.org/10.18608/jla.2019.62.7
Wu, L., & Looi, C.-K. (2012). Agent prompts: Scaffolding for productive reflection in an intelligent learning environment. Journal of Educational Technology & Society, 15(1), 339–353. Retrieved from https://www.jstor.org/stable/jeductechsoci.15.1.339
Yamashita, T., Yamashita, K., & Kamimura, R. (2007). A stepwise AIC method for variable selection in linear regression. Communications in Statistics—Theory and Methods, 36(13), 2395–2403. https://doi.org/10.1080/03610920701215639
Zimmerman, B. J. (2013). From cognitive modeling to self-regulation: A social cognitive career path. Educational Psychologist, 48(3), 135–147. Retrieved from https://doi.org/10.1080/00461520.2013.794676
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
