Multimodal Data Fusion to Track Students’ Distress during Educational Gameplay
DOI:
https://doi.org/10.18608/jla.2022.7631Keywords:
educational game, distress, educational data mining, multimodal data fusion, research paperAbstract
Using multimodal data fusion techniques, we built and tested prediction models to track middle-school student distress states during educational gameplay. We collected and analyzed 1,145 data instances, sampled from a total of 31 middle-school students’ audio- and video-recorded gameplay sessions. We conducted data wrangling with student gameplay data from multiple data sources, such as individual facial expression recordings and gameplay logs. Using supervised machine learning, we built and tested candidate classifiers that yielded an estimated probability of distress states. We then conducted confidence-based data fusion that averaged the estimated probability scores from the unimodal classifiers with a single data source. The results of this study suggest that the classifier with multimodal data fusion improves the performance of tracking distress states during educational gameplay, compared to the performance of unimodal classifiers. The study finding suggests the feasibility of multimodal data fusion in developing game-based learning analytics. Also, this study proposes the benefits of optimizing several methodological means for multimodal data fusion in educational game research.
References
Alam, M. R., Bennamoun, M., Togneri, R., & Sohel, F. (2015). A confidence-based late fusion framework for audio-visual biometric identification. Pattern Recognition Letters, 52, 65–71. https://doi.org/10.1016/j.patrec.2014.10.006
Asbell-Clarke, J., Rowe, E., Almeda, V., Edwards, T., Bardar, E., Gasca, S., Baker, R. S., & Scruggs, R. (2021). The development of students’ computational thinking practices in elementary- and middle-school classes using the learning game, Zoombinis. Computers in Human Behavior, 115, 106587. https://doi.org/10.1016/j.chb.2020.106587
Baker, R. S., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human–Computer Studies, 68(4), 223–241. https://doi.org/10.1016/j.ijhcs.2009.12.003
Baltrušaitis, T., Robinson, P., & Morency, L. P. (2016, March). OpenFace: An open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 1–10). IEEE. https://doi.org/10.1109/WACV.2016.7477553
Beck, J., & Rodrigo, M. M. T. (2014). Understanding wheel spinning in the context of affective factors. In S. Trausan-Matu, K. E. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th International Conference on Intelligent Tutoring Systems (ITS 2014), 5–9 June 2014, Honolulu, HI, USA (pp. 162–167). Springer. https://doi.org/10.1007/978-3-319-07221-0_20
Biglan, A. (1991). Distressed behavior and its context. The Behavior Analyst, 14(2), 157–169. https://doi.org/10.1007/BF03392566
Bosch, N., Chen, Y., & D’Mello, S. (2014). It’s written on your face: Detecting affective states from facial expressions while learning computer programming. In S. Trausan-Matu, K. E. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th International Conference on Intelligent Tutoring Systems (ITS 2014), 5–9 June 2014, Honolulu, HI, USA (pp. 39–44). Springer. https://doi.org/10.1007/978-3-319-07221-0_5
Bosch, N., D’Mello, S., Baker, R., Ocumpaugh, J., Shute, V., Ventura, M., Wang, L., & Zhao, W. (2015). Automatic detection of learning-centered affective states in the wild. Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI ’15), 29 March–1 April 2015, Atlanta, GA, USA (pp. 379–388). ACM Press. https://doi.org/10.1145/2678025.2701397
Bowman, N. D., & Tamborini, R. (2012). Task demand and mood repair: The intervention potential of computer games. New Media & Society, 14(8), 1339–1357. https://doi.org/10.1177/146144481245042
Chrousos, G. P., Loriaux, D. L., & Gold, P. W. (2013). Mechanisms of physical and emotional stress (Vol. 245). Springer Science & Business Media.
Chango, W., Cerezo, R., & Romero, C. (2021). Multi-source and multimodal data fusion for predicting academic performance in blended learning university courses. Computers & Electrical Engineering, 89, 106908. https://doi.org/10.1016/j.compeleceng.2020.106908
Chen, F., Cui, Y., & Chu, M. W. (2020). Utilizing game analytics to inform and validate digital game-based assessment with evidence-centered game design: A case study. International Journal of Artificial Intelligence in Education, 30(3), 481–503. https://doi.org/10.1007/s40593-020-00202-6
DeFalco, J. A., Rowe, J. P., Paquette, L., Georgoulas-Sherry, V., Brawner, K., Mott, B. W., Baker, R. S., & Lester, J. C. (2018). Detecting and addressing frustration in a serious game for military training. International Journal of Artificial Intelligence in Education, 28(2), 152–193. https://doi.org/10.1007/s40593-017-0152-1
Emerson, A., Cloude, E. B., Azevedo, R., & Lester, J. (2020). Multimodal learning analytics for game‐based learning. British Journal of Educational Technology, 51(5), 1505–1526. https://doi.org/10.1111/bjet.12992
Eseryel, D., Law, V., Ifenthaler, D., Ge, X., & Miller, R. (2014). An investigation of the interrelationships between motivation, engagement, and complex problem solving in game-based learning. Journal of Educational Technology & Society, 17(1), 42–53. http://hdl.handle.net/20.500.11937/26368
Fernández, A., Garcia, S., Herrera, F., & Chawla, N. V. (2018). SMOTE for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary. Journal of Artificial Intelligence Research, 61, 863–905. https://doi.org/10.1613/jair.1.11192
Filsecker, M., & Hickey, D. T. (2014). A multilevel analysis of the effects of external rewards on elementary students’ motivation, engagement and learning in an educational game. Computers & Education, 75, 136–148. https://doi.org/10.1016/j.compedu.2014.02.008
Fultz, J., Schaller, M., & Cialdini, R. B. (1988). Empathy, sadness, and distress: Three related but distinct vicarious affective responses to another’s suffering. Personality and Social Psychology Bulletin, 14(2), 312–325. https://doi.org/10.1177/0146167288142009
Giannopoulos, P., Perikos, I., & Hatzilygeroudis, I. (2018). Deep learning approaches for facial emotion recognition: A case study on FER-2013. In I. Hatzilygeroudis & V. Palade (Eds.), Advances in hybridization of intelligent methods (pp. 1–16). Springer. https://doi.org/10.1007/978-3-319-66790-4_1
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., Zhou, Y., Ramaiah, C., Feng, F., Li, R., Wang, X., Athanasakis, D., Shawe-Taylor, J., Milakov, M., Park, J., ... & Bengio, Y. (2013, November). Challenges in representation learning: A report on three machine learning contests. ICML 2013 Workshop on Challenges in Representation Learning (pp. 117–124). Springer. https://doi.org/10.48550/arXiv.1307.0414
Gupta, H., Varshney, H., Sharma, T. K., Pachauri, N., & Verma, O. P. (2021). Comparative performance analysis of quantum machine learning with deep learning for diabetes prediction. Complex & Intelligent Systems, 8, 3073–3087. https://doi.org/10.1007/s40747-021-00398-7
Hamari, J., Shernoff, D. J., Rowe, E., Coller, B., Asbell-Clarke, J., & Edwards, T. (2016). Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. Computers in Human Behavior, 54, 170–179. https://doi.org/10.1016/j.chb.2015.07.045
Henderson, N. L., Rowe, J. P., Mott, B. W., Brawner, K., Baker, R., & Lester, J. C. (2019). 4D affect detection: Improving frustration detection in game-based learning with posture-based temporal data fusion. In S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, & R. Luckin (Eds.), Proceedings of the 20th International Conference on Artificial Intelligence in Education (AIED 2019), 25–29 June 2019, Chicago, IL, USA (pp. 144–156). Springer. https://doi.org/10.1007/978-3-030-23204-7_13
Henderson, N., Rowe, J., Paquette, L., Baker, R. S., & Lester, J. (2020). Improving affect detection in game-based learning with multimodal data fusion. In I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, E. Millán (Eds.), Proceedings of the 21st International Conference on Artificial Intelligence in Education (AIED 2020), 6–10 July 2020, Ifrane, Morocco (pp. 228–239). Springer. https://doi.org/10.1007/978-3-030-52237-7_19
Hofmann, M., & Klinkenberg, R. (Eds.). (2016). RapidMiner: Data mining use cases and business analytics applications. CRC Press.
Jadhav, A., Pramod, D., & Ramanathan, K. (2019). Comparison of performance of data imputation methods for numeric dataset. Applied Artificial Intelligence, 33(10), 913–933. https://doi.org/10.1080/08839514.2019.1637138
Kessler, R. C., Andrews, G., Colpe, L. J., Hiripi, E., Mroczek, D. K., Normand, S. L., Walters, E. E., & Zaslavsky, A. M. (2002). Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological Medicine, 32(6), 959–976. https://doi.org/10.1017/s0033291702006074
LePine, J. A., LePine, M. A., & Jackson, C. L. (2004). Challenge and hindrance stress: Relationships with exhaustion, motivation to learn, and learning performance. Journal of Applied Psychology, 89(5), 883–891. https://doi.org/10.1037/0021-9010.89.5.883
Liu, T., & Israel, M. (2022). Uncovering students’ problem-solving processes in game-based learning environments. Computers & Education, 182, 104462. https://doi.org/10.1016/j.compedu.2022.104462
Malarvizhi, R., & Thanamani, A. S. (2012). K-nearest neighbor in missing data imputation. International Journal of Engineering Research and Development, 5(1), 5–7. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.375.925&rep=rep1&type=pdf
Mouri, K., Suzuki, F., Shimada, A., Uosaki, N., Yin, C., Kaneko, K., & Ogata, H. (2019). Educational data mining for discovering hidden browsing patterns using non-negative matrix factorization. Interactive Learning Environments, 29(7), 1176-1188. https://doi.org/10.1080/10494820.2019.1619594
Mu, T., Jetten, A., & Brunskill, E. (2020). Towards suggesting actionable interventions for wheel-spinning students. Conference on Educational Data Mining (EDM2020), 10–13 July 2020, Online (pp. 183–193). International Educational Data Mining Society. https://educationaldatamining.org/files/conferences/EDM2020/papers/paper_201.pdf
Munshi, A., Rajendran, R., Ocumpaugh, J., Biswas, G., Baker, R. S., & Paquette, L. (2018, July). Modeling learners’ cognitive and affective states to scaffold SRL in open-ended learning environments. Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization (UMAP 2018), 8–11 July 2018, Singapore (pp. 131–138). ACM Press. https://doi.org/10.1145/3209219.3209241
Naylor, R. (2020). Key factors influencing psychological distress in university students: The effects of tertiary entrance scores. Studies in Higher Education, 47(3), 630–642. https://doi.org/10.1080/03075079.2020.1776245
Ochoa, X., & Worsley, M. (2016). Augmenting learning analytics with multimodal sensory data. Journal of Learning Analytics, 3(2), 213–219. https://doi.org/10.18608/jla.2016.32.10
Owen, V. E., Roy, M. H., Thai, K. P., Burnett, V., Jacobs, D., Keylor, E., & Baker, R. S. (2019). Detecting wheel-spinning and productive persistence in educational games. In C. F. Lynch, A. Merceron, M. Desmarais, & R. Nkambou (Eds.), Proceedings of the 12th International Conference on Educational Data Mining (EDM2019), 2–5 July 2019, Montréal, Quebec, Canada (pp. 378–383). International Educational Data Mining Society. https://files.eric.ed.gov/fulltext/ED599202.pdf
Plass, J. L., Homer, B. D., Pawar, S., Brenner, C., & MacNamara, A. P. (2019). The effect of adaptive difficulty adjustment on the effectiveness of a game to develop executive function skills for learners of different ages. Cognitive Development, 49, 56–67. https://doi.org/10.1016/j.cogdev.2018.11.006
Rodrigo, M. M. T. (2011). Dynamics of student cognitive-affective transitions during a mathematics game. Simulation & Gaming, 42(1), 85–99. https://doi.org/10.1177/1046878110361513
Rudland, J. R., Golding, C., & Wilkinson, T. J. (2020). The stress paradox: How stress can be good for learning. Medical Education, 54(1), 40–45. https://doi.org/10.1111/medu.13830
Rowe, E., Almeda, M. V., Asbell-Clarke, J., Scruggs, R., Baker, R., Bardar, E., & Gasca, S. (2021). Assessing implicit computational thinking in Zoombinis puzzle gameplay. Computers in Human Behavior, 120, 106707. https://par.nsf.gov/servlets/purl/10061932
Silvervarg, A., Haake, M., & Gulz, A. (2018). Perseverance is crucial for learning: “OK! but can I take a break?” In C. P. Rosé et al. (Eds.), Proceedings of the 19th International Conference on Artificial Intelligence in Education (AIED 2018), 27–30 June 2018, London, UK (pp. 532–544). Lecture Notes in Computer Science vol. 10947. Springer. https://doi.org/10.1007/978-3-319-93843-1_39
Shute, V. J., D’Mello, S., Baker, R., Cho, K., Bosch, N., Ocumpaugh, J., Ventura, M., & Almeda, V. (2015). Modeling how incoming knowledge, persistence, affective states, and in-game progress influence student learning from an educational game. Computers & Education, 86, 224–235. https://doi.org/10.1016/j.compedu.2015.08.001
Taub, M., Azevedo, R., Bradbury, A. E., & Mudrick, N. V. (2020). Self-regulation and reflection during game-based learning. In J. L. Plass, R. E. Mayer, & B. D. Homer (Eds.), Handbook of game-based learning (pp. 239–262). The MIT Press.
Verma, V., Rheem, H., Amresh, A., Craig, S. D., & Bansal, A. (2020, December). Predicting real-time affective states by modeling facial emotions captured during educational video game play. In I. Marfisi-Schottman, F. Bellotti, L. Hamon, & R. Klemke (Eds.), Proceedings of the International Conference on Games and Learning Alliance (GALA 2020), 1–3 December 2020, La Spezia, Italy. Lecture Notes in Computer Science vol. 12517. (pp. 447–452). Springer. https://doi.org/10.1007/978-3-030-63464-3_45
Yeh, Y. C., Lai, G. J., Lin, C. F., Lin, C. W., & Sun, H. C. (2015). How stress influences creativity in game-based situations: Analysis of stress hormones, negative emotions, and working memory. Computers & Education, 81, 143–153. https://doi.org/10.1016/j.compedu.2014.09.011
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
