Enriching Multimodal Data
A Temporal Approach to Contextualize Joint Attention in Collaborative Problem-Solving
DOI:
https://doi.org/10.18608/jla.2023.7989Keywords:
collaborative problem-solving, temporality, temporal analysis, multimodal learning analytics, collaboration analytics, CSCL, research paperAbstract
Collaboration is a complex, multidimensional process; however, details of how multimodal features intersect and mediate group interactions have not been fully unpacked. Characterizing and analyzing the temporal patterns based on multimodal features is a challenging yet important work to advance our understanding of computer-supported collaborative learning (CSCL). This paper highlights the affordances, as well as the limitations, of different temporal approaches in terms of analyzing multimodal data. To tackle the remaining challenges, we present an empirical example of multimodal temporal analysis that leverages multi-level vector autoregression (mlVAR) to identify temporal patterns of the collaborative problem-solving (CPS) process in an immersive astronomy simulation. We extend previous research on joint attention with a particular focus on the added value from a multimodal, temporal account of the CPS process. We incorporate verbal discussion to contextualize joint attention, examine the sequential and contemporaneous associations between them, and identify significant differences in temporal patterns between low- and high-achieving groups. Our paper does the following: 1) creates interpretable multimodal group interaction patterns, 2) advances understanding of CPS through examination of verbal and non-verbal interactions, and 3) demonstrates the added value of a complete account of temporality including both duration and sequential order.
References
Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511527685
Baker, R. S. J. d., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human–Computer Studies, 68(4), 223–241. https://doi.org/10.1016/j.ijhcs.2009.12.003
Barron, B. (2003). When smart groups fail. Journal of the Learning Sciences, 12(3), 307–359. https://doi.org/10.1207/S15327809JLS1203_1
Bringmann, L. F., Vissers, N., Wichers, M., Geschwind, N., Kuppens, P., Peeters, F., Borsboom, D., & Tuerlinckx, F. (2013). A network approach to psychopathology: New insights into clinical longitudinal data. PLoS ONE, 8(4), e60188. https://doi.org/10.1371/journal.pone.0060188
Brohinsky, J., Marquart, C., Wang, J., Ruis, A. R., & Shaffer, D. W. (2021). Trajectories in epistemic network analysis. In A. R. Ruis & S. B. Lee (Eds.), Advances in quantitative ethnography (pp. 106–121). Springer. https://doi.org/10.1007/978-3-030-67788-6_8
Chang, C.-J., Chang, M.-H., Chiu, B.-C., Liu, C.-C., Fan Chiang, S.-H., Wen, C.-T., Hwang, F.-K., Wu, Y.-T., Chao, P.-Y., Lai, C.-H., Wu, S.-W., Chang, C.-K., & Chen, W. (2017). An analysis of student collaborative problem solving activities mediated by collaborative simulations. Computers & Education, 114, 222–235. https://doi.org/10.1016/j.compedu.2017.07.008
Chango, W., Lara, J. A., Cerezo, R., & Romero, C. (2022). A review on data fusion in multimodal learning analytics and educational data mining. WIREs Data Mining and Knowledge Discovery, 12(4), e1458. https://doi.org/10.1002/widm.1458
Chen, B., Knight, S., & Wise, A. F. (2018). Critical issues in designing and implementing temporal analytics. Journal of Learning Analytics, 5(1). https://doi.org/10.18608/jla.2018.53.1
Chen, B., Resendes, M., Chai, C. S., & Hong, H.-Y. (2017). Two tales of time: Uncovering the significance of sequential patterns among contribution types in knowledge-building discourse. Interactive Learning Environments, 25(2), 162–175. https://doi.org/10.1080/10494820.2016.1276081
Chiu, M. M. (2000). Group problem-solving processes: Social interactions and individual actions. Journal for the Theory of Social Behaviour, 30(1), 26–49. https://doi.org/10.1111/1468-5914.00118
Csanadi, A., Eagan, B., Kollar, I., Shaffer, D. W., & Fischer, F. (2018). When coding-and-counting is not enough: Using epistemic network analysis (ENA) to analyze verbal data in CSCL research. International Journal of Computer-Supported Collaborative Learning, 13(4), 419–438. https://doi.org/10.1007/s11412-018-9292-z
Cukurova, M., Giannakos, M., & Martinez-Maldonado, R. (2020). The promise and challenges of multimodal learning analytics. British Journal of Educational Technology, 51(5), 1441–1449. https://doi.org/10.1111/bjet.13015
Diederich, M., Kang, J., Kim, T., & Lindgren, R. (2021). Developing an in-application shared view metric to capture collaborative learning in a multi-platform astronomy simulation. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 173–183). ACM Press. https://doi.org/10.1145/3448139.3448156
Dillenbourg, P., Baker, M., Blaye, A., & O’Malley, C. (1996). The evolution of research on collaborative learning. In E. Spada & P. Reiman (Eds.), Learning in humans and machine: Towards an interdisciplinary learning science (pp. 189–211). Oxford, Elsevier. http://infoscience.epfl.ch/record/33761
Dillenbourg, P., Järvelä, S., & Fischer, F. (2009). The evolution of research on computer-supported collaborative learning: From design to orchestration (pp. 3–19). https://doi.org/10.1007/978-1-4020-9827-7_1
Echeverria, V., Martinez-Maldonado, R., & Buckingham Shum, S. (2019). Towards collaboration translucence: Giving meaning to multimodal group data. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 4–9 May 2019, Glasgow, Scotland, UK (Paper no. 39). ACM Press. https://doi.org/10.1145/3290605.3300269
Emerson, A., Cloude, E. B., Azevedo, R., & Lester, J. (2020). Multimodal learning analytics for game‐based learning. British Journal of Educational Technology, 51(5), 1505–1526. https://doi.org/10.1111/bjet.12992
Epskamp, S., Deserno, M. K., Bringmann, L. F., & Veenman, M. (2021). mlVAR: Multi-level vector autoregression (0.5) [Computer software]. https://CRAN.R-project.org/package=mlVAR
Epskamp, S., Waldorp, L. J., Mõttus, R., & Borsboom, D. (2018). The Gaussian graphical model in cross-sectional and time-series data. Multivariate Behavioral Research, 53(4), 453–480. https://doi.org/10.1080/00273171.2018.1454823
Fiore, S. M., Rosen, M. A., Smith-Jentsch, K. A., Salas, E., Letsky, M., & Warner, N. (2010). Toward an understanding of macrocognition in teams: Predicting processes in complex collaborative contexts. Human Factors, 52(2), 203–224. https://doi.org/10.1177/0018720810369807
Helske, J., & Helske, S. (2021). seqHMM: Mixture hidden Markov models for social sequence data and other multivariate, multichannel categorical time series (1.2.0) [Computer software]. https://CRAN.R-project.org/package=seqHMM
Janssen, J., Kirschner, F., Erkens, G., Kirschner, P. A., & Paas, F. (2010). Making the black box of collaborative learning transparent: Combining process-oriented and cognitive load approaches. Educational Psychology Review, 22(2), 139–154. https://doi.org/10.1007/s10648-010-9131-x
Kang, J., An, D., Yan, L., & Liu, M. (2019). Collaborative problem-solving process in a science serious game: Exploring group action similarity trajectory. In C. F. Lynch, A. Merceron, M. Desmarais, & R. Nkambou (Eds.), Proceedings of the 12th International Conference on Educational Data Mining (EDM2019), 2–5 July 2019, Montréal, Quebec, Canada (pp. 336–341). International Educational Data Mining Society.
Kapur, M., Voiklis, J., & Kinzer, C. K. (2008). Sensitivities to early exchange in synchronous computer-supported collaborative learning (CSCL) groups. Computers & Education, 51(1), 54–66. https://doi.org/10.1016/j.compedu.2007.04.007
Karumbaiah, S., Andres, J. M. A. L., Botelho, A. F., Baker, R. S., & Ocumpaugh, J. S. (2018). The implications of a subtle difference in the calculation of affect dynamics. In J. C. Yang, M. Chang, L.-H. Wong, & M. T. Rodrigo (Eds.), Proceedings of the 26th International Conference on Computers in Education (ICCE 2018), 26–30 November 2018, Manila, Philippines (pp. 29–38). Asia-Pacific Society for Computers in Education. http://icce2018.ateneo.edu/wp-content/uploads/2018/12/C1-04.pdf
Knight, S., Wise, A. F., & Chen, B. (2017). Time for change: Why learning analytics needs temporal analysis. Journal of Learning Analytics, 4(3), 7–17. https://doi.org/10.18608/jla.2017.43.2
Lämsä, J., Hämäläinen, R., Koskinen, P., Viiri, J., & Lampi, E. (2021). What do we do when we analyse the temporal aspects of computer-supported collaborative learning? A systematic literature review. Educational Research Review, 33, 100387. https://doi.org/10.1016/j.edurev.2021.100387
Martinez, R., Yacef, K., Kay, J., Al-Qaraghuli, A., & Kharrufa, A. (2011). Analysing frequent sequential patterns of collaborative learning activity around an interactive tabletop. In M. Pechenizkiy et al. (Eds.), Proceedings of the 4th Annual Conference on Educational Data Mining (EDM2011), 6–8 July 2011, Eindhoven, Netherlands (pp. 111–120). International Educational Data Mining Society. https://educationaldatamining.org/EDM2011/wp-content/uploads/proc/edm11_proceedings.pdf
Martinez-Maldonado, R., Dimitriadis, Y., Martinez-Monés, A., Kay, J., & Yacef, K. (2013). Capturing and analyzing verbal and physical collaborative learning interactions at an enriched interactive tabletop. International Journal of Computer-Supported Collaborative Learning, 8(4), 455–485. https://doi.org/10.1007/s11412-013-9184-1
Martinez-Maldonado, R., Kay, J., Buckingham Shum, S., & Yacef, K. (2019). Collocated collaboration analytics: Principles and dilemmas for mining multimodal interaction data. Human–Computer Interaction, 34(1), 1–50. https://doi.org/10.1080/07370024.2017.1338956
Mercier, E., Vourloumi, G., & Higgins, S. (2017). Student interactions and the development of ideas in multi-touch and paper-based collaborative mathematical problem solving. British Journal of Educational Technology, 48(1), 162–175. https://doi.org/10.1111/bjet.12351
Molenaar, I., & Wise, A. F. (2022). Temporal aspects of learning analytics: Grounding analyses in concepts of time. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), Handbook of learning analytics (pp. 66–76). https://doi.org/10.18608/hla22.007
Mushtaq, R. (2011, August 17). Augmented Dickey Fuller test. Econometrics. https://doi.org/10.2139/ssrn.1911068
Nizam, S. S. M., Zainal Abidin, R., Che Hashim, N., Lam, M. C., Arshad, H., & Abd Majid, N. A. (2018). A review of multimodal interaction technique in augmented reality environment. International Journal on Advanced Science, Engineering and Information Technology, 8(4–2), 1460–1469. https://doi.org/10.18517/ijaseit.8.4-2.6824
Noroozi, O., Alikhani, I., Järvelä, S., Kirschner, P. A., Juuso, I., & Seppänen, T. (2019). Multimodal data to design visual learning analytics for understanding regulation of learning. Computers in Human Behavior, 100, 298–304. https://doi.org/10.1016/j.chb.2018.12.019
Nystrand, M., Wu, L. L., Gamoran, A., Zeiser, S., & Long, D. A. (2003). Questions in time: Investigating the structure and dynamics of unfolding classroom discourse. Discourse Processes, 35(2), 135–198. https://doi.org/10.1207/S15326950DP3502_3
Phon, D. N. E., Ali, M. B., & Halim, N. D. A. (2014). Collaborative augmented reality in education: A review. 2014 International Conference on Teaching and Learning in Computing and Engineering, 11–13 April 2014, Kuching, Malaysia (pp. 78–83). IEEE Computer Society. https://doi.org/10.1109/LaTiCE.2014.23
Planey, J., Rajarathinam, R. J., Mercier, E., & Lindgren, R. (2023). Gesture-mediated collaboration with augmented reality headsets in a problem-based astronomy task. International Journal of Computer-Supported Collaborative Learning, 18(2), 259–289. https://doi.org/10.1007/s11412-023-09398-w
Praharaj, S., Scheffel, M., Drachsler, H., & Specht, M. (2021). Literature review on co-located collaboration modeling using multimodal learning analytics: Can we go the whole nine yards? IEEE Transactions on Learning Technologies, 14(3), 367–385. https://doi.org/10.1109/TLT.2021.3097766
Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning, 4(3), 239–257. https://doi.org/10.1007/s11412-009-9070-z
Sarmiento, J. W., & Stahl, G. (2008). Extending the joint problem space: Time and sequence as essential features of knowledge building. Proceedings of the 8th International Conference on the Learning Sciences (ICLS ’08), 24–28 June 2008, Utrecht, Netherlands (Vol. 3, pp. 295–302). International Society of the Learning Sciences. http://www.gerrystahl.net/pub/icls2008johann.pdf
Schneider, B., Worsley, M., & Martinez-Maldonado, R. (2021). Gesture and gaze: Multimodal data in dyadic interactions. In U. Cress, C. Rosé, A. F. Wise, & J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning (pp. 625–641). Springer. https://doi.org/10.1007/978-3-030-65291-3_34
Shaffer, D. W., Hatfield, D., Svarovsky, G. N., Nash, P., Nulty, A., Bagley, E., Frank, K., Rupp, A. A., & Mislevy, R. (2009). Epistemic network analysis: A prototype for 21st-century assessment of learning. International Journal of Learning and Media, 1(2), 33–53. https://doi.org/10.1162/ijlm.2009.0013
Sharma, K., & Giannakos, M. (2020). Multimodal data capabilities for learning: What can multimodal data tell us about learning? British Journal of Educational Technology, 51(5), 1450–1484. https://doi.org/10.1111/bjet.12993
Siposova, B., & Carpenter, M. (2019). A new look at joint attention and common knowledge. Cognition, 189, 260–274. https://doi.org/10.1016/j.cognition.2019.03.019
Stahl, G. (2006). Group cognition: Computer support for building collaborative knowledge. The MIT Press. https://doi.org/10.7551/mitpress/3372.001.0001
Syczewska, E. M. (2010). Empirical power of the Kwiatkowski-Phillips-Schmidt-Shin test. In Working Papers 45, Department of Applied Econometrics, Warsaw School of Economics. https://ideas.repec.org/p/wse/wpaper/45.html
Tan, S. C., Wang, X., & Li, L. (2022). The development trajectory of shared epistemic agency in online collaborative learning: A study combing network analysis and sequential analysis. Journal of Educational Computing Research, 59(8), 1655–1681. https://doi.org/10.1177/07356331211001562
Teasley, S. D., & Roschelle, J. (1993). Constructing a joint problem space: The computer as a tool for sharing knowledge. In S. P. Lajoie & S. J. Derry (Eds.), Computers as Cognitive Tools (pp. 229–258). Routledge.
Volet, S., Vauras, M., Salo, A.-E., & Khosa, D. (2017). Individual contributions in student-led collaborative learning: Insights from two analytical approaches to explain the quality of group outcome. Learning and Individual Differences, 53, 79–92. https://doi.org/10.1016/j.lindif.2016.11.006
Worsley, M. (2014). Multimodal learning analytics as a tool for bridging learning theory and complex learning behaviors. Proceedings of the 2014 ACM Workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14), 12–16 November 2014, Istanbul, Turkey (pp. 1–4). ACM Press. https://doi.org/10.1145/2666633.2666634
Zheng, J., Xing, W., & Zhu, G. (2019). Examining sequential patterns of self- and socially shared regulation of STEM learning in a CSCL environment. Computers & Education, 136, 34–48. https://doi.org/10.1016/j.compedu.2019.03.005
Zhou, G., Moulder, R., Sun, C., & D’Mello, S. (2022). Investigating temporal dynamics underlying successful collaborative problem solving behaviors with multilevel vector autoregression. In A. Mitrovic & N. Bosch (Eds.), Proceedings of the 15th International Conference on Educational Data Mining (EDM2022), 24–27 July 2022, Durham, UK (pp. 290–301). International Educational Data Mining Society. https://doi.org/10.5281/zenodo.6853137
Zhou, Y., & Kang, J. (2022). Characterizing joint attention dynamics during collaborative problem-solving in an immersive astronomy simulation. In A. Mitrovic & N. Bosch (Eds.), Proceedings of the 15th International Conference on Educational Data Mining (EDM2022), 24–27 July 2022, Durham, UK (pp. 406–413). International Educational Data Mining Society. https://doi.org/10.5281/zenodo.6852988
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
