Deep Networks for Collaboration Analytics
Promoting Automatic Analysis of Face-to-Face Interaction in the Context of Inquiry-Based Learning
DOI:
https://doi.org/10.18608/jla.2021.7118Keywords:
collaboration analytics, computational models, computer-supported collaborative learning, CSCL, CSCIL, deep networks, inquiry-based learning, word embeddingAbstract
Scholars have applied automatic content analysis to study computer-mediated communication in computer-supported collaborative learning (CSCL). Since CSCL also takes place in face-to-face interactions, we studied the automatic coding accuracy of manually transcribed face-to-face communication. We conducted our study in an authentic higher-education physics context where computer-supported collaborative inquiry-based learning (CSCIL) is a popular pedagogical approach. Since learners’ needs for support in CSCIL vary in the different inquiry phases (orientation, conceptualization, investigation, conclusion, and discussion), we studied, first, how the coding accuracy of five computational models (based on word embeddings and deep neural networks with attention layers) differed in the various inquiry-based learning (IBL) phases when compared to human coding. Second, we investigated how the different features of the best performing computational model improved the coding accuracy. The study indicated that the accuracy of the best performing computational model (differentiated attention with pre-trained static embeddings) was slightly better than that of the human coder (58.9% vs. 54.3%). We also found that considering the previous and following utterances, as well as the relative position of the utterance, improved the model’s accuracy. Our method illustrates how computational models can be trained for specific purposes (e.g., to code IBL phases) with small data sets by using pre-trained models.
References
Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011). Does discovery-based instruction enhance learning? Journal of Educational Psychology, 103(1), 1–18. https://doi.org/10.1037/a0021017
Bell, T., Urhahne, D., Schanze, S., & Ploetzner, R. (2010). Collaborative inquiry learning: Models, tools, and challenges. International Journal of Science Education, 32(3), 349–377. https://doi.org/10.1080/09500690802582241
Bybee, R. W., Taylor, J. A., Gardner, A., Van Scotter, P., Powell, J. C., Westbrook, A., & Landes, N. (2006). The BSCS 5E instructional model: Origins and effectiveness. Biological Sciences Curriculum Study.
de Jong, T., & Lazonder, A. W. (2014). The guided discovery learning principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 371–390). Cambridge, UK: Cambridge University Press.
Dönmez, P., Rosé, C., Stegmann, K., Weinberger, A., & Fischer, F. (2005). Supporting CSCL with automatic corpus analysis technology. In T. Koschmann, D. D. Suthers, & T.-W. Chan (Eds.), Learning 2005: The Next 10 Years! Proceedings of the 2005 Conference on Computer Support for Collaborative Learning (CSCL ’05), May 30–June 4 2005, Taipei, Taiwan (pp. 125–134). International Society of the Learning Sciences.
Donnelly, D. F., Linn, M. C., & Ludvigsen, S. (2014). Impacts and characteristics of computer-based science inquiry learning environments for precollege students. Review of Educational Research, 84(4), 572–608. https://doi.org/10.3102/0034654314546954
Espinoza, C., Lämsä, J., Araya, R., Hämäläinen, R., Jiménez, A. G., Gormaz, R., & Viiri, J. (2019). Automatic content analysis in collaborative inquiry-based learning. In O. Levrini & G. Tasquier (Eds.), The Beauty and Pleasure of Understanding: Engaging with Contemporary Challenges Through Science Education. Proceedings of the European Science Education Research Association Conference (ESERA 2019), 26–30 August, 2019, Bologna, Italy (Part 18, pp. 2041–2050). University of Bologna. https://www.esera.org/publications/esera-conference-proceedings/esera-2019
Hämäläinen, R., De Wever, B., Waaramaa, T., Laukkanen, A., & Lämsä, J. (2018). It’s not only what you say, but how you say it: Investigating the potential of prosodic analysis as a method to study teacher’s talk. Frontline Learning Research, 6(3), 204–227. https://doi.org/10.14786/flr.v6i3.371
Henri, F. (1992). Computer conferencing and content analysis. In A. R. Kaye (Ed.), Collaborative learning through computer conferencing: The Najadan Papers (pp. 117–136). Springer-Verlag.
Howard, S. K., Thompson, K., Yang, J., Ma, J., Pardo, A., & Kanasa, H. (2017). Capturing and visualizing: Classroom analytics for physical and digital collaborative learning processes. In B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (Eds.), Making a Difference: Prioritizing Equity and Access in CSCL. Proceedings of the 12th International Conference on Computer Supported Collaborative Learning (CSCL 2017) 18–22 June 2017, Philadelphia, PA, USA (Vol. 2, pp. 801–802). International Society of the Learning Sciences.
Hu, D. (2020). An introductory survey on attention mechanisms in NLP problems. In Y. Bi, R. Bhatia, & S. Kapoor (Eds.), Intelligent systems and applications: Proceedings of the 2019 Intelligent Systems Conference (IntelliSys) (pp. 432–448). Springer.
Jeong, H., Hmelo-Silver, C. E., & Jo, K. (2019). Ten years of computer-supported collaborative learning: A meta-analysis of CSCL in STEM education during 2005–2014. Educational Research Review, 28, 100284. https://doi.org/10.1016/j.edurev.2019.100284
Kim, Y., Denton, C., Hoang, L., & Rush, A. M. (2017). Structured attention networks. https://arxiv.org/abs/1702.00887
Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. https://arxiv.org/abs/1412.6980
Knight, S., & Littleton, K. (2015). Discourse centric learning analytics: Mapping the terrain. Journal of Learning Analytics, 2(1), 185–209. https://doi.org/10.18608/jla.2015.21.9
Koskinen, P., Lämsä, J., Maunuksela, J., Hämäläinen, R., & Viiri, J. (2018). Primetime learning: Collaborative and technology-enhanced studying with genuine teacher presence. International Journal of STEM Education, 5(20), 1–13. https://doi.org/10.1186/s40594-018-0113-8
Kotsiantis, S. B. (2007). Supervised machine learning: A review of classification techniques. Informatica (Ljubljana), 31(3), 249–268.
Kronholm, H., Caballero, D., Mansikkaniemi, A., Araya, R., Lehesvuori, S., Pertilä, P., Virtanen, T., Kurimo, M., & Viiri, J. (2017). The automatic analysis of classroom talk. Proceedings of the Annual FMSERA Symposium 2016, Joensuu, Finland (pp. 142–151). https://journal.fi/fmsera/article/view/60940/27049
Lämsä, J., Hämäläinen, R., Koskinen, P., Viiri, J., & Mannonen, J. (2020). The potential of temporal analysis: Combining log data and lag sequential analysis to investigate temporal differences between scaffolded and non-scaffolded group inquiry-based learning processes. Computers & Education, 143, 103674. https://doi.org/10.1016/j.compedu.2019.103674
Lämsä, J., Hämäläinen, R., Koskinen, P., & Viiri, J. (2018). Visualising the temporal aspects of collaborative inquiry-based learning processes in technology-enhanced physics learning. International Journal of Science Education, 40(14), 1697–1717. https://doi.org/10.1080/09500693.2018.1506594
Lemke, J. L. (2000). Across the scales of time: Artifacts, activities, and meanings in ecosocial systems. Mind, Culture, and Activity, 7(4), 273–290. https://doi.org/10.1207/S15327884MCA0704_03
Luotolahti, J., Kanerva, J., Laippala, V., Pyysalo, S., & Ginter, F. (2015). Towards universal web parsebanks. Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), 24–26 August 2015, Uppsala, Sweden (pp. 211–220). https://www.aclweb.org/anthology/W15-21
Maldonado, R. M., Worsley, M., Schneider, B., & Kharrufa, A. (2019). International workshop on collaboration analytics: Making learning visible in collaborative settings. In K. Lund, G. Niccolai, E. Lavoué, C. Hmelo-Silver, G. Gweon, & M. Baker (Eds.), A Wide Lens: Combining Embodied, Enactive, Extended, and Embedded Learning in Collaborative Settings. Proceedings of the 13th International Conference on Computer Supported Collaborative Learning (CSCL 2019), 17–21 June 2019, Lyon, France (Vol. 2, p. 1044). International Society of the Learning Sciences.
Matuk, C., & Linn, M. C. (2018). Why and how do middle school students exchange ideas during science inquiry? International Journal of Computer-Supported Collaborative Learning, 13(3), 263–299. https://doi.org/10.1007/s11412-018-9282-1
Mercer, N. (2008). The seeds of time: Why classroom dialogue needs a temporal analysis. Journal of the Learning Sciences, 17(1), 33–59. https://doi.org/10.1080/10508400701793182
Mercer, N., Wegerif, R., & Dawes, L. (1999). Children’s talk and the development of reasoning in the classroom. British Educational Research Journal, 25(1), 95–111. https://doi.org/10.1080/0141192990250107
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. https://arxiv.org/abs/1301.3781
Mu, J., Stegmann, K., Mayfield, E., Rosé, C., & Fischer, F. (2012). The ACODEA framework: Developing segmentation and classification schemes for fully automatic analysis of online discussions. International Journal of Computer-Supported Collaborative Learning, 7(2), 285–305. https://doi.org/10.1007/s11412-012-9147-y
Neuendorf, K. A. (2002). The content analysis: Guidebook (1st ed.). Thousand Oaks, CA: Sage Publications.
Noroozi, O., Alikhani, I., Järvelä, S., Kirschner, P. A., Juuso, I., & Seppänen, T. (2019). Multimodal data to design visual learning analytics for understanding regulation of learning. Computers in Human Behavior, 100, 298–304. https://doi.org/10.1016/j.chb.2018.12.019
Olsen, J. K., Sharma, K., Rummel, N., & Aleven, V. (2020). Temporal analysis of multimodal data to predict collaborative learning outcomes. British Journal of Educational Technology, 51(5), 1527–1547. https://doi.org/10.1111/bjet.12982
Pedaste, M., Mäeots, M., Siiman, L. A., de Jong, T., van Riesen, S. A. N., Kamp, E. T., Manoli, C. C., Zacharia, Z. C., & Tsourlidaki, E. (2015). Phases of inquiry-based learning: Definitions and the inquiry cycle. Educational Research Review, 14, 47–61. https://doi.org/10.1016/j.edurev.2015.02.003
Pennebaker, J. W., Boyd, R. L., Jordan, K., & Blackburn, K. (2015). The development and psychometric properties of LIWC2015. University of Texas at Austin. http://hdl.handle.net/2152/31333
Rosé, C., Wang, Y., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., & Fischer, F. (2008). Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal of Computer-Supported Collaborative Learning, 3(3), 237–271. https://doi.org/10.1007/s11412-007-9034-0
Schneider, B., & Blikstein, P. (2014). Unraveling students’ interaction around a tangible interface using gesture recognition. In J. Stamper et al. (Eds.), Proceedings of the 7th International Conference on Educational Data Mining (EDM2014), 4–7 July 2014, London, UK (pp. 320–323). International Educational Data Mining Society. https://doi.org/10.5281/zenodo.3554729
Schneider, B., & Pea, R. (2013). Real-time mutual gaze perception enhances collaborative learning and collaboration quality. International Journal of Computer-Supported Collaborative Learning, 8(4), 375–397. https://doi.org/10.1007/s11412-013-9181-4
Schneider, B., & Pea, R. (2015). Does seeing one another’s gaze affect group dialogue? A computational approach. Journal of Learning Analytics, 2(2), 107–133. https://doi.org/10.18608/jla.2015.22.9
Sins, P. H. M., Savelsbergh, E. R., van Joolingen, W. R., & van Hout-Wolters, B. H. A. M. (2011). Effects of face-to-face versus chat communication on performance in a collaborative inquiry modeling task. Computers & Education, 56(2), 379–387. https://doi.org/10.1016/j.compedu.2010.08.022
Smith, J., Bratt, H., Richey, C., Bassiou, N., Shriberg, E., Tsiartas, A., D’Angelo, C., & Alozie, N. (2016). Spoken interaction modeling for automatic assessment of collaborative learning. Proceedings of Speech Prosody 8, 31 May–3 June 2016, Boston, MA, USA (pp. 277–281). https://doi.org/10.21437/SpeechProsody.2016-57
Tan, E. (2018). Effects of two differently sequenced classroom scripts on common ground in collaborative inquiry learning. Instructional Science, 46(6), 893–919. https://doi.org/10.1007/s11251-018-9460-6
Thompson, K., Ashe, D., Carvalho, L., Goodyear, P., Kelly, N., & Parisio, M. (2013). Processing and visualizing data in complex learning environments. American Behavioral Scientist, 57(10), 1401–1420. https://doi.org/10.1177/0002764213479368
Tshitoyan, V., Dagdelen, J., Weston, L., Dunn, A., Rong, Z., Kononova, O., Persson, K. A., Ceder, G., & Jain, A. (2019). Unsupervised word embeddings capture latent knowledge from materials science literature. Nature, 571(7763), 95–98. https://doi.org/10.1038/s41586-019-1335-8
van Joolingen, W. R., de Jong, T., Lazonder, A. W., Savelsbergh, E. R., & Manlove, S. (2005). Co-Lab: Research and development of an online learning environment for collaborative scientific discovery learning. Computers in Human Behavior, 21(4), 671–688. https://doi.org/10.1016/j.chb.2004.10.039
van Leeuwen, A. (2015). Learning analytics to support teachers during synchronous CSCL: Balancing between overview and overload. Journal of Learning Analytics, 2(2), 138–162. https://doi.org/10.18608/jla.2015.22.11
White, B. Y., & Frederiksen, J. R. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition and Instruction, 16(1), 3–118. https://doi.org/10.1207/s1532690xci1601_2
Worsley, M., & Blikstein, P. (2015). Using learning analytics to study cognitive disequilibrium in a complex learning environment. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 426–427). New York: ACM. https://doi.org/10.1145/2723576.2723659
Xing, W., Popov, V., Zhu, G., Horwitz, P., & McIntyre, C. (2019). The effects of transformative and non-transformative discourse on individual performance in collaborative-inquiry learning. Computers in Human Behavior, 98, 267–276. https://doi.org/10.1016/j.chb.2019.04.022
Zacharia, Z. C., Manoli, C., Xenofontos, N., de Jong, T., Pedaste, M., van Riesen, S. A. N., Kamp, E. T., Mäeots, M., Siiman, L., & Tsourlidaki, E. (2015). Identifying potential types of guidance for supporting student inquiry when using virtual and remote labs in science: A literature review. Educational Technology Research and Development, 63(2), 257–302. https://doi.org/10.1007/s11423-015-9370-0
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
