A Novel Method for the In-Depth Multimodal Analysis of Student Learning Trajectories in Intelligent Tutoring Systems
DOI:
https://doi.org/10.18608/jla.2018.51.4Keywords:
Learning trajectories, learning curves, intelligent tutoring systems, science learning, multimodal data, mixed methodsAbstract
Temporal analyses are critical to understanding learning processes, yet understudied in education research. Data from different sources are often collected at different grain sizes, which are difficult to integrate. Making sense of data at many levels of analysis, including the most detailed levels, is highly time-consuming. In this paper, we describe a generalizable approach for more efficient yet rich sensemaking of temporal data during student use of intelligent tutoring systems. This multi-step approach involves using coarse-grain temporality — learning trajectories across knowledge components — to identify and further explore “focal” moments worthy of more fine-grain, context-rich analysis. We discuss the application of this approach to data collected from a classroom study in which students engaged in a Chemistry Virtual Lab tutoring system. We show that the application of this multi-step approach efficiently led to interpretable and actionable insights while making use of the richness of the available data. This method is generalizable to many types of datasets and can help handle large volumes of rich data at multiple levels of granularity. We argue that it can be a valuable approach to tackling some of the most prohibitive methodological challenges involved in temporal learning analytics
References
Barbera, E., Gros, B., & Kirschner, P. A. (2015). Paradox of time in research on educational technology. Time & Society, 24(1), 96-108.
Blikstein, P. (2013). Multimodal learning analytics. In Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13). ACM, New York, NY, pp. 102-106.
Blikstein, P., & Worsley, M. (2016). Multimodal Learning Analytics and Education Data Mining: using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220-238.
Booth, J.L., Newton, K.J., Twiss-Garrity, L. (2014). The impact of fraction magnitude knowledge on algebra performance and learning. Journal of Experimental Child Psychology, 118, 110-118.
Bozdogan, H. (1987). Model Selection and Akaike’s Information Criterion (AIC): The General Theory and Its Analytical Extensions. Psychometrika, 52, 345–370.
Butler, D. L. & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of educational research, 65(3): 245–281.
Cen, H., Koedinger, K. R., and Junker, B. (2006). Learning Factors Analysis: A general method for cognitive model evaluation and improvement. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems, pp. 164-175. Berlin: Springer-Verlag.
Datavyu Team. (2014). Datavyu: A Video Coding Tool. Databrary Project, New York University. http://datavyu.org.
Davenport, J.L., Rafferty, A., Karabinos, M., & Yaron, D. (2015). Investigating Chemistry Learning Using Virtual Lab Activities in Real Classrooms. Paper presented in the symposium Theory Driven Design of Online Learning Systems, Data Structures and Analytic Techniques at the Annual Meeting of the American Educational Research Association.
Davenport, J.L., Rafferty, A., Yaron, D, Karabinos, M., & Timms, M. (2014). ChemVLab+: Simulation-based Lab activities to support chemistry learning. Proceedings of the Annual Meeting of the American Educational Research Association.
D’Mello, S.K., & Graesser, A. (2010). Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modeling and User-Adapted Interaction, 20(2), 147-187.
Graesser, A.C., Conley, M., and Olney, A. (2012). Intelligent tutoring systems. In K.R. Harris, S. Graham, and T. Urdan (Eds.), APA Educational Psychology Handbook: Vol. 3. Applications to Learning and Teaching. Washington, DC: American Psychological Association, pp. 451-473.
Hoz, R., Bowman, D., & Kozminsky, E. (2001). The differential effects of prior knowledge on learning: A study of two consecutive courses in earth sciences. Instructional Science, 29(3), 187-211.
Koedinger, K. R., Baker, R. S., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J. (2010). A data repository for the EDM community: The PSLC DataShop. Handbook of educational data mining, 43.
Koedinger, K.R., Corbett, A.T., & Perfetti, C. (2012). The Knowledge‐Learning‐Instruction framework: Bridging the science‐practice chasm to enhance robust student learning. Cognitive science, 36(5), 757-798.
Kubricht, J. R., Holyoak, K. J., & Lu, H. (2017). Intuitive Physics: Current Research and Controversies. Trends in Cognitive Sciences.
Liu, R., Davenport, J., & Stamper, J. (2016). Beyond Log Files: Using Multi-Modal Data Streams Towards Data-Driven KC Model Improvement. In Proceedings of the 9th International Conference on Educational Data Mining (EDM ’16).
Mercer, N. (2008). The seeds of time: why classroom dialogue needs a temporal analysis. Journal of the Learning Sciences, 17(1): 33–59.
Ocumpaugh, J., Baker, R.S.J.d., Gowda, S., Heffernan, N., & Heffernan, C. (2014). Population validity for Educational Data Mining models: A case study in affect detection. British Journal of Educational Technology, 45(3), 487-501.
Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129-184.
Stamper, J. & Koedinger, K. R. Human-machine student model discovery and improvement using data. (2011). In Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 353–360. Auckland, New Zealand.
Taber, K. S. (2013). Revisiting the chemistry triplet: drawing upon the nature of chemical knowledge and the psychology of learning to inform chemistry education. Chemistry Education Research and Practice, 14(2), 156-168.
Worsley, M. (2014). Multimodal Learning Analytics as a Tool for Bridging Learning Theory and Complex Learning Behaviors. In Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge (pp. 1-4). ACM.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2018 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
