Analyzing and Visualizing Learning Data: A System Designer's Perspective
DOI:
https://doi.org/10.18608/jla.2021.7345Keywords:
difficulty, similarity, response time, temporal patterns, biases, cheatingAbstract
In this work, we consider learning analytics for primary and secondary schools from the perspective of the designer of a learning system. We provide an overview of practically useful analytics techniques with descriptions of their applications and specific illustrations. We highlight data biases and caveats that complicate the analysis and its interpretation. Although we intentionally focus on techniques for internal use by designers, many of these techniques may inspire the development of dashboards for teachers or students. We also identify the consequences and challenges for research.
References
Aleven, V., McLaughlin, E. A., Glenn, R. A., & Koedinger, K. R. (2016). Instruction based on adaptive learning technologies. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of Research on Learning and Instruction. New York: Routledge. https://doi.org/10.4324/9781315736419
Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., & Koedinger, K. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185–224. Retrieved from https://psycnet.apa.org/record/2008-07015-001
Baker, R. S. (2016). Stupid tutoring systems, intelligent humans. International Journal of Artificial Intelligence in Education, 26(2), 600–614. https://doi.org/10.1007/s40593-016-0105-0
Brennan, R. L. (1972). A generalized upper-lower item discrimination index. Educational and Psychological Measurement, 32(2), 289–303. https://doi.org/10.1177/001316447203200206
Bull, S., & Kay, J. (2007). Student models that invite the learner in: The SMILI open learner modelling framework. International Journal of Artificial Intelligence in Education, 17(2), 89–120. Retrieved from https://dl.acm.org/doi/10.5555/1435369.1435371
Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. In Proceedings of the Second International Conference on Learning Analytics and Knowledge (LAK 2012), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 134–138). New York: ACM. https://doi.org/10.1145/2330601.2330636
Corrigan-Gibbs, H., Gupta, N., Northcutt, C., Cutrell, E., & Thies, W. (2015). Deterring cheating in online environments. ACM Transactions on Computer-Human Interaction (TOCHI), 22(6), 1–23. https://doi.org/10.1145/2810239
Davidson, J., Liebald, B., Liu, J., Nandy, P., Van Vleet, T., Gargi, U., . . . Sampath, D. (2010). The YouTube video recommendation system. In Proceedings of the Fourth ACM Conference on Recommender Systems (RecSys 2010), 26–30 September 2010, Barcelona, Spain (pp. 293–296). New York: ACM. https://doi.org/10.1145/1864708.1864770
De Ayala, R. J. (2013). The Theory and Practice of Item Response Theory. New York: Guilford Publications. https://doi.org/10.1007/s11336-010-9179-z
Heffernan, N. T., & Heffernan, C. L. (2014). The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education, 24(4), 470–497. https://doi.org/10.1007/s40593-014-0024-x
Inventado, P. S., Inventado, S. G. F., Matsuda, N., Li, Y., Scupelli, P., Ostrow, K., . . . McGuire, P. (2018). Using design patterns for math preservice teacher education. In Proceedings of the 23rd European Conference on Pattern Languages of Programs (EuroPLoP 2018), 4–8 July 2018, Irsee, Germany (pp. 1–8). New York: ACM. https://doi.org/10.1145/3282308.3282340
Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2017). Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005), 15–19 August 2005, Salvador, Brazil (Vol. 51, pp. 4–11). New York: ACM. https://doi.org/10.1145/1076034.1076063
Klinkenberg, S., Straatemeier, M., & Van der Maas, H. (2011). Computer adaptive practice of Maths ability using a new item response model for on the fly ability and difficulty estimation. Computers & Education, 57(2), 1813–1824. https://doi.org/10.1016/j.compedu.2011.02.003
Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012). The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36(5), 757–798. https://doi.org/10.1111/j.1551-6709.2012.01245.x
Koenigstein, N., Dror, G., & Koren, Y. (2011). Yahoo! music recommendations: Modeling music ratings with temporal dynamics and item taxonomy. In Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys 2011), 23–27 October 2011, Chicago, IL, USA (pp. 165–172). New York: ACM. https://doi.org/10.1145/2043932.2043964
Koren, Y., & Bell, R. (2015). Advances in collaborative filtering. In F. Ricci, L. Rokach, B. Shapira, & P. B. Kantor (Eds.), Recommender Systems Handbook (pp. 77–118). Springer. https://doi.org/10.1007/978-0-387-85820-3_5
Molenaar, I., & Knoop-van Campen, C. (2017). Teacher dashboards in practice: Usage and impact. In Proceedings of the 2017 European Conference on Technology Enhanced Learning (ECTEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 125–138). Springer. https://doi.org/10.1007/978-3-319-66610-5_10
Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012). A unifying view on dataset shift in classification. Pattern Recognition, 45(1), 521–530. https:/doi.org/10.1016/j.patcog.2011.06.019
Northcutt, C. G., Ho, A. D., & Chuang, I. L. (2016). Detecting and preventing “multiple-account” cheating in massive open online courses. Computers & Education, 100, 71–80. https://doi.org/10.1016/j.compedu.2016.04.008
Pelánek, R. (2017). Bayesian knowledge tracing, logistic models, and beyond: An overview of learner modeling techniques. User Modeling and User-Adapted Interaction, 27(3), 313–350. https://doi.org/10.1007/s11257-017-9193-2
Pelánek, R. (2018). The details matter: Methodological nuances in the evaluation of student models. User Modeling and User-Adapted Interaction, 28(3), 207–235. https://doi.org/10.1007/s11257-018-9204-y
Pelánek, R. (2020a). Managing items and knowledge components: Domain modeling in practice. Educational Technology Research and Development, 68(1), 529–550. https://doi.org/10.1007/s11423-019-09716-w
Pelánek, R. (2020b). Measuring similarity of educational items: An overview. IEEE Transactions on Learning Technologies, 13(2), 354–366. https://doi.org/10.1109/TLT.2019.2896086
Pelánek, R., & Rihák, J. (2018). Analysis and design of mastery learning criteria. New Review of Hypermedia and Multimedia, 24(3), 133–159. https://doi.org/10.1080/13614568.2018.1476596
Pelánek, R., & Rihák, J. (2016). Properties and applications of wrong answers in online educational systems. In Proceedings of the Ninth International Conference on Educational Data Mining (EDM 2016), 29 June–2 July 2016, Raleigh, NC, USA (pp. 466–471). Retrieved from https://www.educationaldatamining.org/EDM2016/proceedings/edm2016shortpapers.pdf
Ruipérez-Valiente, J. A., Muñoz-Merino, P. J., Leony, D., & Kloos, C. D. (2015). ALAS-KA: A learning analytics extension for better understanding the learning process in the Khan Academy platform. Computers in Human Behavior, 47, 139–148. https://doi.org/10.1016/j.chb.2014.07.002
Van Der Linden, W. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46(3), 247–272. https://doi.org/10.1111/j.1745-3984.2009.00080.x
Wang, Y., Heffernan, N. T., & Heffernan, C. (2015). Towards better affect detectors: Effect of missing skills, class features and common wrong answers. In J. Baron, G. Lynch, N. Maziarz, P. Blikstein, A. Merceron, & G. Siemens (Eds.), Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK 2015), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 31–35). New York: ACM. https://doi.org/10.1145/2723576.2723618
Wauters, K., Desmet, P., & Van Den Noortgate, W. (2012). Item difficulty estimation: An auspicious collaboration between data and judgment. Computers & Education, 58(4), 1183–1193. https://doi.org/10.1016/j.compedu.2011.11.020
Xhakaj, F., Aleven, V., & McLaren, B. M. (2017). Effects of a teacher dashboard for an intelligent tutoring system on teacher knowledge, lesson planning, lessons and student learning. In Proceedings of the 2017 European Conference on Technology Enhanced Learning (ECTEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 315–329). Springer. https://doi.org/10.1007/978-3-319-66610-5_23
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
