Effectiveness of a Learning Analytics Dashboard for Increasing Student Engagement Levels
DOI:
https://doi.org/10.18608/jla.2023.7935Keywords:
learning analytics, dashboard, descriptive analytics, predictive and prescriptive analytics, dashboard evaluation, student engagement behaviours, usability, research paperAbstract
Learning Analytics Dashboards (LADs) are gaining popularity as a platform for providing students with insights into their learning behaviour patterns in online environments. Existing LAD studies are mainly centred on displaying students’ online behaviours with simplistic descriptive insights. Only a few studies have integrated predictive components, while none possess the ability to explain how the predictive models work and how they have arrived at specific conclusions for a given student. A further gap exists within existing LADs with respect to prescriptive analytics that generate data-driven feedback to students on how to adjust their learning behaviour. The LAD in this study attempts to address this gap and integrates a full spectrum of current analytics technologies for sense-making while anchoring them within theoretical educational frameworks. This study’s LAD (SensEnablr) was evaluated for its effectiveness in impacting learning in a student cohort at a tertiary institution. Our findings demonstrate that student engagement with learning technologies and course resources increased significantly immediately following interactions with the dashboard. Meanwhile, results showed that the dashboard boosted the respondents’ learning motivation levels and that the novel analytics insights drawn from predictive and prescriptive analytics were beneficial to their learning. This study, therefore, has implications for future research when investigating student outcomes and optimizing student learning using LAD technologies.
References
Afzaal, M., Nouri, J., Zia, A., Papapetrou, P., Fors, U., Wu, Y., Li, X., & Weegar, R. (2021). Explainable AI for data-driven feedback and intelligent action recommendations to support students self-regulation. Frontiers in Artificial Intelligence, 4. https://doi.org/10.3389/frai.2021.723447
Ahadi, A., Lister, R., Haapala, H., & Vihavainen, A. (2015). Exploring machine learning methods to automatically identify students in need of assistance. Proceedings of the 11th Annual Conference on International Computing Education Research (ICER ’15), 9–13 July 2015, Omaha, NE, USA (pp. 121–130). ACM Press. https://doi.org/10.1145/2787622.2787717
Ali, L., Asadi, M., Gašević, D., Jovanović, J., & Hatala, M. (2013). Factors influencing beliefs for adoption of a learning analytics tool: An empirical study. Computers & Education, 62, 130–148. https://doi.org/10.1016/j.compedu.2012.10.023
Arnold, K. E., & Pistilli, M. D. (2012). Course Signals at Purdue: Using learning analytics to increase student success. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ʼ12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 267–270). ACM Press. https://doi.org/10.1145/2330601.2330666
Bandura, A. (1986). The explanatory and predictive scope of self-efficacy theory. Journal of Social and Clinical Psychology, 4(3), 359–373. https://doi.org/10.1521/jscp.1986.4.3.359
Baneres, D., Rodriguez, M. E., & Serra, M. (2019). An early feedback prediction system for learners at-risk within a first-year higher education course. IEEE Transactions on Learning Technologies, 12(2), 249–263. https://doi.org/10.1109/TLT.2019.2912167
Bañeres, D., Rodríguez, M. E., Guerrero-Roldán, A. E., & Karadeniz, A. (2020). An early warning system to detect at-risk students in online higher education. Applied Sciences, 10(13), 4427. https://doi.org/10.3390/app10134427
Bodily, R., & Verbert, K. (2017). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies, 10(4), 405–418. https://doi.org/10.1109/TLT.2017.2740172
Bodily, R., Ikahihifo, T. K., Mackley, B., & Graham, C. R. (2018). The design, development, and implementation of student-facing learning analytics dashboards. Journal of Computing in Higher Education, 30(3), 572–598. https://doi.org/10.1007/s12528-018-9186-0
Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability Evaluation in Industry (pp.189–194). Taylor & Francis.
Brooke, J. (2013). SUS: A retrospective. Journal of Usability Studies, 8(2), 29–40.
Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324
Chatti, M. A., Muslim, A., Guliani, M., & Guesmi, M. (2020). The LAVA model: Learning analytics meets visual analytics. In D. Ifenthaler & D. Gibson (Eds.), Adoption of data analytics in higher education learning and teaching (pp. 71–93). Springer. https://doi.org/10.1007/978-3-030-47392-1_5
Cranton, P. (2006). Understanding and promoting transformative learning: A guide for educators of adults (2nd ed.). Jossey-Bass.
Domingos, P., & Pazzani, M. (1997). On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning, 29(2), 103–130. https://doi.org/10.1023/A:1007413511361
Dorogush, A. V., Ershov, V., & Gulin, A. (2018). CatBoost: Gradient boosting with categorical features support. http://arxiv.org/abs/1810.11363
Duan, X., Wang, C., & Rouamba, G. (2022). Designing a learning analytics dashboard to provide students with actionable feedback and evaluating its impacts. Proceedings of the 14th International Conference on Computer Supported Education (CSEDU ’22), 22–24 April 2022, Online (volume 2, pp. 117–127). SciTePress. https://doi.org/10.5220/0011116400003182
Fleur, D. S., van den Bos, W., & Bredeweg, B. (2020). Learning analytics dashboard for motivation and performance. In V. Kumar & C. Troussas (Eds.), Intelligent tutoring systems (pp. 411–419). Springer. https://doi.org/10.1007/978-3-030-49663-0_51
Hajian, S., Bonchi, F., & Castillo, C. (2016). Algorithmic bias: From discrimination discovery to fairness-aware data mining. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16), 13–17 August 2016, San Francisco, CA, USA (pp. 2125–2126). ACM Press. https://doi.org/10.1145/2939672.2945386
Han, J., Kim, K. H., Rhee, W., & Cho, Y. H. (2021). Learning analytics dashboards for adaptive support in face-to-face collaborative argumentation. Computers & Education, 163, 104041. https://doi.org/10.1016/j.compedu.2020.104041
Hechenbichler, K., & Schliep, K. (2004). Weighted k-nearest-neighbor techniques and ordinal classification. Collaborative Research Center 386, Discussion Paper 399. https://doi.org/10.5282/ubm/epub.1769
Hellings, J., & Haelermans, C. (2022). The effect of providing learning analytics on student behaviour and performance in programming: A randomised controlled experiment. Higher Education, 83(1), 1–18. https://doi.org/10.1007/s10734-020-00560-z
Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression. John Wiley & Sons. https://doi.org/10.1002/0471722146
Howell, J. A., Roberts, L. D., & Mancini, V. O. (2018). Learning analytics messages: Impact of grade, sender, comparative information and message style on student affect and academic resilience. Computers in Human Behavior, 89, 8–15. https://doi.org/10.1016/j.chb.2018.07.021
Hu, Y.-H., Lo, C.-L., & Shih, S.-P. (2014). Developing early warning systems to predict students’ online learning performance. Computers in Human Behavior, 36, 469–478. https://doi.org/10.1016/j.chb.2014.04.002
Islam, M. R., Ahmed, M. U., Barua, S., & Begum, S. (2022). A systematic review of explainable artificial intelligence in terms of different application domains and tasks. Applied Sciences, 12(3), 1353. https://doi.org/10.3390/app12031353
Jayaprakash, S. M., Moody, E. W., Lauría, E. J. M., Regan, J. R., & Baron, J. D. (2014). Early alert of academically at-risk students: An open source analytics initiative. Journal of Learning Analytics, 1(1), 6–47. https://doi.org/10.18608/jla.2014.11.3
Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness is not enough: Pitfalls of learning analytics dashboards in the educational practice. In É. Lavoué, H. Drachsler, K. Verbert, J. Broisin, & M. Pérez-Sanagustín (Eds.), Data Driven Approaches in Digital Education (pp. 82–96). Springer. https://doi.org/10.1007/978-3-319-66610-5_7
Jivet, I., Scheffel, M., Specht, M., & Drachsler, H. (2018). License to evaluate: Preparing learning analytics dashboards for educational practice. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 31–40). ACM Press. https://doi.org/10.1145/3170358.3170421
Khosravi, H., Shabaninejad, S., Bakharia, A., Sadiq, S., Indulska, M., & Gašević, D. (2021). Intelligent learning analytics dashboards: Automated drill-down recommendations to support teacher data exploration. Journal of Learning Analytics, 8(3), 133–154. https://doi.org/10.18608/jla.2021.7279
Kim, J., Jo, I.-H., & Park, Y. (2016). Effects of learning analytics dashboard: Analyzing the relations among dashboard utilization, satisfaction, and learning achievement. Asia Pacific Education Review, 17(1), 13–24. https://doi.org/10.1007/s12564-015-9403-8
Kitchenham, A. (2008). The evolution of John Mezirow’s transformative learning theory. Journal of Transformative Education, 6(2), 104–123. https://doi.org/10.1177/1541344608322678
Knight, D., Knight, B., Pearson, M., & Quintana, M. (2018). Microsoft Power BI quick start guide: Build dashboards and visualizations to make your data come to life. Packt Publishing.
Kokoç, M., & Altun, A. (2021). Effects of learner interaction with learning dashboards on academic performance in an e-learning environment. Behaviour & Information Technology, 40(2), 161–175. https://doi.org/10.1080/0144929X.2019.1680731
Li, Y., Gou, J., & Fan, Z. (2019). Educational data mining for students’ performance based on fuzzy C-means clustering. The Journal of Engineering, 2019(11), 8245–8250. https://doi.org/10.1049/joe.2019.0938
Lim, L., Dawson, S., Joksimovic, S., & Gašević, D. (2019). Exploring students’ sensemaking of learning analytics dashboards: Does frame of reference make a difference? Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (pp. 250–259). ACM Press. https://doi.org/10.1145/3303772.3303804
Lu, C., & Cutumisu, M. (2022). Online engagement and performance on formative assessments mediate the relationship between attendance and course performance. International Journal of Educational Technology in Higher Education, 19(1), 2. https://doi.org/10.1186/s41239-021-00307-5
Lundberg, S. M., Allen, P. G., & Lee, S.-I. (n.d.). A unified approach to interpreting model predictions. https://github.com/slundberg/shap
Lepenioti, K., Bousdekis, A., Apostolou, D., & Mentzas, G. (2020). Prescriptive analytics: Literature review and research challenges. International Journal of Information Management, 50, 57–70. https://doi.org/10.1016/j.ijinfomgt.2019.04.003
Marsh, H. W. (1986). Verbal and math self-concepts: An internal/external frame of reference model. American Educational Research Journal, 23(1), 129–149. https://doi.org/10.3102/00028312023001129
Marsh, H. W., Trautwein, U., Lüdtke, O., Köller, O., & Baumert, J. (2005). Academic self-concept, interest, grades, and standardized test scores: Reciprocal effects models of causal ordering. Child Development, 76(2), 397–416. https://doi.org/10.1111/j.1467-8624.2005.00853.x
Mathrani, A., Susnjak, T., Ramaswami, G., & Barczak, A. (2021). Perspectives on the challenges of generalizability, transparency and ethics in predictive learning analytics. Computers and Education Open, 2, 100060. https://doi.org/10.1016/j.caeo.2021.100060
Mezirow, J. (1997). Transformative learning: Theory to practice. New Directions for Adult and Continuing Education, 1997(74), 5–12. https://doi.org/10.1002/ace.7401
Namoun, A., & Alshanqiti, A. (2021). Predicting student performance using data mining and learning analytics techniques: A systematic literature review. Applied Sciences, 11(1). https://doi.org/10.3390/app11010237
Piaget, J. (1952). The origins of intelligence in children (M. Cook, trans.). International Universities Press. (Originally published 1936). https://doi.org/10.1037/11494-000
Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). Academic Press. https://doi.org/10.1016/B978-012109890-2/50043-3
de Quincey, E., Briggs, C., Kyriacou, T., & Waller, R. (2019). Student centred design of a learning analytics system. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (pp. 353–362). ACM Press. https://doi.org/10.1145/3303772.3303793
Rets, I., Herodotou, C., Bayer, V., Hlosta, M., & Rienties, B. (2021). Exploring critical factors of the perceived usefulness of a learning analytics dashboard for distance university students. International Journal of Educational Technology in Higher Education, 18(1), 46. https://doi.org/10.1186/s41239-021-00284-9
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why should I trust you?” Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16), 13–17 August 2016, San Francisco, CA, USA (pp. 1135–1144). ACM Press. https://doi.org/10.1145/2939672.2939778
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018). Anchors: High-precision model-agnostic explanations. Proceedings of the 30th Conference on Artificial Intelligence (AAAI-18), 2–7 February 2018, New Orleans, LA, USA (pp. 1527–1535). Palo Alto, CA: AAAI Press. https://doi.org/10.1609/aaai.v32i1.11491
Rienties, B., Herodotou, C., Olney, T., Schencks, M., & Boroowa, A. (2018). Making sense of learning analytics dashboards: A technology acceptance perspective of 95 teachers. The International Review of Research in Open and Distributed Learning, 19(5). https://doi.org/10.19173/irrodl.v19i5.3493
Roberts, L. D., Howell, J. A., & Seaman, K. (2017). Give me a customizable dashboard: Personalized learning analytics dashboards in higher education. Technology, Knowledge and Learning, 22(3), 317–333. https://doi.org/10.1007/s10758-017-9316-1
Ryan, T., & Henderson, M. (2018). Feeling feedback: Students’ emotional responses to educator feedback. Assessment & Evaluation in Higher Education, 43(6), 880–892. https://doi.org/10.1080/02602938.2017.1416456
Schwendimann, B. A., Rodríguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., Gillet, D., & Dillenbourg, P. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30–41. https://doi.org/10.1109/TLT.2016.2599522
Shahiri, A. M., Husain, W., & Rashid, N. A. (2015). A review on predicting student’s performance using data mining techniques. Procedia Computer Science, 72, 414–422. https://doi.org/10.1016/j.procs.2015.12.157
Skaalvik, E. M., & Skaalvik, S. (2004). Self-concept and self-efficacy: A test of the internal/external frame of reference model and predictions of subsequent motivation and achievement. Psychological Reports, 95(3 suppl.), 1187–1202. https://doi.org/10.2466/pr0.95.3f.1187-1202
Stahl, G., Koschmann, T., & Suthers, D. D. (2005). Computer-supported collaborative learning. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 409–426). Cambridge University Press. https://doi.org/10.1017/CBO9780511816833.025
Susnjak, T., Ramaswami, G. S., & Mathrani, A. (2022). Learning analytics dashboard: A tool for providing actionable insights to learners. International Journal of Educational Technology in Higher Education, 19(1), 12. https://doi.org/10.1186/s41239-021-00313-7
Susnjak, T. (2023). Beyond predictive learning analytics modelling and onto explainable artificial intelligence with prescriptive analytics and ChatGPT. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00336-3
Teasley, S. D. (2017). Student facing dashboards: One size fits all? Technology, Knowledge and Learning, 22(3), 377–384. https://doi.org/10.1007/s10758-017-9314-3
Tullis, T. S., & Stetson, J. N. (2004). A comparison of questionnaires for assessing website usability. Proceedings of the 13th Annual UPA Conference (UPA ’04), 7–11 June 2004, Minneapolis, MN, USA (pp. 1–12).
Valle, N., Antonenko, P., Valle, D., Sommer, M., Huggins-Manley, A. C., Dawson, K., Kim, D., & Baiser, B. (2021). Predict or describe? How learning analytics dashboard design influences motivation and statistics anxiety in an online statistics course. Educational Technology Research and Development, 69(3), 1405–1431. https://doi.org/10.1007/s11423-021-09998-z
Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboard applications. American Behavioral Scientist, 57(10), 1500–1509. https://doi.org/10.1177/0002764213479363
Verbert, K., Ochoa, X., De Croon, R., Dourado, R. A., & De Laet, T. (2020). Learning analytics dashboards: The past, the present and the future. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 35–40). ACM Press. https://doi.org/10.1145/3375462.3375504
Vygotsky, L. S. (1980). Mind in society: Development of higher psychological processes. Harvard University Press.
Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304). Routledge.
Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning analytics. Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 203–211). ACM Press. https://doi.org/10.1145/2567574.2567588
Yu, T., & Jo, I.-H. (2014). Educational technology approach toward learning analytics: Relationship between student online behavior and learning performance in higher education. Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 269–270). ACM Press. https://doi.org/10.1145/2567574.2567594
Zacharis, N. Z. (2015). A multivariate approach to predicting student outcomes in web-enabled blended learning courses. The Internet and Higher Education, 27, 44–53. https://doi.org/10.1016/j.iheduc.2015.05.002
Zimmerman, D. W. (1997). Teacher’s corner: A note on interpretation of the paired-samples t test. Journal of Educational and Behavioral Statistics, 22(3), 349–360. https://doi.org/10.3102/10769986022003349
Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13–39). Academic Press. https://doi.org/10.1016/B978-012109890-2/50031-7
Zimmerman, B. J., & Schunk, D. H. (Eds.). (2001). Self-regulated learning and academic achievement: Theory, research, and practice. Springer. https://doi.org/10.1007/978-1-4612-3618-4
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
