Enhancing Feedback Uptake and Self-Regulated Learning in Procedural Skills Training: Design and Evaluation of a Learning Analytics Dashboard

Ignacio Villagrán; Rocio Hernández; Gregory Schuit; Andrés Neyem; Javiera Fuentes; Loreto Larrondo; Elisa Margozzini; María T. Hurtado; Zoe Iriarte; Constanza Miranda; Julián Varas; Isabel Hilliger

doi:10.18608/jla.2024.8195

Authors

Ignacio Villagrán Pontificia Universidad Católica de Chile https://orcid.org/0000-0003-3130-8326
Rocio Hernández Pontificia Universidad Católica de Chile https://orcid.org/0000-0003-4677-2519
Gregory Schuit Pontificia Universidad Católica de Chile https://orcid.org/0000-0002-7219-9915
Andrés Neyem Pontificia Universidad Católica de Chile https://orcid.org/0000-0002-5734-722X
Javiera Fuentes Pontificia Universidad Católica de Chile and Maastricht University https://orcid.org/0000-0003-0720-2169
Loreto Larrondo Pontificia Universidad Católica de Chile https://orcid.org/0009-0008-1617-3228
Elisa Margozzini Pontificia Universidad Católica de Chile https://orcid.org/0009-0004-9905-5964
María T. Hurtado Pontificia Universidad Católica de Chile https://orcid.org/0009-0002-9560-1345
Zoe Iriarte Pontificia Universidad Católica de Chile https://orcid.org/0009-0000-6778-8048
Constanza Miranda Johns Hopkins University https://orcid.org/0000-0002-9110-2832
Julián Varas Pontificia Universidad Católica de Chile https://orcid.org/0000-0002-5828-9623
Isabel Hilliger Pontificia Universidad Católica de Chile https://orcid.org/0000-0001-5270-7655

DOI:

https://doi.org/10.18608/jla.2024.8195

Keywords:

learning analytics dashboard, feedback, procedural skills, self-regulated learning, health professions education, research paper

Abstract

Remote technology has been widely incorporated into health professions education. For procedural skills training, effective feedback and reflection processes are required. Consequently, supporting a self-regulated learning (SRL) approach with learning analytics dashboards (LADs) has proven beneficial in online environments. Despite the potential of LADs, understanding their design to enhance SRL and provide useful feedback remains a significant challenge. Focusing on LAD design, implementation, and evaluation, the study followed a mixed-methods two-phase design-based research approach. The study used a triangulation methodology of qualitative interviews and SRL and sensemaking questionnaires to comprehensively understand the LAD’s effectiveness and student SRL and feedback uptake strategies during remote procedural skills training. Initial findings revealed the value students placed on performance visualization and peer comparison despite some challenges in LAD design and usability. The study also identified the prominent adoption of SRL strategies such as help-seeking, elaboration, and strategic planning. Sensemaking results showed the value of personalized performance metrics and planning resources in the LAD and recommendations to improve reflection and feedback uptake. Subsequent findings suggested that SRL levels significantly predicted the levels of sensemaking. The students valued the LAD as a tool for supporting feedback uptake and strategic planning, demonstrating the potential for enhancing procedural skills learning.

References

Brydges, R., Nair, P., Ma, I., Shanks, D., & Hatala, R. (2012). Directed self‐regulated learning versus instructor‐regulated learning in simulation training. Medical Education, 46(7), 648–656. https://doi.org/10.1111/j.1365-2923.2012.04268.x

Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354

Cecilio-Fernandes, D., Medina-Ramírez, R., Sandars, J., & Costa, M. J. (2023). Self-regulated learning processes across different physiotherapy clinical procedural skills and time intervals: A SRL microanalysis study. Medical Teacher, 45(10), 1170–1176. https://doi.org/10.1080/0142159X.2023.2198096

Chatti, M. A., Muslim, A., Guliani, M., & Guesmi, M. (2020). The LAVA model: Learning analytics meets visual analytics. In D. Ifenthaler & D. Gibson (Eds.), Adoption of data analytics in higher education learning and teaching: Advances in analytics for learning and teaching (pp. 71–93). Springer, Cham. https://doi.org/10.1007/978-3-030-47392-1_5

Creswell, J. (2010). Mapping the developing landscape of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social & behavioral research (pp. 45–68). SAGE Publications. https://doi.org/10.4135/9781506335193

Dawson, P., Henderson, M., Ryan, T., Mahoney, P., Boud, D., Phillips, M., & Molloy, E. (2018). Technology and feedback design. In M. Spector, B. Lockee, & M. Childress (Eds.), Learning, design, and technology. Springer, Cham. https://doi.org/10.1007/978-3-319-17727-4_124-1

Deeley, S. J. (2018). Using technology to facilitate effective assessment for learning and feedback in higher education. Assessment & Evaluation in Higher Education, 43(3), 439–448. https://doi.org/10.1080/02602938.2017.1356906

Dishman, E. (2003). Designing for the new old: Asking, observing and performing future elders. In B. Laurel (Ed.), Design research: Methods and perspectives (pp. 41–48). The MIT Press.

Etoom, M., Aldaher, K. N., Abdelhaq, A. A., Alawneh, A., & Alghwiri, A. A. (2023). Distance learning in physiotherapy education during the COVID-19 pandemic: Students’ satisfaction, perceived quality, and potential predictors of satisfaction. Physiotherapy Theory and Practice, 39(7), 1513–1518. https://doi.org/10.1080/09593985.2022.2042438

Gaete, M. I., Belmar, F., Cortés, M., Alseidi, A., Asbun, D., Durán, V., Escalona, G., Achurra, P., Villagrán, I., Crovari, F., Pimentel, F., & Varas, J. (2023). Remote and asynchronous training network: From a SAGES grant to an eight-country remote laparoscopic simulation training program. Surgical Endoscopy, 37(2), 1458–1465. https://doi.org/10.1007/s00464-022-09386-5

Greene, J. A., & Azevedo, R. (2007). A theoretical review of Winne and Hadwin’s model of self-regulated learning: New perspectives and directions. Review of Educational Research 77(3), 334–372. https://doi.org/10.3102/003465430303953

Heikkinen, S., Saqr, M., Malmberg, J., & Tedre, M. (2023). Supporting self-regulated learning with learning analytics interventions: A systematic literature review. Education and Information Technologies, 28(3), 3059–3088. https://doi.org/10.1007/s10639-022-11281-4

Jivet, I., Scheffel, M., Schmitz, M., Robbers, S., Specht, M., & Drachsler, H. (2020). From students with love: An empirical study on learner goals, self-regulated learning and sense-making of learning analytics in higher education. The Internet and Higher Education, 47, 100758. https://doi.org/10.1016/j.iheduc.2020.100758

Kim, J., Jo, I.-H., & Park, Y. (2016). Effects of learning analytics dashboard: Analyzing the relations among dashboard utilization, satisfaction, and learning achievement. Asia Pacific Education Review, 17(1), 13–24. https://doi.org/10.1007/s12564-015-9403-8

Kitsantas, A., Robert, A. R., & Doster, J. (2004). Developing self-regulated learners: Goal setting, self-evaluation, and organizational signals during acquisition of procedural skills. The Journal of Experimental Education, 72(4), 269–287. https://doi.org/10.3200/JEXE.72.4.269-287

Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017). Self-regulated learning strategies predict learner behavior and goal attainment in massive open online courses. Computers & Education, 104, 18–33. https://doi.org/10.1016/j.compedu.2016.10.001

Lim, L., Dawson, S., Joksimović, S., & Gašević, D. (2019). Exploring students’ sensemaking of learning analytics dashboards: Does frame of reference make a difference? Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (pp. 250–259). ACM. https://doi.org/10.1145/3303772.3303804

Lim, L.-A., Gašević, D., Matcha, W., Uzir, N. A., & Dawson, S. (2021). Impact of learning analytics feedback on self-regulated learning: Triangulating behavioural logs with students’ recall. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 364–374). ACM Press. https://doi.org/10.1145/3448139.3448174

Matcha, W., Uzir N. A., Gašević, D., & Pardo, A. (2020). A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective. IEEE Transactions on Learning Technologies, 13(2), 226–245. https://doi.org/10.1109/TLT.2019.2916802

Mayring, P. (2019). Qualitative content analysis: Demarcation, varieties, developments. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 20(3) https://doi.org/10.17169/fqs-20.3.3343

Nash, R. A., & Winstone, N. E. (2017). Responsibility-sharing in the giving and receiving of assessment feedback. Frontiers in Psychology, 8, 1519. https://doi.org/10.3389/fpsyg.2017.01519

Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090

Nussbaumer, A., Dahn, I., Kroop, S., Mikroyannidis, A., & Albert, D. (2015). Supporting self-regulated learning. In S. Kroop, A. Mikroyannidis, & M. Wolpers (Eds.), Responsive open learning environments (pp. 17–48). Springer, Cham. https://doi.org/10.1007/978-3-319-02399-1_2

Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422. https://doi.org/10.3389/fpsyg.2017.00422

Park, Y., & Jo, I.-H. (2019). Factors that affect the success of learning analytics dashboards. Educational Technology Research and Development, 67(6), 1547–1571. https://doi.org/10.1007/s11423-019-09693-0

Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). Academic Press. https://doi.org/10.1016/B978-012109890-2/50043-3

Rets, I., Herodotou, C., Bayer, V., Hlosta, M., & Rienties, B. (2021). Exploring critical factors of the perceived usefulness of a learning analytics dashboard for distance university students. International Journal of Educational Technology in Higher Education, 18, 46. https://doi.org/10.1186/s41239-021-00284-9

Reimann, P. (2013). Design-based research: Designing as research. In S. Puntambekar, P. Goodyear, B. L. H. Grabowski, J. Underwood, & N. Winters (Eds.), Handbook of design in educational technology (pp. 44–52). Routledge.

Rieman, J., Franzke, M., & Redmiles, D. (1995, May). Usability evaluation with the cognitive walkthrough. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’95), 7–11 May 1995, Denver, CO, USA (pp. 387–388). ACM Press. https://doi.org/10.1145/223355.223735

Roco, M. C. (2015). Principles and methods that facilitate convergence. In W. Bainbridge & M. Roco (Eds.), Handbook of science and technology convergence. Springer, Cham. https://doi.org/10.1007/978-3-319-04033-2_2-2

Schumacher, C., & Ifenthaler, D. (2018). Features students really expect from learning analytics. Computers in Human Behavior, 78, 397–407. https://doi.org/10.1016/j.chb.2017.06.030

Schwendimann, B. A., Rodríguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., Gillet, D., & Dillenbourg, P. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30–41. https://doi.org/10.1109/TLT.2016.2599522

Sedrakyan, G., Malmberg, J., Verbert, K., Järvelä, S., & Kirschner, P. A. (2020). Linking learning behavior analytics and learning science concepts: Designing a learning analytics dashboard for feedback to support learning regulation. Computers in Human Behavior, 107, 105512. https://doi.org/10.1016/j.chb.2018.05.004

Susnjak, T., Ramaswami, G. S., & Mathrani, A. (2022). Learning analytics dashboard: A tool for providing actionable insights to learners. International Journal of Educational Technology in Higher Education, 19, 12. https://doi.org/10.1186/s41239-021-00313-7

Süss-Havemann, C., Kosan, J., Seibold, T., Dibbern, N. M., Daubmann, A., Kubitz, J. C., & Beck, S. (2020). Implementation of basic life support training in schools: A randomised controlled trial evaluating self-regulated learning as alternative training concept. BMC Public Health, 20, 50. https://doi.org/10.1186/s12889-020-8161-7

Vera, M., Kattan, E., Cerda, T., Niklitshek, J., Montaña, R., Varas, J., & Corvetto, M. A. (2021). Implementation of distance-based simulation training programs for healthcare professionals: Breaking barriers during COVID-19 pandemic. Simulation in Healthcare, 16(6), 401–406. https://doi.org/10.1097/SIH.0000000000000550

Vigentini, L., Clayphan, A., Zhang, X., & Chitsaz, M. (2017). Overcoming the MOOC data deluge with learning analytic dashboards. In A. Peña-Ayala (Ed.) Learning analytics: Fundaments, applications, and trends (pp. 171–198). Springer, Cham. https://doi.org/10.1007/978-3-319-52977-6_6

Villagrán, I., Rammsy, F., Del Valle, J., Gregorio de las Heras, S., Pozo, L., García, P., Torres, G., Varas, J., Mandrusiak, A., Corvetto, M., & Fuentes-Cimma, J. (2023). Remote, asynchronous training and feedback enables development of neurodynamic skills in physiotherapy students. BMC Medical Education, 23, 267. https://doi.org/10.1186/s12909-023-04229-w

Wang, D., & Han, H. (2021). Applying learning analytics dashboards based on process-oriented feedback to improve students’ learning effectiveness. Journal of Computer Assisted Learning, 37(2), 487–499. https://doi.org/10.1111/jcal.12502

Wang, P. Z. T., Xie, W. Y., Nair, S., Dave, S., Shatzer, J., & Chahine, S. (2020). A comparison of guided video reflection versus self-regulated learning to teach knot tying to medical students: A pilot randomized controlled trial. Journal of Surgical Education, 77(4), 805–816. https://doi.org/10.1016/j.jsurg.2020.02.014

Wharton, C., Rieman, J., Lewis, C., & Polson, P. (1994). The cognitive walkthrough method: A practitioner’s guide. In J. Nielsen & R. L. Mack (Eds.), Usability inspection methods (pp. 105–140). John Wiley & Sons.

Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–305). Routledge. https://doi.org/10.4324/9781410602350

Wood, J. (2021). A dialogic technology-mediated model of feedback uptake and literacy. Assessment & Evaluation in Higher Education, 46(8), 1173–1190. https://doi.org/10.1080/02602938.2020.1852174

Zhang, H., Liao, A. W. X., Goh, S. H., Wu, X. V., & Yoong, S. Q. (2022). Effectiveness of peer teaching in health professions education: A systematic review and meta-analysis. Nurse Education Today, 118, 105499. https://doi.org/10.1016/j.nedt.2022.105499

Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13–40). Academic Press. https://doi.org/10.1016/B978-012109890-2/50031-7

Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice, 41(2), 64–70. https://doi.org/10.1207/s15430421tip4102_2

Enhancing Feedback Uptake and Self-Regulated Learning in Procedural Skills Training

Design and Evaluation of a Learning Analytics Dashboard

Authors

DOI:

Keywords:

Abstract

References

Downloads

Published

How to Cite

Issue

Section

License

Most read articles by the same author(s)