Scaffolding Feedback Literacy: Designing a Feedback Analytics Tool with Students

Flora Jin; Bhagya Maheshi; Roberto Martinez-Maldonado; Dragan Gašević; Yi-Shan  Tsai

doi:10.18608/jla.2024.8339

Authors

Flora Jin Monash University https://orcid.org/0009-0002-1825-4028
Bhagya Maheshi Monash University https://orcid.org/0009-0000-3268-0627
Roberto Martinez-Maldonado Monash University https://orcid.org/0000-0002-8375-1816
Dragan Gašević Monash University https://orcid.org/0000-0001-9265-1908
Yi-Shan Tsai Monash University https://orcid.org/0000-0001-8967-5327

DOI:

https://doi.org/10.18608/jla.2024.8339

Keywords:

feedback literacy, learning analytics, feedback analytics, feedback traceability, feedback management, research paper

Abstract

Feedback is essential in learning. The emerging concept of feedback literacy underscores the skills students require for effective use of feedback. This highlights students’ responsibilities in the feedback process. Yet, there is currently a lack of mechanisms to understand how students make sense of feedback and whether they act on it. This gap makes it hard to effectively support students in feedback literacy development and improve the quality of feedback. As a specific application of learning analytics, feedback analytics (analytics on learner engagement with feedback) can offer insights into students’ learning engagement and progression, which can in turn be used to scaffold student feedback literacy. This study proposes a feedback analytics tool, designed with students, aimed at aiding students to synthesize feedback received from multiple sources, scaffold the sense-making process, and prompt deeper reflections or actions on feedback based on data about students’ interactions with feedback. We held focus group discussions with 38 students to learn about their feedback experiences and identified tool features. Based on identified user requirements, a prototype was developed and validated with 16 students via individual interviews. Based on the findings, we envision a feedback analytics tool with the aim of scaffolding student feedback literacy

References

Bikanga Ada, M., & Stansfield, M. (2017). The potential of learning analytics in understanding students’ engagement with their assessment feedback. In M. Chang, N.-S. Chen, R. Huang, Kinshuk, D. G. Sampson, & R. Vasiu (Eds.), Proceedings of the 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT 2017), 3–7 July 2017, Timisoara, Romania (pp. 227–229). IEEE. https://doi.org/10.1109/ICALT.2017.40

Boud, D., & Molloy, E. (2013). Rethinking models of feedback for learning: The challenge of design. Assessment & Evaluation in Higher Education, 38(6), 698–712. https://doi.org/10.1080/02602938.2012.691462

Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354

Dai, W., Lin, J., Jin, H., Li, T., Tsai, Y.-S., GaˇseviÅLc, D., & Chen, G. (2023). Can large language models provide feedback to students? A case study on ChatGPT. In M. Chang, N.-S. Chen, R. Kuo, G. Rudolph, D. G. Sampson, & A. Tlili (Eds.), Proceedings of the 2023 IEEE International Conference on Advanced Learning Technologies (ICALT 2023), 10–13 July 2023, Orem, Utah, USA (pp. 323–325). IEEE. https://doi.org/10.1109/ICALT58122.2023.00100

Han, Y., & Xu, Y. (2020). The development of student feedback literacy: The influences of teacher feedback on peer feedback. Assessment & Evaluation in Higher Education, 45(5), 680–696. https://doi.org/10.1080/02602938.2019.1689545

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487

Henderson, M., Ajjawi, R., Boud, D., & Molloy, E. (2019). Identifying feedback that has impact. In M. Henderson, R. Ajjawi, D. Boud, & E. Molloy (Eds.), The impact of feedback in higher education (pp. 15–34). Springer International Publishing. https://doi.org/10.1007/978-3-030-25112-3_2

Jin, H., Martinez-Maldonado, R., Li, T., Chan, P. W. K., & Tsai, Y.-S. (2022). Towards supporting dialogic feedback processes using learning analytics: The educators’ views on effective feedback. In S. Wilson, N. Arthars, D. Wardak, P. Yeoman, E. Kalman, & D. Y. Liu (Eds.), ASCILITE 2022 Conference Proceedings: Reconnecting Relationships through Technology, 4–7 December 2022, Sydney, Australia (e22054). https://doi.org/10.14742/apubs.2022.54

Jivet, I., Scheffel, M., Specht, M., & Drachsler, H. (2018). License to evaluate. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 7–9 March 2018, Sydney, Australia (pp. 31–40). ACM. https://doi.org/10.1145/3170358.3170421

Kim, H.-S., Cha, Y., & Kim, N. Y. (2021). Effects of AI chatbots on EFL students’ communication skills. The Korean Association for the Study of English Language and Linguistics, 21, 712–734. https://doi.org/10.15738/kjell.21..202108.712

Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., Wight, R., Lucas, C., Sándor, Á ., Kitto, K., Liu, M., Mogarkar, R. V., & Buckingham Shum, S. (2020). AcaWriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12(1), 141–186. https://doi.org/10.17239/JOWR-2020.12.01.06

Li, F., & Han, Y. (2022). Student feedback literacy in L2 disciplinary writing: Insights from international graduate students at a UK university. Assessment & Evaluation in Higher Education, 47(2), 198–212. https://doi.org/10.1080/02602938.2021.1908957

Lim, L.-A., Dawson, S., Gašević, D., Joksimovíc, S., Pardo, A., Fudge, A., & Gentili, S. (2021). Students’ perceptions of, and emotional responses to, personalised learning analytics-based feedback: An exploratory study of four courses. Assessment and Evaluation in Higher Education, 46(3), 339–359. https://doi.org/10.1080/02602938.2020.1782831

Lim, L.-A., Gentili, S., Pardo, A., Kovanovíc, V., Whitelock-Wainwright, A., Gašević, D., & Dawson, S. (2021). What changes, and for whom? A study of the impact of learning analytics-based process feedback in a large course. Learning and Instruction, 72, 101202. https://doi.org/10.1016/j.learninstruc.2019.04.003

Lin, J., Thomas, D., Han, F., Gupta, S., Tan, W., Nguyen, N., & Koedinger, K. (2023). Using large language models to provide explanatory feedback to human tutors. arXiv preprint arXiv:2306.15498. https://doi.org/10.48550/arXiv.2306.15498

McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–82. https://doi.org/10.11613/BM.2012.031

Molloy, E., Boud, D., & Henderson, M. (2020). Developing a learning-centred framework for feedback literacy. Assessment & Evaluation in Higher Education, 45(4), 527–540. https://doi.org/10.1080/02602938.2019.1667955

O’Donovan, B. M., den Outer, B., Price, M., & Lloyd, A. (2021). What makes good feedback good? Studies in Higher Education, 46(2), 318–329. https://doi.org/10.1080/03075079.2019.1630812

Olafson, K. M., & Ferraro, F. (2001). Effects of emotional state on lexical decision performance. Brain and Cognition, 45(1), 15–20. https://doi.org/10.1006/brcg.2000.1248

Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in psychology, 8. https://doi.org/10.3389/fpsyg.2017.00422

Pardo, A., Jovanovic, J., Dawson, S., GaˇseviÅLc, D., & Mirriahi, N. (2019). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology, 50(1), 128–138. https://doi.org/10.1111/BJET.12592

Pitt, E., & Norton, L. (2017). “Now that’s the feedback I want!” Students’ reactions to feedback on graded work and what they do with it. Assessment & Evaluation in Higher Education, 42(4), 499–516. https://doi.org/10.1080/02602938.2016.1142500

Price, M., Handley, K., Millar, J., & O’Donovan, B. (2010). Feedback: All that effort, but what is the effect? Assessment & Evaluation in Higher Education, 35(3), 277–289. https://doi.org/10.1080/02602930903541007

Robinson, S., Pope, D., & Holyoak, L. (2013). Can we meet their expectations? Experiences and perceptions of feedback in first year undergraduate students. Assessment & Evaluation in Higher Education, 38(3), 260–272. https://doi.org/10.1080/02602938.2011.629291

Rowe, A. D. (2017). Feelings about feedback: The role of emotions in assessment for learning. In D. Carless, S. M. Bridges, C. K. Y. Chan, & R. Glofcheski (Eds.), Scaling up assessment for learning in higher education (pp. 159–172). https://doi.org/10.1007/978-981-10-3045-1_11

Ryan, T., Henderson, M., Ryan, K., & Kennedy, G. (2023). Identifying the components of effective learner-centred feedback information. Teaching in Higher Education, 28(7), 1565–1582. https://doi.org/10.1080/13562517.2021.1913723

Sedrakyan, G., Malmberg, J., Verbert, K., JÅNarvelÅNa, S., & Kirschner, P. A. (2020). Linking learning behavior analytics and learning science concepts: Designing a learning analytics dashboard for feedback to support learning regulation. Computers in Human Behavior, 107, 105512. https://doi.org/10.1016/j.chb.2018.05.004

Slade, S., & Tait, A. (2019). Global guidelines: Ethics in learning analytics (tech. rep.). Association for the Advancement of Computing in Education. https://aace.org/review/global-guidelines-ethics-in-learning-analytics/

Sutton, P. (2009). Towards dialogic feedback. Critical and Reflective Practice in Education, 1(1), 1–10. https://marjon.repository.guildhe.ac.uk/id/eprint/17582/1/Towards%20dialogic%20feedback.pdf

Tsai, Y.-S. (2022). Why feedback literacy matters for learning analytics. In Proceedings of the 16th International Conference of the Learning Sciences (ICLS 2022), 6–10 June 2022, Hiroshima, Japan (pp. 27–34). International Society of the Learning Sciences. https://2022.isls.org/proceedings/

Tsai, Y.-S., Whitelock-Wainwright, A., & Gašević, D. (2021). More than figures on your laptop: (Dis)trustful implementation of learning analytics. Journal of Learning Analytics, 8(3), 81–100. https://doi.org/10.18608/jla.2021.7379

Whitelock, D., Twiner, A., Richardson, J. T., Field, D., & Pulman, S. (2015). OpenEssayist: A supply and demand learning analytics tool for drafting academic essays. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK 2015), 16–20 March 2015, Poughkeepsie, New York, USA (pp. 208–212). ACM. https://doi.org/10.1145/2723576.2723599

Winstone, N. (2019). Facilitating students’ use of feedback: Capturing and tracking impact using digital tools. In M. Henderson, R. Ajjawi, D. Boud, & E. Molloy (Eds.), The impact of feedback in higher education (pp. 225–242). Springer International Publishing. https://doi.org/10.1007/978-3-030-25112-3_13

Yang, M., & Carless, D. (2013). The feedback triangle and the enhancement of dialogic feedback processes. Teaching in Higher Education, 18(3), 285–297. https://doi.org/10.1080/13562517.2012.719154

Zheng, L., Niu, J., & Zhong, L. (2022). Effects of a learning analytics-based real-time feedback approach on knowledge elaboration, knowledge convergence, interactive relationships and group performance in CSCL. British Journal of Educational Technology, 53(1), 130–149. https://doi.org/10.1111/bjet.13156

Zimbardi, K., Colthorpe, K., Dekker, A., Engstrom, C., Bugarcic, A., Worthy, P., Victor, R., Chunduri, P., Lluka, L., & Long, P. (2017a). Are they using my feedback? The extent of students’ feedback use has a large impact on subsequent academic performance. Assessment & Evaluation in Higher Education, 42(4), 625–644. https://doi.org/10.1080/02602938.2016.1174187

Zimbardi, K., Colthorpe, K., Dekker, A., Engstrom, C., Bugarcic, A., Worthy, P., Victor, R., Chunduri, P., Lluka, L., & Long, P. (2017b). Are they using my feedback? The extent of students’ feedback use has a large impact on subsequent academic performance. Assessment & Evaluation in Higher Education, 42(4), 625–644. https://doi.org/10.1080/02602938.2016.1174187

Scaffolding Feedback Literacy

Designing a Feedback Analytics Tool with Students

Authors

DOI:

Keywords:

Abstract

References

Downloads

Published

How to Cite

Issue

Section

License

Most read articles by the same author(s)