Narrowing the Feedback Gap
Examining Student Engagement with Personalized and Actionable Feedback Messages
DOI:
https://doi.org/10.18608/jla.2021.7184Keywords:
feedback, learning analytics, higher education, feedback gap, data-driven approachesAbstract
One of the major factors affecting student learning is feedback. Although the importance of feedback has been recognized in educational institutions, dramatic changes - such as bigger class sizes and a more diverse student population - challenged the provision of effective feedback. In light of these changes, educators have increasingly been using new digital tools to provide student feedback, given the broader adoption and availability of these new technologies. However, despite these efforts, most educators have limited insight into the recipience of their feedback and wonder which students engage with feedback. This problem is referred to as the "feedback gap," which is the difference between the potential and actual use of feedback, preventing educators and instructional designers from understanding feedback recipience among students. In this study, a set of trackable call-to-action (CTA) links were embedded in feedback messages focused on learning processes and self-regulation of learning in one fully online marketing course and one blended bioscience course. These links helped us examine the association between feedback engagement and course success. We also conducted two focus groups with students from one of the courses to further examine student perceptions of feedback messages. Our results across both courses revealed that early engagement with feedback is positively associated with passing the course and that most students considered feedback messages helpful in their learning. Our study also found some interesting demographic differences between students regarding their engagement with the feedback messages. Such insight enables instructors to ask "why" questions, support students' learning, improve feedback processes, and narrow the gap between potential and actual use of feedback. The practical implications of our findings are further discussed.
References
Applegate, E. (2005). Strategic copywriting: How to create effective advertising. Lanham, MD, USA: Rowman & Littlefield.
Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: Using learning analytics to increase student success. In Proceedings of the Second International Conference on Learning Analytics and Knowledge (LAK ’12), 29 April–2 May 2012, Vancouver, Canada (pp. 267–270). New York: ACM. https://doi.org/https://doi.org/10.1145/2330601.2330666
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64(1), 417–444. https://doi.org/10.1146/annurev-psych-113011-143823
Boud, D., & Molloy, E. (2013a). Feedback in higher and professional education: Understanding it and doing it well. New York, USA: Routledge.
Boud, D., & Molloy, E. (2013b). Rethinking models of feedback for learning: The challenge of design. Assessment & Evaluation in Higher Education, 38(6), 698–712. https://doi.org/10.1080/02602938.2012.691462
Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017). Dashboard for actionable feedback on learning skills: Scalability and usefulness. In P. Zaphiris & A. Ioannou (Eds.), Learning and collaboration technologies. Technology in Education (pp. 229–241). Vancouver, BC, Canada: Springer International Publishing. https://doi.org/10.1007/978-3-319-58515-418
Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245–281. https://doi.org/10.3102/00346543065003245
Carless, D. (2020). Longitudinal perspectives on students’ experiences of feedback: A need for teacher–student partnerships. Higher Education Research & Development, 39(3), 1–14. https://doi.org/10.1080/07294360.2019.1684455
Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354
Chong, S. W. (2019). College students’ perception of e-feedback: A grounded theory perspective. Assessment & Evaluation in Higher Education, 44(7), 1090–1105. https://doi.org/10.1080/02602938.2019.1572067
Chong, S. W. (2020). Reconsidering student feedback literacy from an ecological perspective. Assessment & Evaluation in Higher Education, 46(1), 1–13. https://doi.org/10.1080/02602938.2020.1730765
Corrin, L., & de Barba, P. (2014). Exploring students’ interpretation of feedback delivered through learning analytics dashboards. In Proceedings of the ASCILITE 2014 Conference, 23–26 November 2014, Dunedin, NZ (pp. 629–633). ASCILITE. Retrieved from https://ascilite.org/conferences/dunedin2014/files/concisepapers/223-Corrin.pdf
Corrin, L., & de Barba, P. (2015). How do students interpret feedback delivered via dashboards? In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK 2015), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 430–431). New York: ACM. https://doi.org/10.1145/2723576.2723662
Cugelman, B., Thelwall, M., & Dawes, P. (2011). Online interventions for social marketing health behavior change campaigns: A meta-analysis of psychological architectures and adherence factors. Journal of Medical Internet Research, 13(1), e17. https://doi.org/10.2196/jmir.1367
Dawson, P., Henderson, M., Ryan, T., Mahoney, P., Boud, D., Phillips, M., & Molloy, E. (2018). Technology and feedback design. In M. J. Spector, B. B. Lockee, & M. D. Childress (Eds.), Learning, Design, and Technology (pp. 1–45). Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-319-17727-4124-1
Deighton, J., & Kornfeld, L. (2009). Interactivity’s unanticipated consequences for marketers and marketing. Journal of Interactive Marketing, 23(1), 4–10. (Anniversary Issue) https://doi.org/10.1016/j.intmar.2008.10.001
Ellis, C. (2013). Broadening the scope and increasing the usefulness of learning analytics: The case for assessment analytics. British Journal of Educational Technology, 44(4), 662–664. https://doi.org/10.1111/bjet.12028
Evans, C. (2013). Making sense of assessment feedback in higher education. Review of Educational Research, 83(1), 70–120. https://doi.org/10.3102/0034654312474350
Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1). New York: Springer. https://doi.org/10.1007/978-0-387-84858-7
Gannaway, D., Green, T., & Mertova, P. (2018). So how big is big? Investigating the impact of class size on ratings in student evaluation. Assessment & Evaluation in Higher Education, 43(2), 175–184. https://doi.org/10.1080/02602938.2017.1317327
Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://doi.org/10.1007/s11528-014-0822-x
Gibbs, G., & Simpson, C. (2005). Conditions under which assessment supports students’ learning. Learning and Teaching in Higher Education, 1(1), 3–31. Retrieved from https://eprints.glos.ac.uk/3609/
Glass, R., & Callahan, S. (2014). The big data-driven business: How to use big data to win customers, beat competitors, and boost profits. Hoboken, NJ, USA: John Wiley & Sons.
Grann, J., & Bushway, D. (2014). Competency map: Visualizing student learning to promote student success. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (LAK 2014), 24–28 March 2014, Indianapolis, IN, USA (pp. 168–172). New York: ACM. https://doi.org/10.1145/2567574.2567622
Gunelius, S. (2018). Ultimate guide to email marketing for business. Irvine, CA, USA: Entrepreneur Press.
Hanna, R. C., Swain, S. D., & Smith, J. (2015). Email marketing in a digital world: The basics and beyond. New York: Business Expert Press.
Hartemo, M. (2016). Email marketing in the era of the empowered consumer. Journal of Research in Interactive Marketing, 10(3), 212–230. https://doi.org/10.1108/JRIM-06-2015-0040
Hattie,J. (1999). Influences on student learning. Inaugural Lecture at the University of Auckland, 2 August 1999, Auckland, NZ, 1–25. Retrieved from https://cdn.auckland.ac.nz/assets/education/about/research/documents/influences-on-student-learning.pdf
Hattie, J., & Timperley, H.(2007).The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
Herodotou, C., Heiser, S., & Rienties, B. (2017). Implementing randomised control trials in open and distance learning: A feasibility study. Open Learning: The Journal of Open, Distance and e-Learning, 32(2), 147–162. https://doi.org/10.1080/02680513.2017.1316188
Howell, J. A., Roberts, L. D., & Mancini, V. O. (2018). Learning analytics messages: Impact of grade, sender, comparative information and message style on student affect and academic resilience. Computers in Human Behavior, 89, 8–15. https://doi.org/10.1016/j.chb.2018.07.021
Iraj, H., Fudge, A., Faulkner, M., Pardo, A., & Kovanović, V. (2020). Understanding students’ engagement with personalised feedback messages. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 438–447). New York: ACM. https://doi.org/10.1145/3375462.3375527
Kizilcec, R., & Brooks, C. (2017). Diverse big data and randomized field experiments in massive open online courses. In C. Lang, G. Siemens, A. F. Wise, & D. Gašević (Eds.), The Handbook of Learning Analytics (pp. 211–222). Society for Learning Analytics Research. https://doi.org/10.18608/hla17.018
Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical re-view, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284. https://doi.org/10.1037/0033-2909.119.2.254
Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89. https://doi.org/10.1016/j.iheduc.2015.06.002
Krueger, A. B. (1999). Experimental estimates of education production functions. The Quarterly Journal of Economics, 114(2),497–532. https://doi.org/10.1162/003355399556052
Lacey, A., & Luff, D. (2001). Qualitative data analysis. Nottingham: Trent Focus Sheffield.
Lim, L.-A., Gentili, S., Pardo, A., Kovanović, V., Whitelock-Wainwright, A., Gašević, D., & Dawson, S. (2019). What changes, and for whom? A study of the impact of learning analytics-based process feedback in a large course. Learning and Instruction, 72, 101202. https://doi.org/10.1016/j.learninstruc.2019.04.003
Liu, D. Y.-T., Bartimote-Aufflick, K., Pardo, A., & Bridgeman, A. J. (2017). Data-driven personalization of student learning support in higher education. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends: A view of the current state of the art to enhance e-learning (pp. 143–169). Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-319-52977-65
Lockyer, L., & Dawson, S. (2011). Learning designs and learning analytics. In Proceedings of the First International Conference on Learning Analytics & Knowledge (LAK 2011), 27 February–1 March 2011, Banff, AB, Canada (pp.153–156). https://doi.org/10.1145/2090116.2090140
Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30–40. https://doi.org/10.17471/2499-4324/195
McKay, T., Miller, K., & Tritz, J. (2012). What to do with actionable intelligence: E2coach as an intervention engine. In Proceedings of the Second International Conference on Learning Analytics & Knowledge (LAK 2012), 29 April–2 May2012, Vancouver, BC, Canada (pp. 88–91). New York: ACM. https://doi.org/10.1145/2330601.2330627
Molloy, E., Boud, D., & Henderson, M. (2020). Developing a learning-centred framework for feedback literacy. Assessment &Evaluation in Higher Education, 45(4), 527–540. https://doi.org/10.1080/02602938.2019.1667955
Moodle. (2019).URL Resource—MoodleDocs. Retrieved from https://docs.moodle.org/37/en/URLresource
Nguyen, Q., Huptych, M., & Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance. In Proceedings of the Eighth International Conference on Learning Analytics & Knowledge (LAK 2018), 5–8 March 2018, Sydney, Australia (pp. 141–150). https://doi.org/10.1145/3170358.3170398
Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., & Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior, 76,703–714. https://doi.org/10.1016/j.chb.2017.03.028
Nicol, D. (2010). From monologue to dialogue: Improving written feedback processes in mass higher education. Assessment & Evaluation in Higher Education, 35(5), 501–517. https://doi.org/10.1080/02602931003786559
Nicol, D., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090
O’Neil, C. (2013). On being a data skeptic. Sebastopol, CA, USA: O’Reilly Media.
Pardo, A., Bartimote, K., Shum, S. B., Dawson, S., Gao, J., Gašević, D., . . . Vigentini, L. (2018). OnTask: Delivering data-informed, personalized learning support actions. Journal of Learning Analytics, 5(3), 235–249. https://doi.org/10.18608/jla.2018.53.15
Pardo, A., Jovanovic, J., Dawson, S., Gašević, D., & Mirriahi, N. (2019). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology, 50(1), 128–138. https://doi.org/10.1111/bjet.12592
Parsons, A. L., & Lepkowska-White, E. (2010). Web site references in print advertising: An analysis of calls to action. Journal of Internet Commerce, 9 (3-4), 151–163. https://doi.org/10.1080/15332861.2010.526487
Price, M., Handley, K., Millar, J., & O’Donovan, B. (2010). Feedback: All that effort, but what is the effect? Assessment & Evaluation in Higher Education, 35(3), 277–289. https://doi.org/10.1080/02602930903541007
R Core Team. (2019). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M.(2011). pROC: anopen-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12(1), 77. https://doi.org/10.1186/1471-2105-12-77
Rodgers, J. L., & Nicewander, W. A. (1988). Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1), 59–66. https://doi.org/10.1080/00031305.1988.10475524
Rowe, A. D. (2017). Feelings about feedback: The role of emotions in assessment for learning. In D. Carless, S. M. Bridges, C. K. Y. Chan, & R. Glofcheski (Eds.), Scaling up assessment for learning in higher education (pp. 159–172). Singapore: Springer Singapore. https://doi.org/10.1007/978-981-10-3045-111
Rowe, A. D., & Wood, L. N. (2009). Student perceptions and preferences for feedback. Asian Social Science, 4(3), 78–88. https://doi.org/10.5539/ass.v4n3p78
Ryan, T., Gašević, D., & Henderson, M. (2019). Identifying the impact of feedback over time and at scale: Opportunities for learning analytics. In M. Henderson, R. Ajjawi, D. Boud, & E. Molloy (Eds.), The impact of feedback in higher education: Improving assessment outcomes for learners (pp. 207–223). Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-030-25112-312
Sahni, N. S., Wheeler, S. C., & Chintagunta, P. (2018). Personalization in email marketing: The role of noninformative advertising content. Marketing Science, 37(2), 236–258. https://doi.org/10.1287/mksc.2017.1066
Senko, C., Hulleman, C. S., & Harackiewicz, J. M. (2011). Achievement goal theory at the cross-roads: Old controversies, current challenges, and new directions. Educational Psychologist, 46(1), 26–47. https://doi.org/10.1080/00461520.2011.538646
Sharp, B. (2017).Marketing: Theory, Evidence, Practice. Oxford, UK: Oxford University Press. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795
Singh, G., Singh, H., & Shriwastav, S. (2019). Improving email marketing campaign success rate using personalization. In A. K. Laha (Ed.), Advances in Analytics and Applications (pp. 77–83). Singapore: Springer Singapore. https://doi.org/10.1007/978-981-13-1208-38
Sutton, P. (2012). Conceptualizing feedback literacy: Knowing, being, and acting. Innovations in Education and Teaching International, 49(1), 31–40. https://doi.org/10.1080/14703297.2012.647781
Szymanski, D. M., & Hise, R. T. (2000). E-satisfaction: An initial examination. Journal of Retailing, 76(3), 309–322. https://doi.org/10.1016/S0022-4359(00)00035-X
Teasley, S. D. (2017). Student facing dashboards: One size fits all? Technology, Knowledge and Learning, 22(3), 377–384. https://doi.org/10.1007/s10758-017-9314-3
Tempelaar, D., Rienties, B., Mittelmeier, J., & Nguyen, Q. (2018). Student profiling in a dispositional learning analytics application using formative assessment. Computers in Human Behavior, 78, 408–420. https://doi.org/10.1016/j.chb.2017.08.010
Turner, G., & Gibbs, G. (2010). Are assessment environments gendered? An analysis of the learning responses of male and female students to different assessment environments. Assessment & Evaluation in Higher Education, 35(6), 687–698. https://doi.org/10.1080/02602930902977723
Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304). Mahwah, NJ, USA: Lawrence Erlbaum Associates Publishers. Retrieved from https://psycnet.apa.org/record/1998-07283-011
Winstone, N. E. (2019). Facilitating students’ use of feedback: Capturing and tracking impact using digital tools. In M. Henderson, R. Ajjawi, D. Boud, & E. Molloy (Eds.), The impact of feedback in higher education: Improving assessment outcomes for learners (pp. 225–242). Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-030-25112-313
Winstone, N. E., Mathlin, G., & Nash, R. A. (2019). Building feedback literacy: Students’ perceptions of the developing engagement with feedback toolkit. Frontiers in Education, 4, 1–11. https://doi.org/10.3389/feduc.2019.00039
Winstone, N. E., & Nash, R. (2016). The “Developing Engagement with Feedback Toolkit (DEFT)”: Integrating Assessment literacy into course design (Tech. Rep.). York, UK: Higher Education Academy. Retrieved from https://www.heacademy.ac.uk/system/files/resources/thedevelopingengagementwithfeedbacktoolkitdeft0.pdf
Winstone, N. E., Nash, R. A., Parker, M., & Rowntree, J. (2017). Supporting learners’ agentic engagement with feedback: A systematic review and a taxonomy of recipience processes. Educational Psychologist, 52(1), 17–37. https://doi.org/10.1080/00461520.2016.1207538
Winstone, N. E., Nash, R. A., Rowntree, J., & Parker, M. (2017). ‘It’d be useful, but I wouldn’t use it’: Barriers to university students’ feedback seeking and recipience. Studies in Higher Education, 42(11), 2026–2041. https://doi.org/10.1080/03075079.2015.1130032
Wisniewski, B., Zierer, K., & Hattie, J. (2020). The power of feedback revisited: A meta-analysis of educational feedback research. Frontiers in Psychology, 10, 3087. https://doi.org/10.3389/fpsyg.2019.03087
Wong, B. T. M., & Li, K. C. (2018). Learning analytics intervention: A review of case studies. In 2018 International Symposium on Educational Technology (ISET 2018), 31 July–2 August 2018, Osaka, Japan (pp. 178–182). New York: IEEE. https://doi.org/10.1109/ISET.2018.00047
Zhang, X. A., Kumar, V., & Cosguner, K. (2017). Dynamically managing a profitable email marketing program. Journal of Marketing Research, 54(6), 851–866. https://doi.org/10.1509/jmr.16.0210
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
