Unpacking the Complexity: Why Current Feedback Systems Fail to Improve Learner Self-Regulation of Participation in Collaborative Activities
DOI:
https://doi.org/10.18608/jla.2024.8355Keywords:
equitable participation, feedback systems, self-regulated learning, research paperAbstract
Even before the inception of the term learning analytics, researchers globally had been investigating the use of various feedback systems to support the self-regulation of participation and promote equitable contributions during collaborative learning activities. While some studies indicate positive effects for distinct subgroups of learners, a common finding is that the majority of learners do not modify their behaviour, even after repeated interventions. In this paper, we assessed one such system and, predictably, did not find measurable improvements in equitable participation. Informed by self-regulated learning theory, we conducted a mixed-methods study to explore the diverse paths that learners take in the self-regulation process initiated by the feedback. We found that the observed deviations from the expected path explain the difficulty in measuring a generalized effect. This study proposes a shift in research focus from merely improving the technological aspects of the system to a human- and pedagogicalcentred redesign that takes special consideration of how learners understand and process feedback to self-regulate their participation.
References
Aguilar, S. J. (2018). Examining the relationship between comparative and self-focused academic data visualizations in at-risk college students’ academic motivation. Journal of Research on Technology in Education, 50(1), 84–103. https://doi.org/10.1080/15391523.2017.1401498
Baanqud, N. S., Al-Samarraie, H., Alzahrani, A. I., & Alfarraj, O. (2020). Engagement in cloud-supported collaborative learning and student knowledge construction: A modeling study. International Journal of Educational Technology in Higher Education, 17(1), 56. https://doi.org/10.1186/s41239-020-00232-z
Bachour, K., Kaplan, F., & Dillenbourg, P. (2010). An interactive table for supporting participation balance in face-to-face collaborative learning. IEEE Transactions on Learning Technologies, 3(3), 203–213. https://doi.org/10.1109/TLT.2010.18
Bain, M., Huh, J., Han, T., & Zisserman, A. (2023). Whisperx: Time-accurate speech transcription of long-form audio. arXiv preprint arXiv:2303.00747.
Bergstrom, T., & Karahalios, K. (2007). Seeing more: Visualizing audio cues. In C. Baranauskas, P. Palanque, J. Abascal, & S. D. J. Barbosa (Eds.), Proceedings of the 11th IFIP Conference on Human-Computer Interaction (IFIP 2007), 10–14 September 2007, Rio de Janiero, Brazil (pp. 29–42). Springer. https://doi.org/10.1007/978-3-540-74800-7_3
Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245–281. https://doi.org/10.3102/00346543065003245
Capdeferro, N., & Romero, M. (2012). Are online learners frustrated with collaborative learning experiences? International Review of Research in Open and Distributed Learning, 13(2), 26–44. https://doi.org/10.19173/irrodl.v13i2.1127
Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354
Choi, H., & Hur, J. (2023). Passive participation in collaborative online learning activities: A scoping review of research in formal school learning settings. Online Learning Journal, 27(1). https://doi.org/10.24059/olj.v27i1.3414
Corrin, L., & De Barba, P. (2014). Exploring students’ interpretation of feedback delivered through learning analytics dashboards. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK 2015), 16–20 March 2015, Poughkeepsie, New York, USA (pp. 629–633). ACM. https://doi.org/10.1145/2723576.2723662
DiMicco, J. M., Pandolfo, A., & Bender, W. (2004). Influencing group participation with a shared display. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (CSCW 2004), 6–10 November 2004, Chicago, Illinois, USA (pp. 614–623). ACM. https://doi.org/10.1145/1031607.1031713
Dorfman, R. (1979). A formula for the Gini coefficient. The Review of Economics and Statistics, 61(1), 146–149. https://doi.org/10.2307/1924845
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160. https://doi.org/10.3758/BRM.41.4.1149
Heikkinen, S., Saqr, M., Malmberg, J., & Tedre, M. (2023). Supporting self-regulated learning with learning analytics interventions—A systematic literature review. Education and Information Technologies, 28(3), 3059–3088. https://doi.org/10.1007/s10639-022-11281-4
Hoadley, C. P. (1994). Creating context: Design-based research in creating and understanding CSCL. In G. Stahl (Ed.), Computer support for collaborative learning (pp. 453–462). Routledge. https://doi.org/10.4324/9781315045467
Holton, J. A. (2007). The coding process and its challenges. In A. Bryant & K. Charmaz (Eds.), The Sage handbook of grounded theory (pp. 265–289, Vol. 3). Sage. https://doi.org/10.4135/9781848607941.n13
Hu, L., & Chen, G. (2021). A systematic review of visual representations for analyzing collaborative discourse. Educational Research Review, 34, 100403. https://doi.org/10.1016/j.edurev.2021.100403
Jackson, K. M., & Trochim, W. M. (2002). Concept mapping as an alternative approach for the analysis of open-ended survey responses. Organizational Research Methods, 5(4), 307–336. https://doi.org/10.1177/109442802237114
Järvelä, S., & Hadwin, A. F. (2013). New frontiers: Regulating learning in CSCL. Educational Psychologist, 48(1), 25–39. https://doi.org/10.1080/00461520.2012.748006
Jivet, I., Scheffel, M., Schmitz, M., Robbers, S., Specht, M., & Drachsler, H. (2020). From students with love: An empirical study on learner goals, self-regulated learning and sense-making of learning analytics in higher education. The Internet and Higher Education, 47, 100758. https://doi.org/10.1016/j.iheduc.2020.100758
Jivet, I.,Wong, J., Scheffel, M., Valle Torre, M., Specht, M., & Drachsler, H. (2021). Quantum of choice: How learners’ feedback monitoring decisions, goals and self-regulated learning skills are related. In Proceedings of the 11th international Conference on Learning Analytics and Knowledge (LAK 2021), 12–16 April 2021, Irvine, California, USA (pp. 416–427). ACM. https://doi.org/10.1145/3448139.3448179
John, O. P., Donahue, E. M., & Kentle, R. L. (1991). Big five inventory [APA PsycTest Database. Accessed: 2024-07-29]. https://doi.org/10.1037/t07550-000
Kim, T., Chang, A., Holland, L., & Pentland, A. S. (2008). Meeting mediator: Enhancing group collaboration using sociometric feedback. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW 2008), 8–12 November 2008, San Diego, California, USA (pp. 457–466). ACM. https://doi.org/10.1145/1460563.1460636
Kulyk, O., Wang, J., & Terken, J. (2006). Real-time feedback on nonverbal behaviour to enhance social dynamics in small group meetings. In S. Renals & S. Bengio (Eds.), Machine Learning for Multimodal Interaction: Second International Workshop (MLMI 2005), 11–13 July 2005, Edinburgh, UK, Revised Selected Papers 2 (pp. 150–161). Springer. https://doi.org/10.1007/11677482_13
Kurasaki, K. S. (2000). Intercoder reliability for validating conclusions drawn from open-ended interview data. Field Methods, 12(3), 179–194. https://doi.org/10.1177/1525822X0001200301
Li, Q., Jung, Y., & Wise, A. (2021). Beyond first encounters with analytics: Questions, techniques and challenges in instructors’ sensemaking. In Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK 2021), 12–16 April 2021, Irvine, California, USA (pp. 344–353). ACM. https://doi.org/10.1145/3448139.3448172
Lipnevich, A., & Panadero, E. (2021). A review of feedback models and theories: Descriptions, definitions, and conclusions. Frontiers in Education, 6, 720195. https://doi.org/10.3389/feduc.2021.720195
Molenaar, I., & Knoop-van Campen, C. A. (2018). How teachers make dashboard information actionable. IEEE Transactions on Learning Technologies, 12(3), 347–355. https://doi.org/10.1109/TLT.2018.2851585
Ochoa, X. (2022). Multimodal learning analytics—Rationale, process, examples, and direction. In C. Lang, G. Siemens, A. Friend Wise, D. GaˇseviÅLc, & A. Merceron (Eds.), The handbook of learning analytics (2nd edition, pp. 54–65). SoLAR. https://doi.org/10.18608/hla22.006
Ochoa, X., Echeverria, V., Carrillo, G., Heredia, V., & Chiluiza, K. (2023). Supporting online collaborative work at scale: A mixed-methods study of a learning analytics tool. In Proceedings of the 10th ACM Conference on Learning @ Scale (L@S 2023), 20–22 July 2023, Copenhagen, Denmark (pp. 237–247). ACM. https://doi.org/10.1145/3573051.3596165
Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422. https://doi.org/10.3389/fpsyg.2017.00422
Panadero, E., Broadbent, J., Boud, D., & Lodge, J. M. (2019). Using formative assessment to influence self-and co-regulated learning: The role of evaluative judgement. European Journal of Psychology of Education, 34, 535–557. https://doi.org/10.1007/s10212-018-0407-8
Patel, R., Tarrant, C., Bonas, S., Yates, J., & Sandars, J. (2015). The struggling student: A thematic analysis from the self-regulated learning perspective. Medical Education, 49(4), 417–426. https://doi.org/10.1111/medu.12651
Pedrotti, M., & Nistor, N. (2019). How students fail to self-regulate their online learning experience. In Transforming Learning with Meaningful Technologies: Proceedings of the 14th European Conference on Technology Enhanced Learning (EC-TEL 2019), 16–19 September 2019, Delft, Netherlands (pp. 377–385, Vol. 14). ACM. https://doi.org/10.1007/978-3-030-29736-7_28
Pell, S. J. (2024). Augmented astronaut survival: Updating the “how to survive on the moon” scenario workshop in preparation for an Artemis edition. In Proceedings of the Augmented Humans International Conference 2024 (AHs 2024), 4–6 April 2024, Melbourne, Australia (pp. 331–341). ACM. https://doi.org/10.1145/3652920.3654916
Praharaj, S., Scheffel, M., Drachsler, H., & Specht, M. (2018). Multimodal analytics for real-time feedback in co-located collaboration. In V. Pammer-Schindler, M. PÅLerez-SanagustÅLın, H. Drachsler, R. Elferink, & M. Scheffel (Eds.), Proceedings of the 13th European Conference on Technology Enhanced Learning (EC-TEL 2018), 3–5 September 2018, Leeds, UK (pp. 187–201). Springer. https://doi.org/10.1007/978-3-319-98572-5_15
Praharaj, S., Scheffel, M., Drachsler, H., & Specht, M. (2021). Literature review on co-located collaboration modeling using multimodal learning analytics—Can we go the whole nine yards? IEEE Transactions on Learning Technologies, 14(3), 367–385. https://doi.org/10.1109/TLT.2021.3097766
Reinig, B. A., & Mejias, R. J. (2014). On the measurement of participation equality. International Journal of e-Collaboration (IJeC), 10(4), 32–48. https://doi.org/10.4018/ijec.2014100103
Schneider, B., Sung, G., Chng, E., & Yang, S. (2021). How can high-frequency sensors capture collaboration? A review of the empirical links between multimodal metrics and collaborative constructs. Sensors, 21(24), 8185. https://doi.org/10.3390/s21248185
Soller, A., MartÅLınez, A., Jermann, P., & Muehlenbrock, M. (2005). From mirroring to guiding: A review of state of the art technology for supporting collaborative learning. International Journal of Artificial Intelligence in Education, 15(4), 261–290. https://dl.acm.org/doi/10.5555/1434935.1434937
Starr, E. L., Reilly, J. M., & Schneider, B. (2018). Toward using multi-modal learning analytics to support and measure collaboration in co-located dyads. In Proceedings of the 13th International Conference of the Learning Sciences (ICLS 2018), 23–27 June 2018, London, UK (Vol. 1). ISLS. https://repository.isls.org//handle/1/888
Strau., S., & Rummel, N. (2021). Promoting regulation of equal participation in online collaboration by combining a group awareness tool and adaptive prompts. But does it even matter? International Journal of Computer-Supported Collaborative Learning, 16, 67–104. https://doi.org/10.1007/s11412-021-09340-y
Strau., S., & Rummel, N. (2023). Feed-back about the collaboration process from a group awareness tool. Potential boundary conditions for effective regulation. In O. Noroozi & B. De Wever (Eds.), The power of peer learning: Fostering students’ learning processes and outcomes (pp. 183–213). Springer. https://doi.org/10.1007/978-3-031-29411-2_9
Terken, J., & Sturm, J. (2010). Multimodal support for social dynamics in co-located meetings. Personal and Ubiquitous Computing, 14, 703–714. https://doi.org/10.1007/s00779-010-0284-x
Van Es, E. A., & Sherin, M. G. (2002). Learning to notice: Scaffolding new teachers’ interpretations of classroom interactions. Journal of Technology and Teacher Education, 10(4), 571–596. https://www.learntechlib.org/primary/p/9171/
Van Leeuwen, A., Knoop-van Campen, C. A., Molenaar, I., & Rummel, N. (2021). How teacher characteristics relate to how teachers use dashboards: Results from two case studies in K-12. Journal of Learning Analytics, 8(2), 6–21. https://doi.org/10.18608/jla.2021.7325
Van Leeuwen, A., Rummel, N., & Van Gog, T. (2019). What information should CSCL teacher dashboards provide to help teachers interpret CSCL situations? International Journal of Computer-Supported Collaborative Learning, 14, 261–289. https://doi.org/10.1007/s11412-019-09299-x
Vogel, F., Weinberger, A., Hong, D., Wang, T., Glazewski, K., Hmelo-Silver, C. E., Uttamchandani, S., Mott, B., Lester, J.,Oshima, J., Oshima, R., Yamashita, S., Lu, J., Brandl, L., Richters, C., Stadler, M., Fischer, F., Radkowitsch, A.,Schmidmaier, R., . . . Noroozi, O. (2023). Transactivity and knowledge co-construction in collaborative problem solving. In J. D. Slotta & E. S. Charles (Eds.), Proceedings of the 16th International Conference on Computer- Supported Collaborative Learning (CSCL 2023), 10–15 June 2023, MontrÅLeal, QuÅLebec, Canada (pp. 337–346). ISLS. https://doi.org/10.22318/cscl2023.646214
Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531–566). Elsevier. https://doi.org/10.1016/B978-012109890-2/50045-7
Wise, A., & Jung, Y. (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://doi.org/10.18608/jla.2019.62.4
Wise, A., Knight, S., & Shum, S. B. (2021). Collaborative learning analytics. In U. Cress, C. RosÅLe, A. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning (pp. 425–443). Springer. https://doi.org/10.1007/978-3-030-65291-3_23
Wise, A., Zhao, Y., & Hausknecht, S. (2014). Learning analytics for online discussions: Embedded and extracted approaches. Journal of Learning Analytics, 1(2), 48–71. https://doi.org/10.18608/jla.2014.12.4
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
