Edulyze: Learning Analytics for Real-World Classrooms at Scale
DOI:
https://doi.org/10.18608/jla.2024.8367Keywords:
large scale analytics, classroom sensing, research tools, learning pedagogy research, data and tools reportAbstract
Classroom sensing systems can capture data on teacher-student behaviours and interactions at a scale far greater than human observers can. These data, translated to multi-modal analytics, can provide meaningful insights to educational stakeholders. However, complex data can be difficult to make sense of. In addition, analyses done on these data are often limited by the organization of the underlying sensing system, and translating sensing data into meaningful insights often requires custom analyses across different modalities. We present Edulyze, an analytics engine that processes complex, multi-modal sensing data and translates them into a unified schema that is agnostic to the underlying sensing system or classroom configuration. We evaluate Edulyze’s performance by integrating three sensing systems (Edusense, ClassGaze, and Moodoo) and then present data analyses of five case studies of relevant pedagogical research questions across these sensing systems. We demonstrate how Edulyze’s flexibility and customizability allow us to answer a broad range of research questions made possible by Edulyze’s translation of a breadth of raw sensing data from different sensing systems into relevant classroom analytics.
References
Ahuja, K., Kim, D., Xhakaj, F., Varga, V., Xie, A., Zhang, S., Townsend, J. E., Harrison, C., Ogan, A., & Agarwal, Y. (2019). Edusense: Practical classroom sensing at scale. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(3), 71. https://doi.org/10.1145/3351229
Ahuja, K., Shah, D., Pareddy, S., Xhakaj, F., Ogan, A., Agarwal, Y., & Harrison, C. (2021). Classroom digital twins with instrumentation-free gaze tracking. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI 2021), 8–13 May 2021, Yokohama, Japan (p. 484). ACM. https://doi.org/10.1145/3411764.3445711
Alerby, E. (2020). The significance of silence in pedagogical settings. In Silence within and beyond pedagogical settings (pp. 43–50). Springer International Publishing. https://doi.org/10.1007/978-3-030-51060-2 4
Alzoubi, D., Kelley, J., Baran, E., B. Gilbert, S., Karabulut Ilgu, A., & Jiang, S. (2021). Teachactive feedback dashboard: Using automated classroom analytics to visualize pedagogical strategies at a glance. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI 2021), 8–13 May 2021, Yokohama, Japan (p. 312). ACM.https://doi.org/10.1145/3411763.3451709
Andersen, J. F., Andersen, P. A., & Jensen, A. D. (1979). The measurement of nonverbal immediacy. Journal of Applied Communication Research, 7(2), 153–180. https://doi.org/10.1080/00909887909365204
Bosch, N., D’mello, S. K., Ocumpaugh, J., Baker, R. S., & Shute, V. (2016). Using video to automatically detect learner affect in computer-enabled classrooms. ACM Transactions on Interactive Intelligent Systems, 6(2), 17. https://doi.org/10.1145/2946837
Cukurova, M., Giannakos, M., & Martinez-Maldonado, R. (2020). The promise and challenges of multimodal learning analytics. British Journal of Educational Technology, 51(5), 1441–1449. https://doi.org/10.1111/bjet.13015
Donnelly, P. J., Blanchard, N., Samei, B., Olney, A. M., Sun, X., Ward, B., Kelly, S., Nystran, M., & D’Mello, S. K. (2016). Automatic teacher modeling from live classroom audio. In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization (UMAP 2016), 13–17 July 2016, Halifax, Nova Scotia, Canada (pp. 45–53). ACM. https://doi.org/10.1145/2930238.2930250
Fong, C. J., Dillard, J. B., & Hatcher, M. (2019). Teaching self-efficacy of graduate student instructors: Exploring faculty motivation, perceptions of autonomy support, and undergraduate student engagement. International Journal of Educational Research, 98, 91–105. https://doi.org/10.1016/j.ijer.2019.08.018
Foster, T. J., Justice, L., Villasanti, H. G., Irvin, D., & Messinger, D. (2024). Classroom sensing tools: Revolutionizing classroom-based research in the 21st century. Topics in Early Childhood Special Education, 0(0), 02711214231220800. https://doi.org/10.1177/02711214231220800
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109. https://doi.org/10.3102/00346543074001059
Gao, N., Marschall, M., Burry, J., Watkins, S., & Salim, F. (2022). Understanding occupants’ behaviour, engagement, emotion, and comfort indoors with heterogeneous sensors and wearables. Scientific Data, 9, 261. https://doi.org/10.1038/s41597-022-01347-w
Gao, N., Rahaman, M. S., Shao, W., Ji, K., & Salim, F. D. (2022). Individual and group-wise classroom seating experience: Effects on student engagement in different courses. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6(3). https://doi.org/10.1145/3550335
Gao, N., Shao, W., Rahaman, M. S., & Salim, F. D. (2020). n-Gage: Predicting in-class emotional, behavioural, and cognitive engagement in the wild. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(3), 79. https://doi.org/10.1145/3411813
Gerritsen, D. (2018, October). A socio-technical approach to feedback and instructional development for teaching assistants [Doctoral dissertation, Carnegie Mellon University]. https://doi.org/10.1184/R1/7195181.v1
Heng, B., Cheong, C., & Taib, F. (2017). Instructional proxemics and its impact on classroom teaching and learning. International Journal of Modern Languages and Applied Linguistics, 1(1), 69–85. https://doi.org/10.24191/ijmal.v1i1.7637
Hesler, M. W. (1972). An investigation of instructor use of space [Doctoral dissertation, Purdue University]. https://www.proquest.com/openview/d0b8ef498d382cf5d2ba338d443730c6/1
Hirsch, T., Soma, C., Merced, K., Kuo, P., Dembe, A., Caperton, D. D., Atkins, D. C., & Imel, Z. E. (2018). “It’s hard to argue with a computer”: Investigating psychotherapists’ attitudes towards automated evaluation. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS 2018), 9–13 June 2018, Hong Kong, China (pp. 559–571). ACM. https://doi.org/10.1145/3196709.3196776
Karen Chen, L., & Gerritsen, D. (2021). Building interpretable descriptors for student posture analysis in a physical classroom. In I.-H. S. Hsiao, S. S. Sahebi, F. Bouchet, & J.-J. Vie (Eds.), Proceedings of the 14th International Conference on Educational Data Mining (EDM 2021), 29 June–2 July 2021, online (pp. 713–717). International Educational Data Mining Society. https://educationaldatamining.org/EDM2021/virtual/static/pdf/EDM21_paper_26.pdf
Kuang, Z., Wang, F., Andrasik, F., & Hu, X. (2024). Instructor’s direct gaze not body orientation affects learning. Journal of Computer Assisted Learning, 40(2), 731–741. https://doi.org/10.1111/jcal.12917
Li, I., Dey, A., & Forlizzi, J. (2010). A stage-based model of personal informatics systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2010), 10–15 April 2010, Atlanta, Georgia, USA (pp. 557–566). ACM. https://doi.org/10.1145/1753326.1753409
Lim, F. V., O’Halloran, K. L., & Podlasov, A. (2012). Spatial pedagogy: Mapping meanings in the use of classroom space. Cambridge Journal of Education, 42(2), 235–251. https://doi.org/10.1080/0305764X.2012.676629
Liu, C., Calvo, R. A., & Lim, R. (2016). Improving medical students’ awareness of their non-verbal communication through automated non-verbal behavior feedback. Frontiers in ICT, 3. https://doi.org/10.3389/fict.2016.00011
Mangaroska, K., Sharma, K., Giannakos, M., Tr.tteberg, H., & Dillenbourg, P. (2018). Gaze insights into debugging behavior using learner-centred analysis. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 7–9 March 2018, Sydney, Australia (pp. 350–359). ACM. https://doi.org/10.1145/3170358.3170386
Martinez-Maldonado, R. (2019). “I spent more time with that team”: Making spatial pedagogy visible using positioning sensors. In Proceedings of the Ninth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, Arizona, USA (pp. 21–25). ACM. https://doi.org/10.1145/3303772.3303818
Martinez-Maldonado, R. (2020). Moodoo: Indoor positioning analytics for characterising classroom teaching. https://gitlab.erc.monash.edu.au/rmat0024/moodoo/-/tree/master/test/Merged%5C%20dataset%5C202018-2019
Martinez-Maldonado, R., Echeverria, V., Fernandez-Nieto, G., Yan, L., Zhao, L., Alfredo, R., Li, X., Dix, S., Jaggard, H., Wotherspoon, R., Osborne, A., Buckingham Shum, S., & GaˇseviÅLc, D. (2023). Lessons learnt from a multimodal learning analytics deployment in-the-wild. ACM Transactions on Computer-Human Interaction, 31(1), 8. https://doi.org/10.1145/3622784
Martinez-Maldonado, R., Echeverria, V., Nieto, G. F., & Buckingham Shum, S. (2020). From data to insights: A layered storytelling approach for multimodal learning analytics. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI 2020), 25–30 April 2020, Honolulu, Hawaii, USA, 1–15. https://doi.org/10.1145/3313831.3376148
Martinez-Maldonado, R., Echeverria, V., Schulte, J., Shibani, A., Mangaroska, K., & Buckingham Shum, S. (2020). Moodoo: Indoor positioning analytics for characterising classroom teaching. In I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millán (Eds.), Proceedings of the 21st International Conference on Artificial Intelligence in Education (AIED 2020), 6–10 July 2020, Ifrane, Morocco (pp. 360–373, Vol. 12163 of Lecture Notes in Artificial Intelligence). Springer. https://doi.org/10.1007/978-3-030-52237-7_29
Martinez-Maldonado, R., Mangaroska, K., Schulte, J., Elliott, D., Axisa, C., & Buckingham Shum, S. (2020). Teacher tracking with integrity: What indoor positioning can reveal about instructional proxemics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1), 22. https://doi.org/10.1145/3381017
Martinez-Maldonado, R., Schulte, J., Echeverria, V., Gopalan, Y., & Buckingham Shum, S. (2020). Where is the teacher? Digital analytics for classroom proxemics. Journal of Computer Assisted Learning, 36(5), 741–762. https://doi.org/10.1111/JCAL.12444
Ochoa, X., Domínguez, F., GuamÅLan, B., Maya, R., Falcones, G., & Castells, J. (2018). The RAP system: automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 7–9 March 2018, Sydney, Australia (pp. 360–364). ACM. https://doi.org/10.1145/3170358.3170406
Ochoa, X., Worsley, M., Weibel, N., & Oviatt, S. (2016). Multimodal learning analytics data challenges. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 498–499). ACM. https://doi.org/10.1145/2883851.2883913
Oviatt, S. (2018). Ten opportunities and challenges for advancing student-centered multimodal learning analytics. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI 2018), 16–20 October 2018, Boulder, Colorado, USA (pp. 87–94). ACM. https://doi.org/10.1145/3242969.3243010
Prieto, L. P., Sharma, K., Dillenbourg, P., & Jeśus, M. (2016, April). Teaching analytics: Towards automatic extraction of orchestration graphs using wearable sensors. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 148–157). ACM. https://doi.org/10.1145/2883851.2883927
Pryor, J. B., Gibbons, F. X., Wicklund, R. A., Fazio, R. H., & Hood, R. (1977). Self-focused attention and self-report validity. Journal of Personality, 45(4), 513–527. https://doi.org/10.1111/j.1467-6494.1977.tb00168.x
Reinholz, D., & Shah, N. (2018). Equity analytics: A methodological approach for quantifying participation patterns in mathematics classroom discourse. Journal for Research in Mathematics Education, 49(2), 140–177. https://doi.org/10.5951/jresematheduc.49.2.0140
Rodríguez-Triana, M., Prieto, L., Vozniuk, A., Shirvani Boroujeni, M., Schwendimann, B., Holzer, A., & Gillet, D. (2016). Monitoring, awareness and reflection in blended technology enhanced learning: A systematic review. International Journal of Technology Enhanced Learning, 9(2/3), 126–150. https://doi.org/10.1504/IJTEL.2017.10005147
Saquib, N., Bose, A., George, D., & Kamvar, S. (2018). Sensei: Sensing educational interaction. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(4), 161. https://doi.org/10.1145/3161172
Schlotterbeck, D., Uribe, P., Araya, R., Jimenez, A., & Caballero, D. (2021). What classroom audio tells about teaching: A cost-effective approach for detection of teaching practices using spectral audio features. In Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK 2021), 12–16 April 2021, Irvine, California, USA (pp. 132–140). ACM. https://doi.org/10.1145/3448139.3448152
Sedova, K., Sedláček, M., Šváríček, R., Majcík, M., Navrátilová, J., Drexlerova, A., Kychler, J., & Šalamounová, Z. (2019). Do those who talk more learn more? The relationship between student classroom talk and student achievement. Learning and Instruction, 63, 101217. https://doi.org/10.1016/j.learninstruc.2019.101217
Si, J., Lin, J., Jiang, F., & Shen, R. (2019). Hand-raising gesture detection in real classrooms using improved R-FCN. Neurocomputing, 359(100), 69–76. https://doi.org/10.1016/j.neucom.2019.05.031
Smith, M. K., Jones, F. H. M., Gilbert, S. L., & Wieman, C. E. (2013). The classroom observation protocol for undergraduate STEM (COPUS): A new instrument to characterize university stem classroom practices. CBE—Life Sciences Education, 12(4), 618–627. https://doi.org/10.1187/cbe.13-08-0154
Sümer, Ö ., Goldberg, P., D’Mello, S., Gerjets, P., Trautwein, U., & Kasneci, E. (2023). Multimodal engagement analysis from facial videos in the classroom. IEEE Transactions on Affective Computing, 14(2), 1012–1027. https://doi.org/10.1109/TAFFC.2021.3127692
Vujovic, M., Hernández-Leo, D., Martinez-Maldonado, R., Cukurova, M., & Spikol, D. (2022). Multimodal learning analytics and the design of learning spaces. In M. Giannakos, D. Spikol, D. Di Mitri, K. Sharma, X. Ochoa, & R. Hammad (Eds.), The multimodal learning analytics handbook (pp. 31–49). Springer International Publishing. https://doi.org/10.1007/978-3-031-08076-0_2
White, J., & Gardner, J. (2013). The classroom x-factor: The power of body language and non-verbal communication in teaching. Routledge. https://doi.org/10.4324/9780203818701
Woolfolk, A. E., & Brooks, D. M. (1985). The influence of teachers’ nonverbal behaviors on students’ perceptions and performance. The Elementary School Journal, 85(4), 513–528. https://doi.org/10.1086/461418
Worsley, M. (2018). Multimodal learning analytics’ past, present, and potential futures. In R. Martinez-Maldonado, V. Echeverria, L. P. Prieto, M. J. Rodriguez-Triana, D. Spikol, M. Curukova, M. Mavrikis, X. Ochoa, & M. Worsley (Eds.), Companion Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 7–9 March 2018, Sydney, Australia (paper 5, Vol. 2163). CEUR-WS.
Xhakaj, F. (2021). Investigating how to support teachers in their teaching and help them improve their practices through data and technology [Doctoral dissertation, Carnegie Mellon University]. https://www.proquest.com/openview/aac5ecc54e2f2d54df0f1202066ee87f/1
Yan, L., Zhao, L., Gasevic, D., & Martinez-Maldonado, R. (2022). Scalability, sustainability, and ethicality of multimodal learning analytics. In Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK 2022), 21–25 March 2022, online (pp. 13–23). ACM. https://doi.org/10.1145/3506860.350686
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
