Tcherly: : A Teacher-facing Dashboard for Online Video Lectures

Pankaj Chavan; Ritayan Mitra

doi:10.18608/jla.2022.7555

Authors

Pankaj Chavan Indian Institute of Technology Bombay https://orcid.org/0000-0002-4538-6731
Ritayan Mitra Indian Institute of Technology Bombay

DOI:

https://doi.org/10.18608/jla.2022.7555

Keywords:

learning analytics, teacher dashboard, student feedback, video leacture, research paper

Abstract

The use of online video lectures in universities, primarily for content delivery and learning, is on the rise. Instructors’ ability to recognize and understand student learning experiences with online video lectures, identify particularly difficult or disengaging content and thereby assess overall lecture quality can inform their instructional practice related to such lectures. This paper introduces Tcherly, a teacher-facing dashboard that presents class-level aggregated time series data on students’ self-reported cognitive-affective states they experienced during a lecture. Instructors can use the dashboard to evaluate and improve their instructional practice related to video lectures. We report the detailed iterative prototyping design process of the Tcherly Dashboard involving two stakeholders (instructors and designers) that informed various design decisions of the dashboard, and also provide usability and usefulness data. We demonstrate, with real-life examples of Tcherly Dashboard use generated by the researchers based on data collected from six courses and 11 lectures, how the dashboard can assist instructors in understanding their students’ learning experiences and evaluating the associated instructional materials.

References

Aldridge, J., Bell, L., & Fraser, B. (2010). Using students’ perceptions of the learning environment to guide teacher action research to improve senior secondary classrooms. Paper presented at the AARE Annual Conference, Canberra 2009. Australian Association for Research in Education. https://www.aare.edu.au/data/publications/2009/ald091340.pdf

Aldridge, J. M., & Fraser, B. J. (2008). Outcomes-focused learning environments: Determinants and effects. Brill Sense. https://brill.com/view/title/37331

Atapattu, T., & Falkner, K. (2018). Impact of lecturer’s discourse for students’ video engagement: Video learning analytics case study of MOOCs. Journal of Learning Analytics, 5(3), 182–197. https://doi.org/10.18608/jla.2018.53.12

Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human–Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776

Bargeron, D., Grudin, J., Gupta, A., Sanocki, E., Li, F., & Leetiernan, S. (2002). Asynchronous collaboration around multimedia applied to on-demand education. Journal of Management Information Systems, 18(4), 117–145. https://doi.org/10.1080/07421222.2002.11045706

Bell, L. M., & Aldridge, J. M. (2014). Investigating the use of student perception data for teacher reflection and classroom improvement. Learning Environments Research, 17(3), 371–388. https://doi.org/10.1007/s10984-014-9164-z

Boyd, D., & Crawford, K. (2012). Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon. Information, Communication & Society, 15(5), 662–679. https://doi.org/10.1080/1369118X.2012.678878

Brooke, J. (1996). SUS: A ʻquick and dirty’ usability scale. Usability Evaluation in Industry, 189(194), 4–7. https://doi.org/10.1201/9781498710411-35

Brooks, C., Amundson, K., & Greer, J. (2009). Detecting significant events in lecture video using supervised machine learning. In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Frontiers in Artificial Intelligence and Applications (pp. 483–490). Artificial Intelligence in Education, vol. 200. IOS Press. http://dx.doi.org/10.3233/978-1-60750-028-5-483

Brooks, C., Demmans Epp, C., Logan, G., & Greer, J. (2011). The who, what, when, and why of lecture capture. In P. Long, G. Siemens, G. Conole, & D. Gašević (Eds.), Proceedings of the 1st International Conference on Learning Analytics and Knowledge: Connecting the technical, pedagogical, and social dimensions of learning analytics (LAK ʼ11), 27 February–1 March 2011, Banff, AB, Canada (pp. 86–92). ACM Press. https://doi.org/10.1145/2090116.2090128

Brooks, C., Greer, J., & Gutwin, C. (2014). The data-assisted approach to building intelligent technology-enhanced learning environments. In J. A. Larusson & B. White (Eds.), Learning analytics: From research to practice (pp. 123–156). Springer. https://doi.org/10.1007/978-1-4614-3305-7_7

Brooks, C., Thompson, C., & Greer, J. (2013). Visualizing lecture capture usage: A learning analytics case study. WAVe 2013 Workshop at LAK ’13, 8 April 2013, Leuven, Belgium, (pp. 9–14). http://ceur-ws.org/Vol-983/paper3.pdf

Bustingorry, S. O. (2008). Towards teachers’ professional autonomy through action research. Educational Action Research, 16(3), 407–420. https://doi.org/10.1080/09650790802260398

Chavan, P., Gupta, S., & Mitra, R. (2018). A novel feedback system for pedagogy refinement in large lecture classrooms. In J. C. Yang, M. Chang, L.-H. Wong, & M. T. Rodrigo (Eds.), Proceedings of the 26th International Conference on Computers in Education (ICCE 2018), 26–30 November 2018, Manila, Philippines (pp. 464–469). Asia-Pacific Society for Computers in Education. https://apsce.net/icce/icce2018/wp-content/uploads/2018/12/C4-14.pdf

Chavan, P., & Mitra, R. (2019). Developing a student feedback system using a design-based research approach. Proceedings of the 10th International Conference on Technology for Education (T4E), 9–11 December 2019, Goa, India (pp. 1–8). IEEE Computer Society. https://doi.org/10.1109/T4E.2019.00-59

Chen, Y., & Hoshower, L. B. (2003). Student evaluation of teaching effectiveness: An assessment of student perception and motivation. Assessment & Evaluation in Higher Education, 28(1), 71–88. https://doi.org/10.1080/02602930301683

Corrin, L., & De Barba, P. (2015). How do students interpret feedback delivered via dashboards? Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 430–431). ACM Press. https://doi.org/10.1145/2723576.2723662

Cross, A., Bayyapunedi, M., Cutrell, E., Agarwal, A., & Thies, W. (2013). TypeRighting: Combining the benefits of handwriting and typeface in online educational videos. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ13), 27 April–2 May 2013, Paris, France (pp. 793–796). ACM Press. https://doi.org/10.1145/2470654.2470766

Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. HarperPerennial.

de Barba, P. G., Kennedy, G. E., & Ainley, M. D. (2016). The role of students’ motivation and participation in predicting performance in a MOOC. Journal of Computer Assisted Learning, 32(3), 218–231. https://doi.org/10.1111/jcal.12130

Demmans Epp, C., Perez, R., Phirangee, K., Hewitt, J., & Toope, K. (2019). User-centered dashboard design: Iterative design to support teacher informational needs in online learning contexts. Paper presented at the American Educational Research Association (AERA) Annual Meeting, Toronto, Canada, April, 5–9. https://doi.org/10.3102/1432967

Ellis, R. A., Han, F., & Pardo, A. (2017). Improving learning analytics: Combining observational and self-report data on student learning. Journal of Educational Technology & Society, 20(3), 158–169. https://www.jstor.org/stable/10.2307/26196127

Fiorini, S., Sewell, A., Bumbalough, M., Chauhan, P., Shepard, L., Rehrey, G., & Groth, D. (2018). An application of participatory action research in advising-focused learning analytics. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 89–96). ACM Press. https://doi.org/10.1145/3170358.3170387

Fraser, B. J. (2012). Classroom learning environments: Retrospect, context and prospect. In B. J. Fraser, K. Tobin, & C. J. McRobbie (Eds.), Second international handbook of science education (pp. 1191–1239). Springer. https://doi.org/10.1007/978-1-4020-9041-7_79

Gajos, K. Z., Kim, J., Li, S.-W., Cai, C. J., & Miller, R. C. (2014). Leveraging video interaction data and content analysis to improve video learning. In Proceedings of the CHI 2014 Learning Innovation at Scale workshop, 27 April 2014, Toronto, Canada. https://dash.harvard.edu/handle/1/22719144

Gašević, D., Mirriahi, N., Dawson, S., & Joksimović, S. (2017). Effects of instructional conditions and experience on the adoption of a learning tool. Computers in Human Behavior, 67, 207–220. https://doi.org/10.1016/j.chb.2016.10.026

Graesser, A. C., & D’Mello, S. (2012). Emotions during the learning of difficult material. In B. H. Ross (Ed.), The Psychology of Learning and Motivation (pp. 183–225). https://doi.org/10.1016/B978-0-12-394293-7.00005-4

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102%2F003465430298487

Hoban, G., & Hastings, G. (2006). Developing different forms of student feedback to promote teacher reflection: A 10-year collaboration. Teaching and Teacher Education, 22(8), 1006–1019. http://dx.doi.org/10.1016/j.tate.2006.04.006

Holstein, K., Hong, G., Tegene, M., McLaren, B. M., & Aleven, V. (2018). The classroom as a dashboard: Co-designing wearable cognitive augmentation for K–12 teachers. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 79–88). ACM Press. https://doi.org/10.1145/3170358.3170377

Jacobs, L. C. (1987). University faculty and students’ opinions of student ratings. Indiana Studies in Higher Education No. 55.

Kay, R. H. (2012). Exploring the use of video podcasts in education: A comprehensive review of the literature. Computers in Human Behavior, 28(3), 820–831. https://doi.org/10.1016/j.chb.2012.01.011

Kim, J., Guo, P. J., Seaton, D. T., Mitros, P., Gajos, K. Z., & Miller, R. C. (2014). Understanding in-video dropouts and interaction peaks in online lecture videos. Proceedings of the 1st ACM Conference on Learning @ Scale (L@S 2014), 4–5 March 2014, Atlanta, GA, USA (pp. 31–40). ACM Press. https://doi.org/10.1145/2556325.2566237

Kizilcec, R. F., Bailenson, J. N., & Gomez, C. J. (2015). The instructor’s face in video instruction: Evidence from two large-scale field studies. Journal of Educational Psychology, 107(3), 724. https://psycnet.apa.org/doi/10.1037/edu0000013

Kizilcec, R. F., Papadopoulos, K., & Sritanyaratana, L. (2014). Showing face in video instruction: Effects on information retention, visual attention, and affect. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ14), 26 April–1 May 2014, Toronto, ON, Canada (pp. 2095–2102). ACM Press. https://doi.org/10.1145/2556288.2557207

Larson, R., & Csikszentmihalyi, M. (2014). The experience sampling method. In Flow and the foundations of positive psychology: The collected works of Mihaly Csikszentmihalyi (pp. 21–34). Springer. https://doi.org/10.1007/978-94-017-9088-8_2

Li, N., Kidziński, Ł., Jermann, P., & Dillenbourg, P. (2015). MOOC video interaction patterns: What do they tell us? Proceedings of the 10th European Conference on Technology Enhanced Learning: Design for Teaching and Learning in a Networked World (EC-TEL 2015), 15–17 September 2015, Toledo, Spain (pp. 197–210). Lecture Notes in Computer Science, vol. 9307. Springer. https://doi.org/10.1007/978-3-319-24258-3_15

Marsh, H. W. (1987). Students’ evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11(3), 253–388. https://doi.org/10.1016/0883-0355(87)90001-2

Marsh, H. W., & Roche, L. (1993). The use of students’ evaluations and an individually structured intervention to enhance university teaching effectiveness. American Educational Research Journal, 30(1), 217–251. https://doi.org/10.2307/1163195

Michos, K., Lang, C., Hernández-Leo, D., & Price-Dennis, D. (2020). Involving teachers in learning analytics design: Lessons learned from two case studies. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 94–99). ACM Press. https://doi.org/10.1145/3375462.3375507

Mirriahi, N., & Vigentini, L. (2017). Analytics of learner video use. C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.). (2017). The handbook of learning analytics, 1st ed. (pp. 251–267). Society for Learning Analytics Research (SoLAR). https://doi.org/10.18608/hla17.022

Mitra, R., & Chavan, P. (2019). DEBE feedback for large lecture classroom analytics. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (pp. 426–430). ACM Press. ACM. https://doi.org/10.1145/3303772.3303821

Montrieux, H., Vangestel, S., Raes, A., Matthys, P., & Schellens, T. (2015). Blending face-to-face higher education with web-based lectures: Comparing different didactical application scenarios. Educational Technology and Society, 18(1), 170–182. https://www.jstor.org/stable/jeductechsoci.18.1.170

Morris, L. V. (2009). Little lectures? Innovative Higher Education, 34(2), 67–68. https://doi.org/10.1007/s10755-009-9108-1

Namuddu, J., & Watts, P. N. (2020). Choice of device to view video lectures: An analysis of two independent cohorts of first-year university students. Research in Learning Technology, 28. https://doi.org/10.25304/rlt.v28.2324

Newtson, D. (1973). Attribution and the unit of perception of ongoing behavior. Journal of Personality and Social Psychology, 28(1), 28. https://doi.org/10.1037/h0035584

Pekrun, R. (2006). The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educational Psychology Review, 18(4), 315–341. https://doi.org/10.1007/s10648-006-9029-9

Pham, P., & Wang, J. (2015). AttentiveLearner: Improving mobile MOOC learning via implicit heart rate tracking. In C. Conati, N. Heffernan, A. Mitrovic, & M. Verdejo (Eds.), Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED ʼ15), 22–26 June 2015, Madrid, Spain (pp. 367–376). Springer. https://doi.org/10.1007/978-3-319-19773-9_37

Pham, P., & Wang, J. (2017). AttentiveLearner2: A multimodal approach for improving MOOC learning on mobile devices. In E. André, R. S. Baker, X. Hu, M. M. T. Rodrigo, & B. du Boulay (Eds.), Proceedings of the 18th International Conference on Artificial Intelligence in Education (AIED 2017), 28 June–1 July 2017, Wuhan, China (pp. 561–564). Springer. https://doi.org/10.1007/978-3-319-61425-0_64

Pham, P., & Wang, J. (2018a). Adaptive review for mobile MOOC learning via multimodal physiological signal sensing: A longitudinal study. Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI ’18) 14–18 October 2019, Bolder, CO, USA (pp. 63–72). ACM Press. https://doi.org/10.1145/3242969.3243002

Pham, P., & Wang, J. (2018b). Predicting learners’ emotions in mobile MOOC learning via a multimodal intelligent tutor. In R. Nkambou, R. Azevedo, & J. Vassileva (Eds.), Proceedings of the 14th International Conference on Intelligent Tutoring Systems (ITS 2018), 11–15 June 2018, Montreal, QC, Canada (Lecture Notes in Computer Science, vol. 10858, pp. 150–159). Springer. https://doi.org/10.1007/978-3-319-91464-0_15

Preston, G., Phillips, R., Gosper, M., McNeill, M., Woo, K., & Green, D. (2010). Web-based lecture technologies: Highlighting the changing nature of teaching and learning. Australasian Journal of Educational Technology, 26(6). https://doi.org/10.14742/ajet.1038

Rhine, S. (1998). The role of research and teachers’ knowledge base in professional development. Educational Researcher, 27(5), 27–31. https://doi.org/10.2307/1176739

Richardson, J. T. E. (2005). Instruments for obtaining student feedback: A review of the literature. Assessment & Evaluation in Higher Education, 30(4), 387–415. https://doi.org/10.1080/02602930500099193

Risko, E. F., Foulsham, T., Dawson, S., & Kingstone, A. (2012). The collaborative lecture annotation system (CLAS): A new TOOL for distributed learning. IEEE Transactions on Learning Technologies, 6(1), 4–13. https://doi.org/10.1109/TLT.2012.15

Rivera-Pelayo, V., Munk, J., Zacharias, V., & Braun, S. (2013). Live interest meter: Learning from quantified feedback in mass lectures. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 23–27). ACM Press. https://doi.org/10.1145/2460296.2460302

Roche, L. A., & Marsh, H. W. (2002). Teaching self-concept in higher education. In N. Hativa & P. Goodyear (Eds.), Teacher thinking, beliefs and knowledge in higher education (pp. 179–218). Springer. https://doi.org/10.1007/978-94-010-0593-7_9

Santagata, R., Zannoni, C., & Stigler, J. W. (2007). The role of lesson analysis in pre-service teacher education: An empirical investigation of teacher learning from a virtual video-based field experience. Journal of Mathematics Teacher Education, 10(2), 123–140. https://doi.org/10.1007/s10857-007-9029-9

Scagnoli, N. I., Choo, J., & Tian, J. (2017). Students’ insights on the use of video lectures in online classes. British Journal of Educational Technology, 50(1), 399–414. https://doi.org/10.1111/bjet.12572

Scheffel, M. (2017). The evaluation framework for learning analytics. Open Universiteit in the Netherlands, Welten Institute, Research Centre for Learning, Teaching and Technology. https://core.ac.uk/download/pdf/95759389.pdf

Scheffel, M., Drachsler, H., Stoyanov, S., & Specht, M. (2014). Quality indicators for learning analytics. Journal of Educational Technology & Society, 17(4), 117–132. https://www.jstor.org/stable/jeductechsoci.17.4.117

Schön, D. A. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. Jossey-Bass.

Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., Gillet, D., & Dillenbourg, P. (2016). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30–41. https://doi.org/10.1109/TLT.2016.2599522

Sharma, K., Alavi, H. S., Jermann, P., & Dillenbourg, P. (2016). A gaze-based learning analytics model: In-video visual feedback to improve learner’s attention in MOOCs. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 417–421). ACM Press. https://doi.org/10.1145/2883851.2883902

Speer, N. K., Swallow, K. M., & Zacks, J. M. (2003). Activation of human motion processing areas during event perception. Cognitive, Affective, & Behavioral Neuroscience, 3(4), 335–345. https://doi.org/10.3758/CABN.3.4.335

Srivastava, N., Nawaz, S., Newn, J., Lodge, J., Velloso, E., M. Erfani, S., Gašević, D., & Bailey, J. (2021). Are you with me? Measurement of learners’ video-watching attention with eye tracking. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 88–98). ACM Press. https://doi.org/10.1145/3448139.3448148

Srivastava, N., Velloso, E., Lodge, J. M., Erfani, S., & Bailey, J. (2019). Continuous evaluation of video lectures from real-time difficulty self-report. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 4–9 May 2019, Glasgow, Scotland, UK (Paper No. 586). ACM Press. https://doi.org/10.1145/3290605.3300816

Stöhr, C., Demazière, C., & Adawi, T. (2020). The polarizing effect of the online flipped classroom. Computers & Education, 147, 103789. https://doi.org/10.1016/j.compedu.2019.103789

Tze, V. M., Daniels, L. M., & Klassen, R. M. (2016). Evaluating the relationship between boredom and academic outcomes: A meta-analysis. Educational Psychology Review, 28(1), 119–144. http://dx.doi.org/10.1007/s10648-015-9301-y

van Es, E. A., Stockero, S. L., Sherin, M. G., Van Zoest, L. R., & Dyer, E. (2015). Making the most of teacher self-captured video. Mathematics Teacher Educator, 4(1), 6–19. https://doi.org/10.5951/mathteaceduc.4.1.0006

Vatrapu, R. K. (2012). Towards semiology of teaching analytics. Workshop Towards Theory and Practice of Teaching Analytics at the European Conference on Technology Enhanced Learning (TaPTA-2012) 18 September 2012, Saarbrücken, Germany. CEUR Workshop Proceedings. http://ceur-ws.org/Vol-894/paper1.pdf

Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management Science, 46(2), 186–204. http://dx.doi.org/10.1287/mnsc.46.2.186.11926

Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboard applications. American Behavioral Scientist, 57(10), 1500–1509. https://doi.org/10.1177%2F0002764213479363

Verbert, K., Govaerts, S., Duval, E., Santos, J. L., Assche, F., Parra, G., & Klerkx, J. (2014). Learning dashboards: An overview and future research opportunities. Personal and Ubiquitous Computing, 18(6), 1499–1514. https://doi.org/10.1007/s00779-013-0751-2

Wise, A. F., & Jung, Y. (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://doi.org/10.18608/jla.2019.62.4

Xhakaj, F., Aleven, V., & McLaren, B. M. (2016). How teachers use data to help students learn: Contextual inquiry for the design of a dashboard. Proceedings of the 11th European Conference on Technology Enhanced Learning (EC-TEL 2016), 13–16 September 2016, Lyon, France (pp. 340–354). Lecture Notes in Computer Science, vol. 9891. Springer. https://doi.org/10.1007/978-3-319-45153-4_26

Xiao, X., Pham, P., & Wang, J. (2017). Dynamics of affective states during MOOC learning. In E. André, R. S. Baker, X. Hu, M. M. T. Rodrigo, & B. du Boulay (Eds.), Proceedings of the 18th International Conference on Artificial Intelligence in Education (AIED 2017), 28 June–1 July 2017, Wuhan, China (pp. 586–589). Springer. http://dx.doi.org/10.1007/978-3-319-61425-0_70

Xiao, X., & Wang, J. (2015). Towards attentive, bi-directional MOOC learning on mobile devices. Proceedings of the 17th ACM International Conference on Multimodal Interaction (ICMI ’15), 9–13 November 2015, Seattle, WA, USA (pp. 163–170). ACM Press. https://doi.org/10.1145/2818346.2820754

Xiao, X., & Wang, J. (2016). Context and cognitive state triggered interventions for mobile MOOC learning. Proceedings of the 18th ACM International Conference on Multimodal Interaction (ICMI ’16), 12–16 November 2016, Tokyo, Japan (pp. 378–385). ACM Press.. https://doi.org/10.1145/2993148.2993177

Yoon, M., Lee, J., & Jo, I.-H. (2021). Video learning analytics: Investigating behavioral patterns and learner clusters in video-based online learning. The Internet and Higher Education, 50, 100806. https://doi.org/10.1016/j.iheduc.2021.100806

Tcherly:

A Teacher-facing Dashboard for Online Video Lectures

Authors

DOI:

Keywords:

Abstract

References

Downloads

Published

How to Cite

Issue

Section

License