A Closer Look at Instructor Use and Sensemaking Processes of Analytics Dashboards: Past, Present, and Future

Dana AlZoubi; Evrim Baran

doi:10.18608/jla.2024.7961

Authors

Dana AlZoubi Mississippi State University https://orcid.org/0000-0002-0483-6839
Evrim Baran Iowa State University https://orcid.org/0000-0002-1935-0188

DOI:

https://doi.org/10.18608/jla.2024.7961

Keywords:

classroom analytics, automated classroom observation, feedback dashboards, instructor sensemaking, dashboard use, active learning, research paper

Abstract

There is a growing interest in the research and use of automated feedback dashboards that display classroom analytics; yet little is known about the detailed processes instructors use to make sense of these tools, and to determine the impact on their teaching practices. This research was conducted at a public Midwestern university within the context of an automated classroom observation and feedback implementation project. Fifteen engineering instructors engaged in this research. The overarching goal was to investigate instructor teaching beliefs, pedagogical practices, and sensemaking processes regarding dashboard use. A grounded theory approach was used to identify categories related to instructor perceptions. Results revealed that instructor experiences inform both their present use of the dashboard and consequential future actions. A model is presented that illustrates categories included in instructor pre-use, use, and post-use of an automated feedback dashboard. An extension to this model is presented and accompanied by recommendations for a more effective future use of automated dashboards. The model’s practical implications inform both instructors and designers on effective design and use of dashboards, ultimately paving a way to improve pedagogical practices and instruction

References

Ahn, J., Campos, F., Hays, M., & Digiacomo, D. (2019). Designing in context: Reaching beyond usability in learning analytics dashboard design. Journal of Learning Analytics, 6(2), 70–85. https://doi.org/10.18608/jla.2019.62.5

Ahuja, K., Kim, D., Xhakaj, F., Varga, V., Xie, A., Zhang, S., Townsend, J. E., Harrison, C., Ogan, A., & Agarwal, Y. (2019). EduSense: Practical classroom sensing at scale. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(3), 71. ACM Press. https://doi.org/10.1145/3351229

AlZoubi, D., Kelley, J., Baran, E., Gilbert, S. B., & Karabulut Ilgu, A., & Jiang, S. (2021). TEACHActive feedback dashboard: Using automated classroom analytics to visualize pedagogical strategies at a glance. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21), 8–13 May 2021, Yokohama, Japan (article 312). ACM Press. https://doi.org/10.1145/3411763.3451709

Baran, E., AlZoubi, D., & Karabulut-Ilgu, A. (2022). Leveraging engineering instructors’ professional development with classroom analytics. In E. Langran (Ed.), Proceedings of Society for Information Technology & Teacher Education International Conference, 11 April 2022, San Diego, CA, USA (pp. 1769–1775). Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/primary/p/220948/

Baran, E., AlZoubi, D., & Morales, A. S. (2023). Design and implementation of an automated classroom analytics system: Stakeholder engagement and mapping. TechTrends, 67(6), 945–954. https://doi.org/10.1007/s11528-023-00905-2

Bodily, R., & Verbert, K. (2017). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies, 10(4), 405–418. https://doi.org/10.1109/TLT.2017.2740172

Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). How people learn: Brain, mind, experience, and school (Expanded Ed.). National Academy Press.

Brown, M. (2020). Seeing students at scale: How faculty in large lecture courses act upon learning analytics dashboard data. Teaching in Higher Education, 25(4), 384–400. https://doi.org/10.1080/13562517.2019.1698540

Butcher, K. R., & Sumner, T. (2011). Self-directed learning and the sensemaking paradox. Human–Computer Interaction, 26(1–2), 123–159. https://doi.org/10.1080/07370024.2011.556552

Chi, M. T., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219–243. https://doi.org/10.1080/00461520.2014.965823

Corbin, J., & Holt, N. L. (2005). Grounded theory. In B. Somekh & C. Lewin (Eds.), Research methods in the social sciences (pp. 49–55). Sage Publications.

Dawson, S., Joksimović, S., Poquet, O., & Siemens, G. (2019). Increasing the impact of learning analytics. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (pp. 446–455). ACM Press. https://doi.org/10.1145/3303772.3303784

Deakin Crick, R. E., Knight, S., & Barr, S. (2017). Towards analytics for wholistic school improvement: Hierarchical process modelling and evidence visualization. Journal of Learning Analytics, 4(2), 160–188. https://doi.org/10.18608/jla.2017.42.13

Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39), 19251–19257. https://doi.org/10.1073/pnas.1821936116

Felder, R. M., & Brent, R. (2010). The national effective teaching institute: Assessment of impact and implications for faculty development. Journal of Engineering Education, 99(2), 121–134. https://doi.org/10.1002/j.2168-9830.2010.tb01049.x

Fernandez Nieto, G. M., Kitto, K., Buckingham Shum, S., & Martinez-Maldonado, R. (2022). Beyond the learning analytics dashboard: Alternative ways to communicate student data insights combining visualisation, narrative and storytelling. Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK ’22), 21–25 March 2022, Online (pp. 219–229). ACM Press. https://doi.org/10.1145/3506860.3506895

Few, S. (2006). Information dashboard design: The effective visual communication of data. O’Reilly Media, Inc.

Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410–8415. https://doi.org/10.1073/pnas.1319030111

Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002

Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E. (2019). Multimodal data as a means to understand the learning experience. International Journal of Information Management, 48, 108–119. https://doi.org/10.1016/j.ijinfomgt.2019.02.003

Gibbs, G., & Coffey, M. (2004). The impact of training of university teachers on their teaching skills, their approach to teaching and the approach to learning of their students. Active Learning in Higher Education, 5(1), 87–100. https://doi.org/10.1177/1469787404040463

Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012). The student activity meter for awareness and self-reflection. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ12), 5–10 May 2012, Austin, TX, USA (pp. 869–884). ACM Press. https://doi.org/10.1145/2212776.2212860

Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Educational Technology & Society, 15(3), 42–57. https://www.jstor.org/stable/jeductechsoci.15.3.42

Haug, B. S., & Mork, S. M. (2021). Taking 21st century skills from vision to classroom: What teachers highlight as supportive professional development in the light of new demands from educational reforms. Teaching and Teacher Education, 100, 103286. https://doi.org/10.1016/j.tate.2021.103286

Henderson, C., Beach, A., & Finkelstein, N. (2011). Facilitating change in undergraduate STEM instructional practices: An analytic review of the literature. Journal of Research in Science Teaching, 48(8), 952–984. https://doi.org/10.1002/tea.20439

Holstein, K., McLaren, B. M., & Aleven, V. (2018). Informing the design of teacher awareness tools through causal alignment analysis. In J. Kay & R. Luckin (Eds.), Rethinking Learning in the Digital Age: Making the Learning Sciences Count. Proceedings of the 13th International Conference of the Learning Sciences (ICLS ’18), 23–27 June 2018, London, UK (3 volumes, pp. 104–111). International Society of the Learning Sciences. https://repository.isls.org//handle/1/477

Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness is not enough: Pitfalls of learning analytics dashboards in the educational practice. In É. Lavoué, H. Draschler, K. Verbert, J. Broisin, M. Pérez-Sanagustín (Eds.), Data Driven Approaches in Digital Education: Proceedings of the 12th European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 82–96). Springer. https://doi.org/10.1007/978-3-319-66610-5_7

Jivet, I., Scheffel, M., Schmitz, M., Robbers, S., Specht, M., & Drachsler, H. (2020). From students with love: An empirical study on learner goals, self-regulated learning and sense-making of learning analytics in higher education. The Internet and Higher Education, 47, 100758. https://doi.org/10.1016/j.iheduc.2020.100758

Karumbaiah, S., Liu, P., Maksimova, A., De Vylder, L., Rummel, N., & Aleven, V. (2023). Multimodal analytics for collaborative teacher reflection of human–AI hybrid teaching: Design opportunities and constraints. Proceedings of the 18th European Conference on Technology Enhanced Learning (EC-TEL 2023), 4–8 September 2023, Aveiro, Portugal (pp. 580–585). Springer Cham. https://doi.org/10.1007/978-3-031-42682-7_45

Kersting, N. (2008). Using video clips of mathematics classroom instruction as item prompts to measure teachers’ knowledge of teaching mathematics. Educational and Psychological Measurement, 68(5), 845–861. https://doi.org/10.1177/0013164407313369

Kitto, K., Lupton, M., Davis, K., & Waters, Z. (2017). Designing for student-facing learning analytics. Australasian Journal of Educational Technology, 33(5). https://doi.org/10.14742/ajet.3607

Klein, G., Moon, B., & Hoffman, R. R. (2006). Making sense of sensemaking 1: Alternative perspectives. IEEE Intelligent Systems, 21(4), 70–73. https://doi.org/10.1109/MIS.2006.75

Leony, D., Pardo, A., de la Fuente Valentín, L., de Castro, D. S., & Kloos, C. D. (2012). GLASS: A learning analytics visualization tool. In S. Buckingham Shum, D. Gašević, & R. Ferguson (Eds.), Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ʼ12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 162–163). ACM Press. https://doi.org/10.1145/2330601.2330642

Li, Q., Jung, Y., & Friend Wise, A. (2021). Beyond first encounters with analytics: Questions, techniques and challenges in instructors’ sensemaking. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 344–353). ACM Press. https://doi.org/10.1145/3448139.3448172

Marsh, B., & Mitchell, N. (2014). The role of video in teacher professional development. Teacher Development, 18(3), 403–417. https://doi.org/10.1080/13664530.2014.938106

Martinez-Maldonado, R., Echeverria, V., Fernandez Nieto, G., & Buckingham Shum, S. (2020). From data to insights: A layered storytelling approach for multimodal learning analytics. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), 25–30 April 2020, Honolulu, HI, USA (pp. 1–15). ACM Press. https://doi.org/10.1145/3313831.3376148

Martinez-Maldonado, R., Echeverria, V., Mangaroska, K., Shibani, A., Fernandez-Nieto, G., Schulte, J., & Buckingham Shum, S. (2022). Moodoo the tracker: Spatial classroom analytics for characterising teachers’ pedagogical approaches. International Journal of Artificial Intelligence in Education, 32(4), 1025–1051. https://martinezmaldonado.net/files/IJAIED-MoodooV8_clean.pdf

Martinez-Maldonado, R., Echeverria, V., Santos, O. C., Dos Santos, A. D. P., & Yacef, K. (2018). Physical learning analytics: A multimodal perspective. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 375–379). ACM Press. https://doi.org/10.1145/3170358.3170379

Matcha, W., Uzir, N. A., Gašević, D., & Pardo, A. (2019). A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective. IEEE Transactions on Learning Technologies, 13(2), 226–245. https://doi.org/10.1109/TLT.2019.2916802

Miles, M. B., Huberman, A. M., & Saldaña, J. (2020). Qualitative data analysis: A methods sourcebook (4th ed.). Sage Publications.

Molenaar, I., & Knoop-van Campen, C. A. N. (2019). How teachers make dashboard information actionable. IEEE Transactions on Learning Technologies, 12(3), 347–355. https://doi.org/10.1109/TLT.2018.2851585

Molenaar, I., Horvers, A., & Baker, R. S. (2021). What can moment-by-moment learning curves tell about students’ self-regulated learning? Learning and Instruction, 72, 101206. https://doi.org/10.1016/j.learninstruc.2019.05.003

Ndukwe, I. G., & Daniel, B. K. (2020). Teaching analytics, value and tools for teacher data literacy: A systematic and tripartite approach. International Journal of Educational Technology in Higher Education, 17, 22. https://doi.org/10.1186/s41239-020-00201-6

Oleson, A., & Hora, M. T. (2014). Teaching the way they were taught? Revisiting the sources of teaching knowledge and the role of prior experience in shaping faculty teaching practices. Higher Education, 68(1), 29–45. https://doi.org/10.1007/s10734-013-9678-9

Palermo, C., & Thomson, M. M. (2018). Teacher implementation of self-regulated strategy development with an automated writing evaluation system: Effects on the argumentative writing performance of middle school students. Contemporary Educational Psychology, 54, 255–270. https://doi.org/10.1016/j.cedpsych.2018.07.002

Park, Y., & Jo, I.-H. (2015). Development of the learning analytics dashboard to support students’ learning performance. Journal of Universal Computer Science, 21(1), 110–133. https://doi.org/10.3217/jucs-021-01-0110

Prieto, L. P., Sharma, K., Wen, Y., & Dillenbourg, P. (2015). The burden of facilitating collaboration: Towards estimation of teacher orchestration load using eye-tracking measures. In O. Lindwall, P. Hakkinen, T. Koschmann, P. Tchounikine, & S. Ludvigsen (Eds.), Exploring the Material Conditions of Learning: Proceedings of the 11th International Conference on Computer Supported Collaborative Learning (CSCL 2015), 7–11 June 2015, Gothenburg, Sweden (pp. 212–219). International Society of the Learning Sciences. https://repository.isls.org/bitstream/1/410/1/206.pdf

Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223–231. https://doi.org/10.1002/j.2168-9830.2004.tb00809.x

Reeves, T. D., Marbach-Ad, G., Miller, K. R., Ridgway, J., Gardner, G. E., Schussler, E. E., & Wischusen, E. W. (2016). A conceptual framework for graduate teaching assistant professional development evaluation and research. CBE—Life Sciences Education, 15(2). https://doi.org/10.1187/cbe.15-10-0225

Rienties, B., Herodotou, C., Olney, T., Schencks, M., & Boroowa, A. (2018). Making sense of learning analytics dashboards: A technology acceptance perspective of 95 teachers. International Review of Research in Open and Distributed Learning, 19(5), 186–202. https://doi.org/10.19173/irrodl.v19i5.3493

Rocca, K. A. (2010). Student participation in the college classroom: An extended multidisciplinary literature review. Communication Education, 59(2), 185–213. https://doi.org/10.1080/03634520903505936

Santos, J. L., Verbert, K., & Duval, E. (2012). Empowering students to reflect on their activity with StepUp!: Two case studies with engineering students. Proceedings of the 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning. In conjunction with the 7th European Conference on Technology Enhanced Learning (EC-TEL 2012), 18 September 2012, Saarbrücken, Germany (pp. 73–86). CEUR Workshop Proceedings.

Sedrakyan, G., Malmberg, J., Verbert, K., Järvelä, S., & Kirschner, P. A. (2020). Linking learning behavior analytics and learning science concepts: Designing a learning analytics dashboard for feedback to support learning regulation. Computers in Human Behavior, 107, 105512. https://doi.org/10.1016/j.chb.2018.05.004

Seidel, T., & Stürmer, K. (2014). Modeling and measuring the structure of professional vision in preservice teachers. American Educational Research Journal, 51(4), 739–771. https://doi.org/10.3102/0002831214531321

Sergis, S., & Sampson, D. G. (2017). Teaching and learning analytics to support teacher inquiry: A systematic literature review. In A. Peña-Ayala (Eds.) Learning analytics: Fundaments, applications, and trends (pp. 25–63). Springer Cham. https://doi.org/10.1007/978-3-319-52977-6_2

Sherin, M., & van Es, E. (2005). Using video to support teachers’ ability to notice classroom interactions. Journal of Technology and Teacher Education, 13(3), 475–491. https://www.learntechlib.org/primary/p/4824/

Strauss, A. L., & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory, 2nd ed. Sage.

Suresh, A., Jacobs, J., Lai, V., Tan, C., Ward, W., Martin, J. H., & Sumner, T. (2021). Using transformers to provide teachers with personalized feedback on their classroom discourse: The TalkMoves application. arXiv. https://doi.org/10.48550/arXiv.2105.07949

Trigwell, K., & Prosser, M. (2004). Development and use of the approaches to teaching inventory. Educational Psychology Review, 16(4), 409–424. https://doi.org/10.1007/s10648-004-0007-9

Valle, N., Antonenko, P., Dawson, K., & Huggins‐Manley, A. C. (2021). Staying on target: A systematic literature review on learner‐facing learning analytics dashboards. British Journal of Educational Technology, 52(4), 1724–1748. https://doi.org/10.1111/bjet.13089

van Leeuwen, A., Knoop-van Campen, C. A. N., Molenaar, I., & Rummel, N. (2021). How teacher characteristics relate to how teachers use dashboards: Results from two case studies in K–12. Journal of Learning Analytics, 8(2), 6–21. https://doi.org/10.18608/jla.2021.7325

van Leeuwen, A., van Wermeskerken, M., Erkens, G., & Rummel, N. (2017). Measuring teacher sense making strategies of learning analytics: A case study. Learning: Research and Practice, 3(1), 42–58. https://doi.org/10.1080/23735082.2017.1284252

Verbert, K., Govaerts, S., Duval, E., Santos, J. L., Van Assche, F., Parra, G., & Klerkx, J. (2014). Learning dashboards: An overview and future research opportunities. Personal and Ubiquitous Computing, 18(6), 1499–1514. https://doi.org/10.1007/s00779-013-0751-2

Weick, K. E. (1995). Sensemaking in organizations. Sage Publications.

Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. (2005). Organizing and the process of sensemaking. Organization Science, 16(4), 409–421. https://doi.org/10.1287/orsc.1050.0133

Wise, A. F., & Jung, Y. (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://doi.org/10.18608/jla.2019.62.4

Worsley, M., & Blikstein, P. (2015). Leveraging multimodal learning analytics to differentiate student learning strategies. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 360–367). ACM Press. https://doi.org/10.1145/2723576.2723624

A Closer Look at Instructor Use and Sensemaking Processes of Analytics Dashboards

Past, Present, and Future

Authors

DOI:

Keywords:

Abstract

References

Downloads

Published

How to Cite

Issue

Section

License