Teaching with Analytics: Towards a Situated Model of Instructional Decision-Making
DOI:
https://doi.org/10.18608/jla.2019.62.4Keywords:
Learning analytics, Analytics use, Data-informed decision-making, Instructional improvement, Instructional dashboards, Pedagogical support, Teaching analytics, Learning analytics design, Learning analytics implementationAbstract
The process of using analytic data to inform instructional decision-making is acknowledged to be complex; however, details of how it occurs in authentic teaching contexts have not been fully unpacked. This study investigated five university instructors’ use of a learning analytics dashboard to inform their teaching. The existing literature was synthesized to create a template for inquiry that guided interviews, and inductive qualitative analysis was used to identify salient emergent themes in how instructors 1) asked questions, 2) interpreted data, 3) took action, and 4) checked impact. Findings showed that instructors did not always come to analytics use with specific questions, but rather with general areas of curiosity. Questions additionally emerged and were refined through interaction with the analytics. Data interpretation involved two distinct activities, often along with affective reactions to data: reading data toidentify noteworthy patterns and explaining their importance in the course using contextual knowledge. Pedagogical responses to the analytics included whole-class scaffolding, targeted scaffolding, and revising course design, as well two new non-action responses: adopting a wait-and-see posture and engaging in deep reflection on pedagogy. Findings were synthesized into a model of instructor analytics use that offers useful categories of activities for future study and support
References
Ali, L., Hatala, M., Gašević, D., & Jovanović, J. (2012). A qualitative evaluation of evolution of a learning analytics tool. Computers & Education, 58(1), 470–489. https://dx.doi.org/10.1016/j.compedu.2011.08.030
Avramides, K., Hunter, J., Oliver, M., & Luckin, R. (2015). A method for teacher inquiry in cross-curricular projects: Lessons from a case study. British Journal of Educational Technology, 46(2), 249–264. https://dx.doi.org/10.1111/bjet.12233
Bakharia, A., Corrin, L., de Barba, P., Kennedy, G., Gašević, D., Mulder, R., Williams, D., Dawson, S., & Lockyer, L. (2016). A conceptual framework linking learning design with learning analytics. In Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 329–338). New York: ACM. http//:dx.doi.org/10.1145/2883851.2883944
Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. Educational Researcher, 33(8), 3–15. https://dx.doi.org/10.3102/0013189X033008003
Borko, H., & Shavelson, R. J. (1990). Teacher decision making. In B. F. Jones & L. Idol (Eds.), Dimensions of thinking and cognitive instruction (pp. 311–340). Mahwah, NJ: Lawrence Erlbaum Associates.
Cochran-Smith, M., & Lytle, S. L. (1999). The teacher research movement: A decade later. Educational Researcher, 28(7), 15–25. https://dx.doi.org/10.3102/0013189X028007015
Cuban, L. (2001). Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard University Press. Darling-Hammond, L., & Richardson, N. (2009). Research review/teacher learning: What matters. Educational
Leadership, 66(5), 46–53.
Dazo, S. L., Stepanek, N. R., Chauhan, A., & Dorn, B. (2017). Examining instructor use of learning analytics. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’17), 6–11 May 2017, Denver, Colorado, USA (pp. 2504–2510). New York: ACM. http://dx.doi.org/10.1145/3027063.3053256
Dillenbourg, P. (2013). Design for classroom orchestration. Computers & Education, 69, 485–492. https://dx.doi.org/10.1016/j.compedu.2013.04.013
Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and implementation of a learning analytics toolkit for teachers. Journal of Educational Technology & Society, 15(3), 58–76.
Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittlemeier, J., Rienties, B., Ullman, T., & Vuorikari,
R. (2016). Research evidence on the use of learning analytics: Implications for education policy. In R. Vuorikari & J. Castano-Munoz (Eds.), A European framework for action on learning analytics. Luxembourg: Joint Research Centre Science for Policy Report. http://dx.doi.org/10.2791/955210
Gibson, W., & Brown, A. (2009). Working with qualitative data. Los Angeles, CA: Sage.
Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Journal of Educational Technology & Society, 15(3), 42–57.
Guba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. Educational Technology Research & Development, 29(2), 75–91. https://dx.doi.org/10.1007/BF02766777
Hansen, C. J., & Wasson, B. (2016). Teacher inquiry into student learning: The TISL heart model and method for use in teachers’ professional development. Nordic Journal of Digital Literacy, 11(1), 24–49. http://dx.doi/10.18261/issn.1891-943x-2016-01-02
Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M., & Naydenova, G. (2017). Implementing predictive learning analytics on a large scale: The teacher’s perspective. In Proceedings of the 7th International Learning Analytics and Knowledge Conference (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 267–271). New York: ACM. http://dx.doi.org/10.1145/3027385.3027397
Herodotou, C., Rienties, B., Verdin, B., & Boroowa, A. (2019). Predictive learning analytics “at scale”: Towards guidelines to successful implementation in higher education based on the case of the Open University UK. Journal of Learning Analytics, 6(1), 85–95. https://doi.org/10.18608/jla.2019.61.5
Holstein, K., Hong, G., Tegene, M., McLaren, B. M., & Aleven, V. (2018). The classroom as a dashboard: Co-designing wearable cognitive augmentation for K–12 teachers. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 79–88). New York: ACM. https://dx.doi.org/10.1145/3170358.3170377
Kitto, K., Buckingham Shum, S., & Gibson, A. (2018). Embracing imperfection in learning analytics. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 451–460). New York: ACM. http://dx.doi.org/10.1145/3170358.3170413
Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. https://dx.doi.org/10.1177/0002764213479367
Maguire, M. (2001). Methods to support human-centred design. International Journal of Human–Computer Studies, 55(4), 587–634. https://dx.doi.org/10.1006/ijhc.2001.0503
McKenney, S., & Mor, Y. (2015). Supporting teachers in data-informed educational design. British Journal of Educational Technology, 46(2), 265–279. https://dx.doi.org/10.1111/bjet.12262
McLaughlin, M. W., & Talbert, J. E. (2006). Building school-based teacher learning communities: Professional strategies to improve student achievement (Vol. 45). New York: Teachers College Press.
Molenaar, I., & Knoop-van Campen, C. (2018). How teachers make dashboard information actionable. IEEE Transactions on Learning Technologies. (Early Access). http://dx.doi.org/10.1109/TLT.2018.2851585
Mor, Y., Ferguson, R., & Wasson, B. (2015). Learning design, teacher inquiry into student learning and learning analytics: A call for action. British Journal of Educational Technology, 46(2), 221–229. https://dx.doi.org/10.1111/bjet.12273
Park, Y., & Jo, I. H. (2015). Development of the learning analytics dashboard to support students’ learning performance. Journal of Universal Computer Science, 21(1), 110–133.
Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action evaluation framework: A review of evidence-based learning analytics interventions at Open University UK. Journal of Interactive Media in Education, 2016(1), 2. http://dx.doi.org/10.5334/jime.394
Rust, F. (2009). Teacher research and the problem of practice. Teachers College Record, 111(8), 1882–1893.
Sergis, S., & Sampson, D. G. (2017). Teaching and learning analytics to support teacher inquiry: A systematic literature review. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends (pp. 25–63). Cham, Switzerland: Springer. https://dx.doi.org/10.1007/978-3-319-52977-6_2
Siemens, G., Gašević, D., Haythornthwaite, C., Dawson, S., Shum, S. S., Ferguson, R., Duval, E., Verbert, K., & Baker, R. S. (2011). Open learning analytics: An integrated & modularized platform. [Concept paper]. Society for Learning
Analytics Research.
Spillane, J. P. (2012). Data in practice: Conceptualizing the data-based decision-making phenomena. American Journal of Education, 118(2), 113–141. https://dx.doi.org/10.1086/663283
Steen, M. (2011). Tensions in human-centred design. CoDesign, 7(1), 45–60. https://dx.doi.org/10.1080/15710882.2011.563314
Tan, J. P-L., Koh, E., & Jonathan, C. R. (2018). Visible teaching in action: Using the WiREAD learning analytics dashboard for pedagogical adaptivity. Paper presented at the American Educational Research Association Annual Conference (AERA 2018), 13–17 April 2018, New York, NY, USA. Retrieved from the AERA Online Paper Repository. http://dx.doi.org/10.302/1321813
Tarmazdi, H., Vivian, R., Szabo, C., Falkner, K., & Falkner, N. (2015). Using learning analytics to visualise computer science teamwork. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE ’15) 4–8 July 2015, Vilnius, Lithuania (pp. 165–170). New York: ACM. http://dx.doi.org/10.1145/2729094.2742613
Tempelaar, D. T., Rienties, B., & Nguyen, Q. (2017). Towards actionable learning analytics using dispositions. IEEE Transactions on Learning Technologies, 10(1), 6–16. http://dx.doi.org10.1109/TLT.2017.2662679
van Harmelen, M., & Workman, D. (2012). Analytics for learning and teaching. CETIS Analytics Series, 1(3), 1–40.
van Leeuwen, A. (2015). Learning analytics to support teachers during synchronous CSCL: Balancing between overview and overload. Journal of Learning Analytics, 2(2), 138–162. https://dx.doi.org/10.18608/jla.2015.22.11
van Leeuwen, A., van Wermeskerken, M., Erkens, G., & Rummel, N. (2017). Measuring teacher sense making strategies of learning analytics: A case study. Learning: Research and Practice, 3(1), 42–58. https://dx.doi.org/10.1080/23735082.2017.1284252
Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboard applications. American Behavioral Scientist, 57(10), 1500–1509. https://dx.doi.org/10.1177/0002764213479363
Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning analytics. In Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 203–211). New York: ACM. http://dx.doi.org/10.1145/2567574.2567588
Wise, A. F., Knight S., & Ochoa, X. (2018). When are learning analytics ready and what are they ready for? Journal of Learning Analytics, 5(3), 1–4. http://dx.doi.org/10.18608/jla.2018.53.1e
Wise, A. F., & Vytasek, J. (2017). Learning analytics implementation design. In C. Lang, G. Siemens, A. F. Wise, & D. Gašević (Eds.), Handbook of learning analytics (pp. 151–160). Beaumont, AB: Society for Learning Analytics Research. http://dx.doi.org/10.18608/hla17.013
Wise, A. F., Vytasek, J. M., Hausknecht, S., & Zhao, Y. (2016). Developing learning analytics design knowledge in the “middle space”: The student tuning model and align design framework for learning analytics use. Online Learning, 20(2), 155–182.
Wise, A. F., Zhao, Y., & Hausknecht, S. (2014). Learning analytics for online discussions: Embedded and extracted approaches. Journal of Learning Analytics, 1(2), 48–71. https://doi.org/10.18608/jla.2014.12.4
Xhakaj, F., Aleven, V., & McLaren, B. M. (2017). Effects of a teacher dashboard for an intelligent tutoring system on teacher knowledge, lesson planning, lessons and student learning. In E. André, R. S. Baker, X. Hu, M. M. T. Rodrigo, & B. du Boulay (Eds.), Proceedings of the 18th International Conference on Artificial Intelligence in Education (AIED 2017), 28 June–1 July 2017, Wuhan, China (pp. 315–329). Cham, Switzerland: Springer. http://dx.doi.org/10.1007/978-3-319- 61425-0_69
Zirkel, S., Garcia, J. A., & Murphy, M. C. (2015). Experience-sampling research methods and their potential for education research. Educational Researcher, 44(1), 7–16 https://dx.doi.org/10.3102/0013189X14566879
Downloads
Published
How to Cite
Issue
Section
License
TEST
