Are We Living In LA (P)LA Land? Reporting on the Practice of 30 STEM Tutors in their Use of a Learning Analytics Implementation at the Open University

Tom Olney; Steve  Walker; Carlton Wood; Anactoria  Clarke

doi:10.18608/jla.2021.7261

Authors

Tom Olney The Open University https://orcid.org/0000-0003-1132-4230
Steve Walker The Open University https://orcid.org/0000-0002-6838-8334
Carlton Wood The Open University https://orcid.org/0000-0001-5567-1694
Anactoria Clarke The Open University https://orcid.org/0000-0002-5616-2610

DOI:

https://doi.org/10.18608/jla.2021.7261

Keywords:

tutor practice, learning analytics implementation, social informatics, shadow practice, research paper

Abstract

Most higher education institutions view their increasing use of learning analytics as having significant potential to improve student academic achievement, retention outcomes, and learning and teaching practice but the realization of this potential remains stubbornly elusive. While there is an abundance of published research on the creation of visualizations, dashboards, and predictive models, there has been little work done to explore the impact of learning analytics on the actual practice of teachers. Through the lens of social informatics (an approach that views the users of technologies as active social actors whose technological practices constitute a wider socio-technical system) this qualitative study reports on an investigation into the practice of 30 tutors in the STEM faculty at Europe’s largest distance learning organization, The Open University UK (OU). When asked to incorporate learning analytics (including predictive learning analytics) contained in the Early Alert Indicator (EAI) dashboard during the 2017–2018 academic year into their practice, we found that tutors interacted with this dashboard in certain unanticipated ways and developed three identifiable “shadow practices”.

References

Agnihotri, L., & Ott, A. (2014). Building a student at-risk model: An end-to-end perspective from user to data scientist. In J. Stamper et al. (Eds.), Proceedings of the 7th International Conference on Educational Data Mining (EDM2014), 4–7 July 2014, London, UK (pp. 209–212). International Educational Data Mining Society. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.660.2885&rep=rep1&type=pdf

Arthars, N., Dollinger, M., Vigentini, L., Liu, Y.-T., Kondo, E., & King, D. (2019). Empowering teachers to personalise learning support. In D. Ifenthaler, J. Yau, & D. Mah (Eds), Utilising learning analytics to support study success. Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-319-64792-0_13

Bartimote, K., Pardo, A., & Reimann, P. (2019). The perspective realism brings to learning analytics in the classroom. In J. M. Lodge, J. C. Horvath & L. Corrin (Eds.), Learning analytics in the classroom: Translating learning analytics research for teachers. Abingdon-on-Thames, UK: Routledge.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Calvert, C. (2014). Developing a model and applications for probabilities of student success: A case study of predictive analytics. Open Learning: The Journal of Open, Distance and e-Learning, 29(2), 160–173. http://doi.org/10.1080/02680513.2014.931805

Castaneda, L., & Selwyn, N. (2018). More than tool? Making sense of the ongoing digitizations of higher education. International Journal of Education Technology in Higher Education, 15(22), 1–10. https://doi.org/10.1186/s41239-018-0109-y

Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, 18(6), 683–695. https://doi.org/10.1080/13562517.2013.827653

Conole, G. (2012). Designing for learning in an open world. Dordrecht, Netherlands: Springer.

Creanor, L., & Walker, S. (2012). Learning technology in context: A case for the sociotechnical interaction framework as an analytical lens for networked learning research. In L. Dirckinck-Holmfeld, V. Hodgson, & D. McConnell (Eds.), Exploring the pedagogy and practice of networked learning (pp. 173–187). London, UK: Springer. https://doi.org/10.1007/978-1-4614-0496-5_10

Dalziel, J., Conole, G., Wills, S., Walker, S., Bennett, S., Dobozy, E., Cameron, L., Badilescu-Buga, E., & Bower, M. (2016). The Larnaca declaration on learning design. Journal of Interactive Media in Education, 1(7), 1–24. https://doi.org/10.5334/jime.407

Daniel, B. (2015). Big data and analytics in higher education: Opportunities and challenges. British Journal of Educational Technology, 46(5), 904–920. https://doi.org/10.1111/bjet.12230

Davis, F. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008

Edwards, P. (2018). We have been assimilated: Some principles for thinking about algorithmic systems. In U. Schultze, M. Aanestad, M. Mähring, C. Østerlund, & K. Riemer (Eds.), Living with monsters? Social implications of algorithmic phenomena, hybrid agency, and the performativity of technology. Proceedings of the Working Conference on Information Systems and Organizations (IS&O 2018), 11–12 December 2018, San Francisco, CA, USA (pp. 19–27). IFIP Advances in Information and Communication Technology, vol 543. Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-030-04091-8_3

Ferguson, R., Clow, D., Griffiths, D., & Brasher, A. (2019). Moving forward with learning analytics: Expert views. Journal of Learning Analytics, 6(3), 43–59. http://doi.org/10.18608/jla.2019.63.8

Foster, E. (2015). What have we learnt from implementing learning analytics at NTU? Retrieved from https://www.yumpu.com/en/document/view/54413096/what-have-we-learnt-from-implementing-learning-analytics-at-ntu

Gašević, D., Dawson, S., Rogers, T., & Gašević, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting learning success. Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002

Herodotou, C., Hlosta, M., Boroowa, A., Rienties, B., Zdrahal, Z., & Mangafa, C. (2019c). Empowering online teachers through predictive learning analytics. British Journal of Educational Technology, 50(6), 3064–3079. https://doi.org/10.1111/bjet.12853

Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., & Hlosta, M. (2019a). A large-scale implementation of PLA in higher education: The teacher’s role and perspective. Educational Technology Research and Development, 67, 1273–1306. https://doi.org/10.1007/s11423-019-09685-0

Herodotou, C., Rienties, B., Hlosta, M., Boroowa, A., & Mangafa, C. (2020). The scalable implementation of predictive learning analytics at a distance learning university: Insights from a longitudinal case study. The Internet and Higher Education, 45, 1–3. https://doi.org/10.1016/j.iheduc.2020.100725

Herodotou, C., Rienties, B., Verdin, B., & Boroowa, A. (2019b). Predictive Learning Analytics “at scale”: Towards guidelines to successful implementation in higher education based on the case of The Open University UK. Journal of Learning Analytics, 6(1), 85–95. https://doi.org/10.18608/jla.2019.61.5

Hidalgo, R. (2018). Analytics for action: Using data analytics to support students in improving their learning outcomes. In G. Ubachs & L. Konings (Eds.), The envisioning report for empowering universities (2nd ed.), (pp. 6–8). Maastricht: EADTU. Retrieved from https://tinyurl.com/envisioning-report

Hu, Y., Lo, C., & Shih, S. (2014). Developing early warning systems to predict students’ online learning performance. Computers in Human Behaviour, 36, 469–478. https://doi.org/10.1016/j.chb.2014.04.002

Jivet, I., Scheffel, M., Spect, M., & Drachsler, H. (2018). License to evaluate: Preparing learning analytics dashboards for educational practice. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 31–40). New York: ACM. https://doi.org/10.1145/3170358.3170421

Klein, C., Lester, J., Rangwala, H., & Johri, A. (2019). Learning analytics tools in higher education: Adoption at the Intersection of institutional commitment and individual action. The Review of Higher Education, 42(2), 565–593. Baltimore, MD: John Hopkins University Press. https://doi.org/10.1353/rhe.2019.0007

Kling, R. (2007). What is social informatics and why does it matter? The Information Society, 23(4), 205–220. https://doi.org/10.1080/01972240701441556

Knight, S., & Buckingham Shum, S. (2017). Theory and learning analytics. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), The handbook of learning analytics, (pp. 17–22). Beaumont, AB: Society for Learning Analytics Research (SoLAR). https://doi.org/10.18608/hla17

Kop, R., Fournier, H., & Guillaume, D. (2017). A critical perspective on learning analytics and educational data mining. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), The handbook of learning analytics, (pp. 319–326). Beaumont, AB: Society for Learning Analytics Research (SoLAR). https://doi.org/10.18608/hla17

Kuzilek, J., Hlosta, M., Herrmannova, D., Zdrahal, Z., & Wolff, A. (2015). OU analyse: Analysing at-risk students at The Open University. Learning Analytics Review, LAK15-1, 1–16. Retrieved from https://oro.open.ac.uk/42529/1/__userdata_documents4_ctb44_Desktop_analysing-at-risk-students-at-open-university.pdf

Lamb, R., & Kling, R. (2003). Reconceptualising users as social actors in information systems research. MIS Quarterly, 27, 197–235. https://doi.org/10.2307/30036529

Lodge, J., Horvath, J., & Corrin, L. (2019). Introduction. In J. M. Lodge, J. C. Horvath & L. Corrin (Eds.), Learning analytics in the classroom: Translating learning analytics research for teachers. Abingdon-on-Thames, UK: Routledge.

Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, September/October, 31–40. Retrieved from https://er.educause.edu/~/media/files/article-downloads/erm1151.pdf

Maguire, M., & Delahunt, B. (2017). Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars. All Ireland Journal of Higher Education 9(3), 33501–33514. Retrieved from https://ojs.aishe.org/index.php/aishe-j/article/view/335

McCoy, C., & Rosenbaum, H. (2019). Uncovering unintended and shadow practices of users of decision support systems decision support in higher education institutions, Journal of the American Association for Information Science and Technology, 70(4), 370–384. https://doi.org/10.1002/asi.24131

Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, F., & Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behaviour, 76, 703–714. https://doi.org/10.1016/j.chb.2017.03.028

Olney, T., Rienties, B., & Toetenel, L. (2019). Gathering, visualising and interpreting learning design analytics to inform classroom practice and curriculum design: Learning analytics in the classroom. In J. M. Lodge, J. C. Horvath & L. Corrin (Eds.), Learning analytics in the classroom: Translating learning analytics research for teachers, (pp. 71–92). Abingdon-on-Thames, UK: Routledge.

Orlikowski, W., & Gash, D. (1994). Technological frames: Making sense of information technology in organizations. ACM Transactions on Information Systems, 12(2) 174–207. https://doi.org/10.1145/196734.196745

Piderit, S. (2000). Rethinking resistance and recognizing ambivalence: A multidimensional view of attitudes toward an organizational change. The Academy of Management Review, 25(4), 783–794. https://doi.org/10.2307/259206

Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action evaluation framework: A review of evidence-based learning analytics interventions at Open University UK. Journal of Interactive Media in Education, 1(2), 1–12. https://doi.org/10.5334/jime.394

Rienties, B., Herodotou, C., Olney, T., Schencks, M., & Boroowa, A. (2018). Making sense of learning analytics dashboards: A technology acceptance perspective of 95 teachers. The International Review of Research in Open and Distributed Learning, 19(5). https://doi.org/10.19173/irrodl.v19i5.3493

Sawyer, S., & Hartswood, M. (2014). Advancing social informatics. In P. Fichman & H. Rosenbaum (Eds.), Social informatics: Past, present and future (pp. 197–209). Newcastle upon Tyne, UK: Cambridge Scholars Publishing.

Schumacher, C., & Ifenthaler, D. (2018). Features students really expect from learning analytics. Computers in Human Behaviour, 78, 397–407. https://doi.org/10.1016/j.chb.2017.06.030

Schwendimann, B., Rodríguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Shirvani Boroujeni, M., Holzer, A., Gillet, D., & Dillenbourg, P. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30–41. https://doi.org/10.1109/TLT.2016.2599522

Sclater, N., Peasgood, A., & Mullan, J. (2016, April 19). Learning analytics in higher education: A review of UK and international practice. Jisc. Retrieved from https://analytics.jiscinvolve.org/wp/2016/04/19/learning-analytics-in-higher-education-a-review-of-uk-and-international-practice/

Selwyn, N. (2019). What’s the problem with learning analytics? Journal of Learning Analytics, 6(3), 11–19. https://doi.org/10.18608/jla.2019.63.3

Shacklock, X. (2016). From bricks to clicks: The potential of data and analytics in higher education. Policy Connect. Higher Education Commission. Retrieved from https://www.policyconnect.org.uk/hec/research/report-bricks-clicks-potential-data-and-analytics-higher-education

SHEILA. (2018). Supporting higher education to integrate learning analytics framework. Retrieved from https://sheilaproject.eu/wp-content/uploads/2018/08/SHEILA-framework_Version-2.pdf

Simpson, O. (2006). Predicting student success in open and distance learning. Open Learning: The Journal of Open, Distance and e-Learning, 21(2), 125–138. https://doi.org/10.1080/02680510600713110

Slade, S. & Boroowa, A. (2014). Policy on ethical use of student data for learning analytics. Milton Keynes, UK: The Open University. Retrieved from https://help.open.ac.uk/documents/policies/ethical-use-of-student-data

Suchman, L. (1996). Supporting articulation work. In R. Kling (Ed.), Computerization and controversy: Value conflicts and social choices (2nd ed), (pp. 407–423). San Francisco, CA: Morgan Kaufmann.

Tanes, Z., Arnold, K., King, A., & Remnet, M. (2011). Using signals for appropriate feedback: Perceptions and practices. Computers & Education, 57(4), 2414–2422. https://doi.org/10.1016/j.compedu.2011.05.016

TEF. (2019). Teaching excellence and student outcomes framework. Retrieved from https://www.officeforstudents.org.uk/advice-and-guidance/teaching/what-is-the-tef/

Tsai, Y.-S., Moreno-Marcos, P., Jivet, I., Scheffel, M., Tammets, K., Kollom, K., & Gašević, D. (2018). The SHEILA framework: Informing institutional strategies and policy processes of learning analytics. Journal of Learning Analytics, 5(3), 5–20. https://doi.org/10.18608/jla.2018.53.2

van Harmelen, M., & Workman, D. (2012). Analytics for learning and teaching. Technical Report in CETIS Analytics Series, vol. 1, no. 3. Bolton, UK: CETIS.

Walker, S., & Creanor, L. (2009). The STIN in the tale: A sociotechnical interaction perspective on networked learning. Journal of Educational Technology and Society, 12(4), 305–316. Retrieved from https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.303.6882&rep=rep1&type=pdf

Walker, S., Olney, T., Wood, C., Clarke, A., & Dunworth, M. (2019). How do tutors use data to support their students? Open Learning: The Journal of Open, Distance and e-Learning, 34, 118–133. https://doi.org/10.1080/02680513.2018.1554476

White, S., & White, S. (2016). Learning designers in the “third space”: The socio-technical construction of MOOCs and their relationship to educator and learning designer roles in HE. Journal of Interactive Media in Education, 2016(1) 1–12. https://doi.org/10.5334/jime.429

Williamson, B. (2016). Digital education governance: Data visualization, predictive analytics, and real-time policy instruments. Journal of Education Policy, 31(2), 123–141. https://doi.org/10.1080/02680939.2015.1035758

Wise, A., & Vytasek, J. (2017). Learning analytics implementation design. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), The handbook of learning analytics, (pp. 151–160). Beaumont, AB: Society for Learning Analytics Research (SoLAR). https://doi.org/10.18608/hla17

Are We Living In LA (P)LA Land?

Reporting on the Practice of 30 STEM Tutors in their Use of a Learning Analytics Implementation at the Open University

Authors

DOI:

Keywords:

Abstract

References

Downloads

Published

How to Cite

Issue

Section

License