More Than Figures on Your Laptop: (Dis)trustful Implementation of Learning Analytics
DOI:
https://doi.org/10.18608/jla.2021.7379Keywords:
learning analytics, trust, institutional adoption, human-computer interaction, dataficationAbstract
The adoption of learning analytics (LA) in complex educational systems is woven into sociocultural and technical challenges that have induced distrust in data and difficulties in scaling LA. This paper presents a study that investigated areas of distrust and threats to trustworthy LA through a series of consultations with teaching staff and students at a large UK university. Surveys and focus groups were conducted to explore participant expectations of LA. The observed distrust is broadly attributed to three areas: the subjective nature of numbers, the fear of power diminution, and approaches to design and implementation of LA. The paper highlights areas to maintain existing trust with policy procedures and areas to cultivate trust by engaging with tensions arising from the social process of LA.
References
Arnold, K. E., Lonn, S., & Pistilli, M. D. (2014). An exercise in institutional reflection: The learning analytics readiness instrument (LARI). In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (LAK 2014), 24–28 March 2014, Indianapolis, IN, USA (pp. 163–167). New York: ACM. https://doi.org/10.1145/2567574.2567621
Baker, R., Xu, D., Park, J., Yu, R., Li, Q., Cung, B., . . . Smyth, P. (2020). The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: Opening the black box of learning processes. International Journal of Educational Technology in Higher Education, 17, 1–24. https://doi.org/10.1186/s41239-020-00187-1
Bakharia, A., Corrin, L., De Barba, P., Kennedy, G., Gašević, D., Mulder, R., . . . Lockyer, L. (2016). A conceptual framework linking learning design with learning analytics. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 329–338). New York: ACM. https://doi.org/10.1145/2883851.2883944
Beer, D. (2018). The Data Gaze: Capitalism, Power and Perception. Sage.
Brown, M. (2020). Seeing students at scale: How faculty in large lecture courses act upon learning analytics dashboard data. Teaching in Higher Education, 25(4), 384–400. https://doi.org/10.1080/13562517.2019.1698540
Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-centred learning analytics. Journal of Learning Analytics, 6(2), 1–9. https://doi.org/10.18608/jla.2019.62.1
Castañeda, L., & Selwyn, N. (2018). More than tools? Making sense of the ongoing digitizations of higher education. International Journal of Educational Technology in Higher Education, 15, 22. https://doi.org/10.1186/s41239-018-0109-y
Chen, B., & Zhu, H. (2019). Towards value-sensitive learning analytics design. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, AZ, USA (pp. 343–352). New York: ACM. https://doi.org/10.1145/3303772.3303798
Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155. https://doi.org/10.1037/0033-2909.112.1.155
Corrin, L., Kennedy, G., & Mulder, R. (2013). Enhancing learning analytics by understanding the needs of teachers. In Proceedings of the Australian Society for Computers in Learning in Tertiary Education 30th Annual Conference (ASCILITE 2013), 1–4 December 2013, Sydney, Australia (pp. 201–205). Retrieved from https://ascilite.org/conferences/sydney13/program/papers/Corrin.pdf
Davis, F. D., Bagozzi, R. P., &Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management science, 35(8), 982–1003. Retrieved from https://www.jstor.org/stable/2632151
Dawson, S., Joksimovic, S., Poquet, O., & Siemens, G. (2019). Increasing the impact of learning analytics. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, AZ, USA (pp. 446–455). New York: ACM. https://doi.org/10.1145/3303772.3303784
Dawson, S., Poquet, O., Colvin, C., Rogers, T., Pardo, A., & Gašević, D. (2018). Rethinking learning analytics adoption through complexity leadership theory. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 5–9 March 2018, Sydney, Australia (pp. 236–244). New York: ACM. https://doi.org/10.1145/3170358.3170375
Dollinger, M., Liu, D., Arthars, N., & Lodge, J. (2019). Working together in learning analytics towards the co-creation of value. Journal of Learning Analytics, 6(2), 10–26. https://doi.org/10.18608/jla.2019.62.2
Dollinger, M., & Lodge, J. M. (2018). Co-creation strategies for learning analytics. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 5–9 March 2018, Sydney, Australia (pp. 97–101). New York: ACM. https://doi.org/10.1145/3170358.3170372
Drachsler, H., & Greller, W. (2016). Privacy and analytics: It’s a DELICATE issue. a checklist for trusted learning analytics. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 89–98). New York: ACM. https://doi.org/10.1145/2883851.2883893
Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304–317. https://doi.org/10.1504/IJTEL.2012.051816
Ferguson, R., Clow, D., Griffiths, D., & Brasher, A. (2019). Moving forward with learning analytics: Expert views. Journal of Learning Analytics, 6(3), 43–59. https://doi.org/10.18608/jla.2019.63.8
Ferguson, R., Hoel, T., Scheffel, M., & Drachsler, H. (2016). Guest editorial: Ethics and privacy in learning analytics. Journal of Learning Analytics, 3(1), 5–15. https://doi.org/10.18608/jla.2016.31.2
Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://doi.org/10.1007/s11528-014-0822-x
Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the learning analytics puzzle: A consolidated model of a field of research and practice. Learning: Research and Practice, 3(1), 63–78. https://doi.org/10.1080/23735082.2017.1286142
Gašević, D., Tsai, Y.-S., Dawson, S., & Pardo, A. (2019). How do we start? An approach to learning analytics adoption in higher education. The International Journal of Information and Learning Technology, 36(4), 342–353. https://doi.org/10.1108/IJILT-02-2019-0024
Goodyear, P. (2015). Teaching as design. HERDSA Review of Higher Education, 2(2), 27–50.
Gray, G., Schalk, A., Rooney, P., & Lang, C. (2021). A stakeholder informed professional development framework to support engagement with learning analytics. In Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK 2021), 12–16 April 2021, online (pp. 237–247). New York: ACM. https://doi.org/10.1145/3448139.3448162
Grbich, C. (2012). Qualitative Data Analysis: An Introduction. Sage.
Hakami, E., & Hernández-Leo, D. (2020). How are learning analytics considering the societal values of fairness, accountability, transparency and human well-being? A literature review. In LASI-SPAIN 2020: Learning Analytics Summer Institute (Spain 2020: Learning Analytics. Time for Adoption?), 15–16 June 2020, Valladolid, Spain (pp. 121–141). Aachen, Germany: CEUR. Retrieved from http://ceur-ws.org/Vol-2671/paper12.pdf
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., & Hlosta, M. (2019). A large-scale implementation of predictive learning analytics in higher education: The teachers’ role and perspective. Educational Technology Research and Development, 67(5), 1273–1306. https://doi.org/10.1007/s11423-019-09685-0
Holstein, K., McLaren, B. M., & Aleven, V. (2019). Co-designing a real-time classroom orchestration tool to support teacher–AI complementarity. Journal of Learning Analytics, 6(2), 27–52. https://doi.org/10.18608/jla.2019.62.3
Howell, J. A., Roberts, L. D., Seaman, K., & Gibson, D. C. (2018). Are we on our way to becoming a “helicopter university”? Academics’ views on learning analytics. Technology, Knowledge and Learning, 23(1), 1–20. https://doi.org/10.1007/s10758-017-9329-9
Ifenthaler, D., & Schumacher, C. (2016). Student perceptions of privacy principles for learning analytics. Educational Technology Research and Development, 64(5), 923–938. https://doi.org/10.1007/s11423-016-9477-y
Jones, K. M., Asher, A., Goben, A., Perry, M. R., Salo, D., Briney, K. A., & Robertshaw, M. B. (2020). “We’re being tracked at all times”: Student perspectives of their privacy in relation to learning analytics in higher education. Journal of the Association for Information Science and Technology. https://doi.org/10.1002/asi.24358
Jones, K. M., Rubel, A., & LeClere, E. (2019). A matter of trust: Higher education institutions as information fiduciaries in an age of educational data mining and learning analytics. Journal of the Association for Information Science andTechnology, 71(10), 1227–1241. https://doi.org/10.1002/asi.24327
Jovanovic, J., Gasevic, D., Brooks, C., Devedzic, V., Hatala, M., Eap, T., & Richards, G. (2007). Using semantic web technologies to analyze learning content. IEEE Internet Computing, 11(5), 45–53. https://doi.ieeecomputersociety.org/10.1109/MIC.2007.116
Klein, C., Lester, J., Rangwala, H., & Johri, A. (2019). Learning analytics tools in higher education: Adoption at the intersection of institutional commitment and individual action. The Review of Higher Education, 42(2), 565–593. https://doi.org/10.1353/rhe.2019.0007
Knox, J. (2017). Data power in education: Exploring critical awareness with the “Learning Analytics Report Card”. Television & New Media, 18(8), 734–752. https://doi.org/10.1177/1527476417690029
Kwet, M., & Prinsloo, P. (2020). The “smart” classroom: A new frontier in the age of the smart university. Teaching in Higher Education, 1–17. https://doi.org/10.1080/13562517.2020.1734922
Lang, C., Macfadyen, L. P., Slade, S., Prinsloo, P., & Sclater, N. (2018). The complexities of developing a personal code of ethics for learning analytics practitioners: Implications for institutions and the field. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 5–9 March 2018, Sydney, Australia (pp. 436–440). New York: ACM. https://doi.org/10.1145/3170358.3170396
Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education—A literature review. In A. Peña-Ayala (Ed.), Learning Analytics: Fundaments, Applications, and Trends (pp. 1–24). Springer. https://doi.org/10.1007/978-3-319-52977-6_1
Lodge, J. M., & Corrin, L. (2017). What data and analytics can and do say about effective learning. npj Science of Learning, 2, 5. https://doi.org/10.1038/s41539-017-0006-5
Lodge, J. M., & Lewis, M. J. (2012). Pigeon pecks and mouse clicks: Putting the learning back into learning analytics. In M. Brown, M. Hartnett, & T. Stewart (Eds.), Proceedings of the 29th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education (ASCILITE 2012), 25–28 November 2012, Wellington, New Zealand (pp. 560–564). Australian Society for Computers in Learning in Tertiary Education. Retrieved from https://www.ascilite.org/conferences/Wellington12/2012/images/custom/lodge,_jason_-_pigeon_pecks.pdf
Long, P., Siemens, G., Conole, G., & Gašević, D. (2011). Proceedings of the International Conference on Learning Analytics and Knowledge. In Proceedings of the First International Conference on Learning Analytics and Knowledge (LAK 2011), 27 February–1 March 2011, Banff, AB, Canada. New York: ACM.
Macfadyen, L. P., Dawson, S., Pardo, A., & Gašević, D. (2014). Embracing big data in complex educational systems: The learning analytics imperative and the policy challenge. Research & Practice in Assessment, 9(Winter), 17–28.
Matcha, W., Gašević, D., Uzir, N. A., Jovanović, J., & Pardo, A. (2019). Analytics of learning strategies: Associations with academic performance and feedback. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, AZ, USA (pp. 461–470). New York: ACM. https://doi.org/10.1145/3303772.3303787
Matcha, W., Uzir, N. A., Gašević, D., & Pardo, A. (2020). A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective. IEEE Transactions on Learning Technologies, 13(2), 226–245. https://doi.org/10.1109/TLT.2019.2916802
Norris, C., & Lyon, D. (2003). Surveillance as Social Sorting: Privacy, Risk, and Digital Discrimination. Routledge.
Oster, M., Lonn, S., Pistilli, M. D., & Brown, M. G. (2016). The learning analytics readiness instrument. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 173–182). New York: ACM. https://doi.org/10.1145/2883851.2883925
Pardo, A., Jovanovic, J., Dawson, S., Gašević, D., & Mirriahi, N. (2019). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology, 50(1), 128–138. https://doi.org/10.1111/bjet.12592
Pardo, A., & Siemens, G. (2014). Ethical and privacy principles for learning analytics. British Journal of Educational Technology, 45(3), 438–450. https://doi.org/10.1111/bjet.12152
Prieto, L. P., Rodríguez-Triana, M. J., Martínez-Maldonado, R., Dimitriadis, Y., & Gašević, D. (2019). Orchestrating learning analytics (OrLA): Supporting inter-stakeholder communication about adoption of learning analytics at the classroom level. Australasian Journal of Educational Technology, 35(4). https://doi.org/10.14742/ajet.4314
Prieto-Alvarez, C. G., Martinez-Maldonado, R., & Anderson, T. (2018). Co-designing learning analytics tools with learners. In J. M. Lodge, J. C. Horvath, & L. Corrin (Eds.), Learning Analytics in the Classroom: Translating Learning Analytics Research for Teachers. Abingdon-on-Thames, UK: Routledge. https://doi.org/10.4324/9781351113038
Prinsloo, P., & Slade, S. (2015). Student privacy self-management: Implications for learning analytics. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK 2015), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 83–92). New York: ACM. https://doi.org/10.1145/2723576.2723585
Prinsloo, P., & Slade, S. (2016). Student vulnerability, agency, and learning analytics: An exploration. Journal of Learning Analytics, 3(1), 159–182. https://doi.org/10.18608/jla.2016.31.10
Prinsloo, P., Slade, S., & Khalil, M. (2018). Stuck in the middle? Making sense of the impact of micro, meso and macro institutional, structural and organisational factors on implementing learning analytics. In A. Volungeviciene & A. Szucs (Eds.), Proceedings of the 27th Annual Conference of the European Distance and E-learning Network (EDEN 2018), 17–20 June 2018, Genoa, Italy (pp. 326–334). European Distance and E-learning Network. Retrieved from http://oro.open.ac.uk/55598/
Roberts, L. D., Howell, J. A., Seaman, K., & Gibson, D. C. (2016). Student attitudes toward learning analytics in higher education: “The fitbit version of the learning world”. Frontiers in Psychology, 7, 1959. https://doi.org/10.3389/fpsyg.2016.01959
Rubel, A., & Jones, K. M. (2016). Student privacy in learning analytics: An information ethics perspective. The Information Society, 32(2), 143–159. https://doi.org/10.1080/01972243.2016.1130502
Schumacher, C., & Ifenthaler, D. (2018). Features students really expect from learning analytics. Computers in Human Behavior, 78, 397–407. https://doi.org/10.1016/j.chb.2017.06.030
Sclater, N., & Bailey, P. (2015). Code of Practice for Learning Analytics. Jisc. Retrieved from https://www.jisc.ac.uk/guides/code-of-practice-for-learning-analytics
Selwyn, N. (2015). Data entry: Towards the critical study of digital data and education. Learning, Media and Technology, 40(1), 64–82. https://doi.org/10.1080/17439884.2014.921628
Selwyn, N. (2019). What’s the problem with learning analytics? Journal of Learning Analytics, 6(3), 11–19. https://doi.org/10.18608/jla.2019.63.3
Selwyn, N. (2020). Re-imagining “learning analytics”. . . A case for starting again? The Internet and Higher Education, 46, 100745. https://doi.org/10.1016/j.iheduc.2020.100745
Selwyn, N., & Gašević, D. (2020). The datafication of higher education: Discussing the promises and problems. Teaching in Higher Education, 25(4), 527–540. https://doi.org/10.1080/13562517.2019.1689388
Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American Behavioral Scientist, 57(10), 1510–1529. https://doi.org/10.1177/0002764213479366
Slade, S., Prinsloo, P., & Khalil, M. (2019). Learning analytics at the intersections of student trust, disclosure and benefit. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, AZ, USA (pp. 235–244). New York: ACM. https://doi.org/10.1145/3303772.3303796
The Open University. (2014). Policy on Ethical Use of Student Data for Learning Analytics. The Open University. Retrieved from https://help.open.ac.uk/documents/policies/ethical-use-of-student-data
Thompson, K., Alhadad, S. S., Buckingham Shum, S., Howard, S., Knight, S., Martinez-Maldonado, R., & Pardo, A. (2018). Connecting expert knowledge in the design of classroom learning experiences. In J. M. Lodge, J. C. Horvath, & L. Corrin (Eds.), From Data and Analytics to the Classroom: Translating Learning Analytics for Teachers. Taylor & Francis. Retrieved from http://hdl.handle.net/10453/127709
Tsai, Y.-S., & Gašević, D. (2017). Learning analytics in higher education—Challenges and policies: A review of eight learning analytics policies. In Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (LAK 2017), 13–17 March 2017, Vancouver, BC, Canada (pp. 233–242). New York: ACM. https://doi.org/10.1145/3027385.3027400
Tsai, Y.-S., Moreno-Marcos, P. M., Jivet, I., Scheffel, M., Tammets, K., Kollom, K., & Gasevi ˇ c, D. (2018). The SHEILA framework: Informing institutional strategies and policy processes of learning analytics. Journal of Learning Analytics, 5(3), 5–20. https://doi.org/10.18608/jla.2018.53.2
Tsai, Y.-S., Perrotta, C., & Gašević, D. (2019). Empowering learners with personalised learning approaches? Agency, equity and transparency in the context of learning analytics. Assessment & Evaluation in Higher Education, 1–14. https://doi.org/10.1080/02602938.2019.1676396
Tsai, Y.-S., Poquet, O., Dawson, S., Pardo, A., & Gašević, D. (2019). Complexity leadership in learning analytics: Drivers, ´challenges, and opportunities. British Journal of Educational Technology, 50(6), 2839–2854. https://doi.org/10.1111/bjet.12846
Tsai, Y.-S., Rates, D., Moreno-Marcos, P. M., Munoz-Merino, P. J., Jivet, I., Scheffel, M., . . . Gašević, D. (2020). Learning analytics in European higher education–Trends and barriers. Computers & Education, 103933. https://doi.org/10.1016/j.compedu.2020.103933
Tsai, Y.-S., Whitelock-Wainwright, A., & Gašević, D. (2020). The privacy paradox and its implications for learning analytics. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 230–239). New York: ACM. https://doi.org/10.1145/3375462.3375536
Uhl-Bien, M., & Arena, M. (2018). Leadership for organizational adaptability: A theoretical synthesis and integrative framework. The Leadership Quarterly, 29(1), 89–104. https://doi.org/10.1016/j.leaqua.2017.12.009
Viberg, O., Hatakka, M., Balter, O., & Mavroudi, A. (2018). The current landscape of learning analytics in higher education. Computers in Human Behavior, 89, 98–110. https://doi.org/10.1016/j.chb.2018.07.027
Vigentini, L., Liu, D. Y., Arthars, N., & Dollinger, M. (2020). Evaluating the scaling of a LA tool through the lens of the SHEILA framework: A comparison of two cases from tinkerers to institutional adoption. The Internet and Higher Education, 45, 100728. https://doi.org/10.1016/j.iheduc.2020.100728
Whitelock-Wainwright, A., Gašević, D., Tejeiro, R., Tsai, Y.-S., & Bennett, K. (2019). The student expectations of learning analytics questionnaire. Journal of Computer Assisted Learning, 35(5), 633–666. https://doi.org/10.1111/jcal.12366
Williamson, B. (2018). The hidden architecture of higher education: Building a big data infrastructure for the “smarter university”. International Journal of Educational Technology in Higher Education, 15(1), 12. https://doi.org/10.1186/s41239-018-0094-1
Williamson, B., Bayne, S., & Shay, S. (2020). The datafication of teaching in higher education: Critical issues and perspectives. Teaching in Higher Education, 25(4), 351–365. https://doi.org/10.1080/13562517.2020.1748811
Wintrup, J. (2017). Higher education’s panopticon? Learning analytics, ethics and student engagement. Higher Education Policy, 30(1), 87–103. https://doi.org/10.1057/s41307-016-0030-8
Zuboff, S. (2015). Big other: Surveillance capitalism and the prospects of an information civilization. Journal of Information Technology, 30(1), 75–89. https://doi.org/10.1057/jit.2015.5
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
