Moving Forward with Learning Analytics: Expert Views
DOI:
https://doi.org/10.18608/jla.2019.63.8Keywords:
Affect, Complexity, Ethics, Implementation, Learning analytics, Pedagogy, Policy Delphi, Power, Regulation, ValidityAbstract
Learning analytics involve the measurement, collection, analysis, and reporting of data about learners and their contexts, in order to understand and optimise learning and the environments in which it occurs. Since emerging as a distinct field in 2011, learning analytics has grown rapidly, and early adopters around the world are already developing and deploying these new tools. This paper reports on a study that investigated how the field is likely to develop by 2025, in order to make recommendations for action to those concerned with the implementation of learning analytics. The study used a Policy Delphi approach, presenting a range of future scenarios to international experts in the field and asking for responses related to the desirability and feasibility of these scenarios, as well as actions that would be required. Responses were received from 103 people from 21 countries. Responses were coded thematically, inter-rater reliability was checked using Cohen’s kappa coefficient, and data were recoded if kappa was below 0.6. The seven major themes that were identified within the data were power, pedagogy, validity, regulation, complexity, ethics, and affect. The paper considers in detail each of these themes and its implications for the implementation of learning analytics.
References
Adams Becker, S., Freeman, A., Giesinger Hall, C., Cummins, M., & Yuhnke, B. (2016). NMC/CoSN Horizon Report: 2016 K-12 Edition. Austin, TX: NMC. https://www.nmc.org/publication/nmc-cosn-horizon-report-2016-k-12-edition/.
ADL. (2018). An organized list of ADL’s resources available on github. http://adlnet.github.io/.
ADL. (2017) xAPI-Spec. https://github.com/adlnet/xAPI-Spec.
Apereo. (2016). Apereo Open Analytics at a Glance. https://www.apereo.org/communities/learning-analytics-initiative/apereo-open-analytics-glance.
Arnold, K. E., & Pistilli, M. (2012). Course Signals at Purdue: Using learning analytics to increase student success. Paper presented at LAK ’12 (30 April–2 May), Vancouver, Canada. New York, USA: ACM. http://dx.doi.org/10.1145/2330601.2330666
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77–101. http://dx.doi.org/10.1191/1478088706qp063oa
Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-centred learning analytics. Journal of Learning Analytics, 6(2), 1–9. https://doi.org/10.18608/jla.2019.62.1
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: a new tool for a new era. Educause Review, 42(4), 40–57. https://er.educause.edu/articles/2007/7/academic-analytics-a-new-tool-for-a-new-era
Clow, D., Cross, S., Ferguson, R. and Rienties, B. (2014). Evidence Hub Review. Milton Keynes: LACE Project. http://www.laceproject.eu/deliverables/d2-5-evidence-hub/
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, XX(1), 37–46. https://doi.org/10.1177/001316446002000104
Cormack, A. (2016). A data protection framework for learning analytics. Journal of Learning Analytics, 3(1), 91–106. https://doi.org/10.18608/jla.2016.31.6
Dalkey, N. C. (1969). The Delphi Method: An Experimental Study of Group Opinion. Santa Monica, CA: Rand Corporation. https://www.rand.org/pubs/research_memoranda/RM5888.html.
Dawson, S., Gašević, D., Siemens, G., & Joksimovic, S. (2014). Current state and future trends: A citation network analysis of the learning analytics field. Paper presented at LAK ’14 (24–28 March), Indianapolis, IN. New York, USA: ACM. http://dx.doi.org/2567574.2567585
Drachsler, H., & Greller, W. (2012). The pulse of learning analytics: Understandings and expectations from stakeholders. Paper presented at LAK ’12 (29 April–2 May, 2012), Vancouver, Canada. New York, USA: ACM. http://dx.doi.org/10.1145/2330601.2330634
Drachsler, H., & Greller, W. (2016). Privacy and analytics: It’s a DELICATE issue. A checklist for trusted learning analytics. Paper presented at LAK ’16 (25–29 April, 2016), Edinburgh, UK. New York, USA: ACM. http://dx.doi.org/10.1145/2883851.2883893
Dringus, L. P. (2012). Learning analytics considered harmful. Journal of Asynchronous Learning Networks, 16(3), 87–100.
Engelfriet, A., Manderveld, J., & Jeunink, E. (2017). Learning Analytics Under the Dutch Data Protection Act. SURFnet. https://www.surf.nl/handreiking-learning-analytics-onder-de-wet-bescherming-persoonsgegevens
Essa, A. (2019) Is data dark? Lessons from Borges’s ‘Funes the Memorius’. Journal of Learning Analytics, 6(3), 35–42. http://dx.doi.org/10.18608/jla.2019.63.7
European Parliament and Council. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data. http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32016R0679.
Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning (IJTEL), 4(5/6), 304–317.
Ferguson, R. (2019). Ethical challenges for learning analytics. Journal of Learning Analytics, 6(3), 25–30. http://dx.doi.org/10.18608/jla.2019.63.5
Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., Rienties, B., Ullmann, T., & Vuorikari, R. (2016). Research evidence on the use of learning analytics: Implications for education policy — Study. Joint Research Centre Science for Policy Report. Seville, Spain: EUR 28294. http://dx.doi.org/10.2791/955210
Ferguson, R., Cooper, A., Drachsler, H., Kismihok, G., Boyer, A., Tammets, K., & Martinez Mones, A. (2015). Learning analytics: European perspectives. Paper presented at LAK ’15 (16–20 March, 2015), Poughkeepsie, NY, USA. New York, USA: ACM. http://dx.doi.org/10.1145/2723576.2723637
Ferguson, R., Hoel, T., Scheffel, M., & Drachsler, H. (2016). Guest editorial: Ethics and privacy in learning analytics. Journal of Learning Analytics, 3(1), 5–15. https://doi.org/10.18608/jla.2016.31.2
Ferguson, R., Macfadyen, L. P., Clow, D., Tynan, B., Alexander, S., & Dawson, S. (2015). Setting learning analytics in context: Overcoming the barriers to large-scale adoption. Journal of Learning Analytics, 1(3), 120–144. https://doi.org/10.18608/jla.2014.13.7
Goodyear, P., Banks, S., Hodgson, V., & McConnell, D. (Eds.). (2004). Advances in Research on Networked Learning. Norwell, Massachusetts, USA: Kluwer Academic Publishers. http://dx.doi.org/10.1007/1-4020-7909-5
Griffiths, D., Brasher, A., Clow, D., Ferguson, R., & Yuan, L. (2016). Visions of the future: Horizon report. http://www.laceproject.eu/deliverables/d3-2-visions-of-the-future-2/
Hernández Leo, D., Rodríguez-Triana, M., Inventado, P., Mor, Y. (Eds.). 2017. Special issue of IxD&A Connecting Learning Design and Learning Analytics. http://ixdea.uniroma2.it/inevent/events/idea2010/index.php?s=10&a=10&link=ToC_33_P
Hoel, T., & Chen, W. (2016). Privacy-driven design of learning analytics applications: exploring the design space of solutions for data sharing and interoperability. Journal of Learning Analytics, 3(1), 139–158.
Hoel, T., Griffiths, D., & Chen, W. (2017). The influence of data protection and privacy frameworks on the design of learning analytics systems. Paper presented at LAK ’17, Vancouver, British Columbia, Canada. New York, USA: ACM. https://doi.org/10.1145/3027385.3027414
IMS Global Learning Consortium. (2018). Caliper Analytics Specification, version 1.1. https://www.imsglobal.org/sites/default/files/caliper/v1p1/caliper-spec-v1p1/caliper-spec-v1p1.html
Jisc. (n.d.). Learning Analytics. Harness the Power of Your Data. https://www.jisc.ac.uk/learning-analytics
Jiscdev. (n.d.). Jisc Learning Analytics Statement Templates. https://github.com/jiscdev/xapi
Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Hall, C. (2016). NMC Horizon Report: 2016 Higher Education Edition. Austin, Texas, USA: NMC. https://www.nmc.org/publication/nmc-horizon-report-2016-higher-education-edition/.
Knight, S., Buckingham Shum, S., & Littleton, K. (2014). Epistemology, assessment, pedagogy: Where learning meets analytics in the middle space. Journal of Learning Analytics, 1(2), 23–47. https://doi.org/10.18608/jla.2014.12.3
Kurzweil, R. (2005). The Singularity Is Near. London, UK: Duckworth.
LACE. (n.d.). Ethics and Privacy in Learning Analytics. http://www.laceproject.eu/ethics-privacy-learning-analytics/
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. http://dx.doi.org/10.2307/2529310
Linstone, H. A., & Turoff, M. (2002). The Delphi Method: Techniques and Applications. Boston, Massachusetts, USA: Addison-Wesley.
Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, 46(5), 31–40. https://er.educause.edu/articles/2011/9/penetrating-the-fog-analytics-in-learning-and-education
Macfadyen, L., & Dawson, S. (2012). Numbers are not enough. Why e-learning analytics failed to inform an institutional strategic plan. Educational Technology & Society, 15(3), 149–163. https://www.jstor.org/stable/jeductechsoci.15.3.149
Macfadyen, L., Dawson, S., Pardo, A., & Gašević, D. (2014). Embracing Big Data in complex educational systems: The learning analytics imperative and the policy challenge. Research & Practice in Assessment, 9(Winter), 17–28. https://eric.ed.gov/?id=EJ1062692
McIntosh, N. E. (1979). Barriers to implementing research in higher education. Studies in Higher Education, 4(1), 77–86. http://dx.doi.org/10.1080/03075077912331377121
National Academy of Education. (2017). Big Data in Educaton, Balancing the Benefits of Educational Research and Student Privacy. http://naeducation.org/wp-content/uploads/2017/05/NAEd_BD_Booklet_FINAL_051717_3.pdf
Orwell, G. (1949). Nineteen Eighty-Four. London: Secker & Warburg.
Perry, M. R., et al. (2018). SPEC Kit 360. https://publications.arl.org/Learning-Analytics-SPEC-Kit-360/.
Prinsloo, P. (2019). Commentary on Neil Selwyn’s LAK18 keynote address. Journal of Learning Analytics, 6(3), 20–24. http://dx.doi.org/10.18608/jla.2019.63.4
Prinsloo, P., & Slade, S. (2013). An evaluation of policy frameworks for addressing ethical considerations in learning analytics. Paper presented at LAK ’13 (8–12 April 2013), Leuven, Belgium. New York, USA: ACM. http://dx.doi.org/10.1145/2460296.2460344
Prinsloo, P., & Slade, S. (2017). Ethics and learning analytics: Charting the (un)charted. In Lang, C. et al. (Eds.), Handbook of Learning Analytics (pp. 49–57). SoLAR. https://doi.org/10.18608/hla17
Rosé, C. P. (2019). Monolith, multiplicity, or multivocality: What do we stand for and where do we go from here? Journal of Learning Analytics, 6(3), 31–34. http://dx.doi.org/10.18608/jla.2019.63.6
Sclater, N., & Mullan, J. (2017). Learning Analytics and Student Success — Assessing the Evidence. Bristol, UK: Jisc. https://repository.jisc.ac.uk/6560/1/learning-analytics_and_student_success.pdf.
Selwyn, N. (2019). What’s the problem with learning analytics? Journal of Learning Analytics, 6(3), 11–19. http://dx.doi.org/10.18608/jla.2019.63.3
Sharples, M. (2000). The design of personal mobile technologies for lifelong learning. Computers & Education, 34(3-4), 177–193. https://dx.doi.org/10.1016/S0360-1315(99)00044-5
Sharples, M., de Roock, R., Ferguson, R., Gaved, M., Herodotou, C., Koh, E., Kukulska-Hulme, A., Looi, C-K., McAndrew, P., Rienties, B., Weller, M. & Wong, L. H. (2016). Innovating Pedagogy 2016: Open University Innovation Report 5. Milton Keynes, UK: The Open University. https://iet.open.ac.uk/file/innovating_pedagogy_2016.pdf.
Siemens, G. (2005). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning, 2(1).
Siemens, G., Gašević, D., Haythornthwaite, C., Dawson, S., Buckingham Shum, S., Ferguson, R., Duval, E., Verbert, K., & Baker, R. S. J. d. (2011). Open Learning Analytics: An Integrated and Modularized Platform (Concept Paper). https://solaresearch.org/core/open-learning-analytics-an-integrated-modularized-platform/
Slade, S., & Prinsloo, P. (2015). Student perspectives on the use of their data: Between intrusion, surveillance and care. European Journal of Distance and E-learning, 18(1). http://www.eurodl.org/index.php?p=special&sp=articles&inum=6&abstract=672&article=679
Tsai, Y. S., Moreno-Marcos, P. M., Tammets, K., Kollom, K., & Gašević, D. (2018, March). SHEILA policy framework: Informing institutional strategies and policy processes of learning analytics. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 320–329). ACM. http://dx.doi.org/10.1145/3170358.3170367
Turoff, M. (1970). The Design of a Policy Delphi, reprinted in The Science of Public Policy, vol. IV, Policy Analysis II (2002), T. Miyakawa (Ed.), pp. 488–517. https://dx.doi.org/10.1016/0040-1625(70)90161-7
Turoff, M. (2002). The Policy Delphi. In H. A. Linstone & M. Turoff (Eds.), The Delphi Method: Techniques and Applications (pp. 80–96). Boston, Massachusetts, USA: Addison-Wesley.
Downloads
Published
How to Cite
Issue
Section
License
TEST
