A LAK of Direction
Misalignment Between the Goals of Learning Analytics and its Research Scholarship
DOI:
https://doi.org/10.18608/jla.2023.7913Keywords:
learning analytics, systematic review, commentary paperAbstract
Learning analytics defines itself with a focus on data from learners and learning environments, with corresponding goals of understanding and optimizing student learning. In this regard, learning analytics research, ideally, should be characterized by studies that make use of data from learners engaged in education systems, should measure student learning, and should make efforts to intervene and improve these learning environments. However, a common concern among members of the learning analytics research community is that these standards are not being met. In two analysis waves, we review a large and comprehensive sample of research articles from the proceedings of the three most recent Learning Analytics and Knowledge conferences, the premier conference venue for learning analytics research, and from articles published during the same time in the Journal of Learning Analytics (over the years of 2020, 2021, and 2022). We find that 37.4% of articles do not analyze data from learners in an education system, 71.1% do not include any measure of learning, and 89.0% of articles do not attempt to intervene in the learning environment. We contrast these findings with the stated definition of learning analytics and infer, like others before us, that scholarship in learning analytics research presently lacks clear direction toward its stated goals. We invite critical discussion of these findings from the learning analytics community, through open peer commentary.
References
Ahn, J., Nguyen, H., Campos, F., & Young, W. (2021). Transforming everyday information into practical analytics with crowdsourced assessment tasks. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 66–76). ACM Press. https://doi.org/10.1145/3448139.3448146
Baek, C., & Doleck, T. (2021). Educational data mining versus learning analytics: A review of publications from 2015 to 2019. Interactive Learning Environments, 1–23. https://doi.org/10.1080/10494820.2021.1943689
Baker, R. S., & Inventado, P. S. (2014). Educational data mining and learning analytics. In J. A. Larusson & B. White (Eds.), Learning analytics (pp. 61–75). Springer. https://doi.org/10.1007/978-1-4614-3305-7_4
Bergner, Y., Gray, G., & Lang, C. (2018). What does methodology mean for learning analytics? Journal of Learning Analytics, 5(2), 1–8. https://doi.org/10.18608/jla.2018.52.1
Bruyat, C., & Julien, P.-A. (2001). Defining the field of research in entrepreneurship. Journal of Business Venturing, 16(2), 165–180. https://doi.org/10.1016/S0883-9026(99)00043-9
Canning, E. A., Priniski, S. J., & Harackiewicz, J. M. (2019). Unintended consequences of framing a utility-value intervention in two-year colleges. Learning and Instruction, 62, 37–48. https://doi.org/10.1016/j.learninstruc.2019.05.001
Carpenter, D., Cloude, E., Rowe, J., Azevedo, R., & Lester, J. (2021). Investigating student reflection during game-based learning in middle grades science. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 280–291). ACM Press. https://doi.org/10.1145/3448139.3448166
Cerratto Pargman, T., & McGrath, C. (2021). Mapping the ethics of learning analytics in higher education: A systematic literature review of empirical research. Journal of Learning Analytics, 8(2), 123–139. https://doi.org/10.18608/jla.2021.1
Chen, B., Chen, X., & Xing, W. (2015). “Twitter archeology” of learning analytics and knowledge conferences. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 340–349). ACM Press. https://doi.org/10.1145/2723576.2723584
Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445–459. https://doi.org/10.3102/00346543053004445
Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ʼ12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 134–138). ACM Press. https://doi.org/10.1145/2330601.2330636
Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, 18(6), 683–695. https://doi.org/10.1080/13562517.2013.827653
Dawson, S., Joksimović, S., Poquet, O., & Siemens, G. (2019). Increasing the impact of learning analytics. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (pp. 446–455). ACM Press. https://doi.org/10.1145/3303772.3303784
Dowell, N., Joksimović, S., Scheffel, M., & Siemens, G. (2021). Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA. ACM Press. https://doi.org/10.1145/3448139
Emara, M., Hutchins, N., Grover, S., Snyder, C., & Biswas, G. (2021). Examining student regulation of collaborative, computational, problem-solving processes in open-ended learning environments. Journal of Learning Analytics, 8(1), 49–74. http://doi.org/10.18608/jla.2021.7230
Ferguson, R., & Clow, D. (2017). Where is the evidence? A call to action for learning analytics. Proceedings of the 7th International Conference on Learning Analytics and Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 56–65). ACM Press. https://doi.org/10.1145/3027385.3027396
Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., Rienties, B., Ullmann, T., & Vuorikari, R. (2016). Research evidence on the use of learning analytics: Implications for education policy. In R. Vuorikari, J. Castaño Muñoz (Eds.), Joint Research Centre Science for Policy Report (EUR 28294 EN). https://doi.org/10.2791/955210
Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59, 64–71. https://doi.org/10.1007/s11528-014-0822-x
Gray, G., & Bergner, Y. (2022). A practitioner’s guide to measurement in learning analytics: Decisions, opportunities, and challenges. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), Handbook of learning analytics, 2nd. ed. (pp. 20–28). Society for Learning Analytics Research. https://doi.org/10.18608/hla22.002
Ionita, R. F., Corlatescu, D. G., Dascalu, M., & McNamara, D. S. (2021). How has the LAK community evolved in its first decade? A detailed modeling through interactive CNA sociograms. Proceedings of the 17th International Scientific Conference on eLearning and Software for Education (eLSE 2021), 22–23 April 2016, Bucharest, Romania (pp. 52–68).
Joksimović, S., Siemens, G., Wang, Y. E., San Pedro, M. O. Z., & Way, J. (2020). Editorial: Beyond cognitive ability. Journal of Learning Analytics, 7(1), 1–4. https://doi.org/10.18608/jla.2020.71.1
Jørnø, R. L., & Gynther, K. (2018). What constitutes an “actionable insight” in learning analytics? Journal of Learning Analytics, 5(3), 198–221. https://doi.org/10.18608/jla.2018.53.13
Kitto, K., Whitmer, J., Silvers, A. E., & Webb, M. (2020). Creating data for learning analytics ecosystems. Society for Learning Analytics Research (SoLAR). https://www.solaresearch.org/core/creating-data-for-learning-analytics-ecosystems/
Klebanov, B. B., Loukina, A., Lockwood, J., Liceralde, V. R. T., Sabatini, J., Madnani, N., Gyawali, B., Wang, Z., & Lentini, J. (2020). Detecting learning in noisy data: The case of oral reading fluency. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 490–495). ACM Press. https://doi.org/10.1145/3375462.3375490
Knight, S., Buckingham Shum, S., & Littleton, K. (2014). Epistemology, assessment, pedagogy: Where learning meets analytics in the middle space. Journal of Learning Analytics, 1(2), 23–47. https://doi.org/10.18608/jla.2014.12.3
Kornell, N., & Bjork, R. A. (2007). The promise and perils of self-regulated study. Psychonomic Bulletin & Review, 14(2), 219–224. https://doi.org/10.3758/BF03194055
Kovanović, V., Scheffel, M., Pinkwart, N., & Verbert, K. (2020). Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany. ACM Press. https://doi.org/10.1145/3375462
Larrabee Sønderlund, A., Hughes, E., & Smith, J. (2019). The efficacy of learning analytics interventions in higher education: A systematic review. British Journal of Educational Technology, 50(5), 2594–2618. https://doi.org/10.1111/bjet.12720
Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education — A literature review. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends. Studies in Systems, Decision and Control, vol. 94. Springer. https://doi.org/10.1007/978-3-319-52977-6_1
Liu, L. T., Wang, S., Britton, T., & Abebe, R. (2022). Lost in translation: Reimagining the machine learning life cycle in education. https://doi.org/10.48550/arXiv.2209.03929
Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30–40.
Ma, Y., Celepkolu, M., & Boyer, K. E. (2022). Detecting impasse during collaborative problem solving with multimodal learning analytics. Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK ’22), 21–25 March 2022, Online (pp. 45–55). ACM Press. https://doi.org/10.1145/3506860.3506865
Motz, B. A., Carvalho, P. F., de Leeuw, J. R., & Goldstone, R. L. (2018). Embedding experiments: Staking causal inference in authentic educational contexts. Journal of Learning Analytics, 5(2), 47–59. https://doi.org/10.18608/jla.2018.52.4
Motz, B., Brooks, C., Quick, J., Bergner, Y., Gray, G., Lang, C., Li, W., & Marmolejo-Ramos, F. (2022). A baseline measure of open research practices in learning analytics. Pre-conference Workshop of the 12th International Conference on Learning Analytics & Knowledge. https://doi.org/10.35542/osf.io/325d7
Newell, A. (1973). You can’t play 20 questions with nature and win: Projective comments on the papers of this symposium. In W. G. Chase (Ed.), Visual information processing (pp. 283–308). Academic Press. https://doi.org/10.1016/B978-0-12-170150-5.50012-3
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259. https://doi.org/10.1037/0033-295X.84.3.231
Núñez, R., Allen, M., Gao, R., Miller Rigoli, C., Relaford-Doyle, J., & Semenuks, A. (2019). What happened to cognitive science? Nature Human Behaviour, 3, 782–791. https://doi.org/10.1038/s41562-019-0626-2
Ochoa, X., & Merceron, A. (2018). Quantitative and qualitative analysis of the learning analytics and knowledge conference 2018. Journal of Learning Analytics, 5(3), 154–166. https://doi.org/10.18608/jla.2018.53.10
Papamitsiou, Z., & Economides, A. A. (2014). Learning analytics and educational data mining in practice: A systematic literature review of empirical evidence. Educational Technology & Society, 17(4), 49–64. https://www.jstor.org/stable/jeductechsoci.17.4.49
Papamitsiou, Z., Giannakos, M. N., & Ochoa, X. (2020). From childhood to maturity: Are we there yet? Mapping the intellectual progress in learning analytics during the past decade. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 559–568). ACM Press. https://doi.org/10.1145/3375462.3375519
Pardo, A., Poquet, O., Martínez-Maldonado, R., & Dawson, S. (2017). Provision of data-driven student feedback in LA & EDM. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), Handbook of learning analytics (pp. 163–174). Society for Learning Analytics Research. https://doi.org/10.18608/hla17.014
Petrosino, A., Turpin‐Petrosino, C., Hollis‐Peel, M. E., & Lavenberg, J. G. (2013). Scared Straight and other juvenile awareness programs for preventing juvenile delinquency: A systematic review. Campbell Systematic Reviews, 9(1), 1–55. https://doi.org/10.4073/csr.2013.5
Pressley, M., Goodchild, F., Fleet, J., Zajchowski, R., & Evans, E. D. (1989). The challenges of classroom strategy instruction. The Elementary School Journal, 89(3), 301–342. http://doi.org/10.1086/461578
Reeves, T. C., & Lin, L. (2020). The research we have is not the research we need. Educational Technology Research and Development, 68, 1991–2001. https://doi.org/10.1007/s11423-020-09811-3
Reinhart, A. L., Haring, S. H., Levin, J. R., Patall, E. A., & Robinson, D. H. (2013). Models of not-so-good behavior: Yet another way to squeeze causality and recommendations for practice out of correlational data. Journal of Educational Psychology, 105(1), 241–247. https://doi.org/10.1037/a0030368
Romero, C., & Ventura, S. (2020). Educational data mining and learning analytics: An updated survey. WIREs Data Mining and Knowledge Discovery, 10(3), e1355. https://doi.org/10.1002/widm.1355
Shibani, A., Knight, S., & Buckingham Shum, S. (2022). Questioning learning analytics? Cultivating critical engagement as student automated feedback literacy. Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK ’22), 21–25 March 2022, Online (pp. 326–335). ACM Press. https://doi.org/10.1145/3506860.3506912
Si Na, K., & Tasir, Z. (2017). A systematic review of learning analytics intervention contributing to student success in online learning. Proceedings of the 2017 International Conference on Learning and Teaching in Computing and Engineering (LaTICE ’17), 20–23 April 2017, Hong Kong, China (pp. 62–68). ACM Press. https://doi.org/10.1109/LaTICE.2017.18
Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. https://doi.org/10.1177/0002764213498851
Siemens, G. (2020, July 15). Learning engineering, learning analytics, educational data mining. [Online discussion]. Learning Analytics Google Group. https://bookdown.org/cl3584/book/Learning%20engineering,%20learning%20analytics,%20educational%20data%20mining.pdf
Society for Learning Analytics Research. (2021). What is learning analytics? https://www.solaresearch.org/about/what-is-learning-analytics/
Viberg, O., Hatakka, M., Bälter, O., & Mavroudi, A. (2018). The current landscape of learning analytics in higher education. Computers in Human Behavior, 89, 98–110. https://doi.org/10.1016/j.chb.2018.07.027
Viberg, O., Khalil, M., & Baars, M. (2020). Self-regulated learning and learning analytics in online learning environments: A review of empirical research. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 524–533). ACM Press. https://doi.org/10.1145/3375462.3375483
Werch, C. E., & Owen, D. (2002). Iatrogenic effects of alcohol and drug prevention programs. Journal of Studies on Alcohol, 63(5), 581–590. https://doi.org/10.15288/jsa.2002.63.581
Wise, A. F., & Shaffer, D. W. (2015). Why theory matters more than ever in the age of big data. Journal of Learning Analytics, 2(2), 5–13. https://doi.org/10.18608/jla.2015.22.2
Wise, A. F., Knight, S., & Ochoa, X. (2021). What makes learning analytics research matter. Journal of Learning Analytics, 8(3), 1–9. https://doi.org/10.18608/jla.2021.7647
Wise, A. F., Martinez-Maldonado, R., & Hilliger, I. (2022). Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK ’22), 21–25 March 2022, Online. ACM Press. https://doi.org/10.1145/3506860
Wong, B. T.-M., & Li, K. C. (2020). A review of learning analytics intervention in higher education (2011–2018). Journal of Computers in Education, 7, 7–28. https://doi.org/10.1007/s40692-019-00143-7
Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. https://doi.org/10.1016/j.learninstruc.2012.03.004
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
