MMALA: Developing and Evaluating a Maturity Model for Adopting Learning Analytics
DOI:
https://doi.org/10.18608/jla.2024.8099Keywords:
maturity model, processes, policy development, learning analytics, higher education, research paperAbstract
Learning analytics (LA) adoption is a challenging task for higher education institutions (HEIs) since it involves different aspects of the academic environment, such as information technology infrastructure, human resource management, ethics, and pedagogical issues. Therefore, it is necessary to provide institutions with supporting instruments to deal with these challenges. Although there has been much research on factors that are associated with the adoption of LA in HEIs, there has been much less research on specific models that can be used to guide actual adoption. In this sense, we developed MMALA, a Maturity Model for Adopting Learning Analytics. It is a guide that describes the necessary practices for taking the first steps in this area and enables institutions to reach higher levels of maturity in LA use, culminating in an organized and systematic adoption. In this paper, we describe the development process of MMALA, focusing on the model evaluation, which used both the questionnaire and the expert opinion method. MMALA can also give institutions an overview of their current situation regarding LA adoption. In this sense, we present the results of the maturity evaluation of three Brazilian HEIs using MMALA.
References
ABNT NBR ISO/IEC 38500. (2009). Brazilian association of technical standards / associação brasileira de normas técnicas. governança corporativa de tecnologia da informação. Rio de Janeiro, Brazil.
Al-Sai, Z. A., Abdullah, R., & Husin, M. H. (2019). A review on big data maturity models. In IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT) (p. 156-161). doi: https://doi.org/10.1109/JEEIT.2019.8717398
Alzahrani, A. S., Tsai, Y.-S., Aljohani, N., Whitelock-Wainwright, E., & Gaˇsevi´c, D. (2023). Do teaching staff trust stakeholders and tools in learning analytics? A mixed methods study. Educational Technology Research and Development, 71(4), 1471–1501. https://doi.org/10.1007/s11423-023-10229-w
Alzahrani, A. S., Tsai, Y.-S., Iqbal, S., Marcos, P. M. M., Scheffel, M., Drachsler, H., Kloos, C. D., Aljohani, N., & Gašević, D. (2023). Untangling connections between challenges in the adoption of learning analytics in higher education. Education and Information Technologies, 28(4), 4563–4595. https://doi.org/10.1007/s10639-022-11323-x
Aničić, K. P., Begičević Redep, N., & Kadoić, N. (2022). Towards LA adoption maturity framework. In 2022 45th Jubilee International Convention on Information, Communication and Electronic Technology (MIPRO) (p. 711-716). doi: https://doi.org/10.23919/MIPRO55190.2022.9803750
Armstrong, J. (1985). Long-range forecasting from crystal ball to computer. John Wiley.
Arroway, P., Morgan, G., O’Keefe, M., & Yanosky, R. (2016). Learning Analytics in higher education. ECAR. Retrieved from https://library.educause.edu/˜/media/files/library/2016/2/ers1504la
Ayyub, B. M. (2000). Methods for expert-opinion elicitation of probabilities and consequences for corps facilities. IWR Report-00-R-10. Retrieved from https://www.iwr.usace.army.mil/Portals/70/docs/iwrreports/00-R-101.pdf
Becker, J., Knackstedt, R., & Pöppelbuß, J. (2009). Developing maturity models for IT management – a procedure model and its application. Business Information Systems Engineering, 1(3), 213-222. doi: https://doi.org/10.1007/s12599-009-0044-5
Beecham, S., Hall, T., Britton, C., Cottee, M., & Rainer, A. (2005). Using an expert panel to validate a requirements process improvement model. Journal of Systems and Software, 76(3), 251-275. doi: https://doi.org/10.1016/j.jss.2004.06.004
Broos, T., Hilliger, I., Pérez-Sanagustín, M., Htun, N.-N., Millecamp, M., Pesántez-Cabrera, P., Solano-Quinde, L., Siguenza-Guzman, L., Zuñiga-Prieto, M., Verbert, K., & De Laet, T. (2020). Coordinating learning analytics policymaking and implementation at scale. British Journal of Educational Technology, 51(4), 938-954. doi: https://doi.org/10.1111/bjet.12934
Caralli, R., Knight, M., & Montgomery, A. (2012). Maturity models 101: A primer for applying maturity models to smart grid security, resilience, and interoperability. Carnegie Mellon University. Retrieved from https://resources.sei.cmu.edu/assetfiles/WhitePaper/201201900158920.pdf.
Charleer, S., Klerkx, J., Duval, E., De Laet, T., & Verbert, K. (2016). Creating effective learning analytics dashboards: Lessons learnt. In K. Verbert, M. Sharples, & T. Klobucar (Eds.), Adaptive and adaptable learning (pp. 42–56). Cham: Springer International Publishing. doi: https://doi.org/10.1007/978-3-319-45153-4_4
Colvin, C., Dawson, S., Wade, A., & Gašević, D. (2017). Addressing the challenges of institutional adoption. In C. Lang, G. Siemens, A. Wise, D. Gašević (Eds.), Handbook of learning analytics, 281-289. doi: https://doi.org/10.18608/hla17
Dawson, S., Joksimovic, S., Poquet, O., & Siemens, G. (2019). Increasing the impact of learning analytics. In Proceedings of the 9th International Conference on Learning Analytics and Knowledge (p. 446–455). Arizona, USA: Association for Computing Machinery. doi: https://doi.org/10.1145/3303772.3303784
Dawson, S., Poquet, O., Colvin, C., Rogers, T., Pardo, A., & Gasevic, D. (2018). Rethinking learning analytics adoption through complexity leadership theory. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (p. 236–244). Sydney, Australia: Association for Computing Machinery. doi: https://doi.org/10.1145/3170358.3170375
del Blanco, A., Ángel Serrano, Freire, M., Martínez-Ortiz, I., & Fernández-Manjón, B. (2013). E-Learning standards and learning analytics. Can data collection be improved by using standard data models? In IEEE Global Engineering Education Conference (EDUCON) (p. 1255-1261). doi: https://doi.org/10.1109/EduCon.2013.6530268
DMM. (2014). Data management maturity model – 1.0 version (1st ed.). CMMI Institute.
Dyba, T. (2000). An instrument for measuring the key factors of success in software process improvement. Empirical Software Engineering, 5, 357–390. doi: https://doi.org/10.1023/A:1009800404137
Ferguson, R. (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5), 304–317. doi: https://doi.org/10.1504/IJTEL.2012.051816
Freitas, E., Falcão, T. P. d. R., & Ferreira Mello, R. (2020). Desmistificando a adoção de Learning Analytics: um guia conciso sobre ferramentas e instrumentos. In Jornada de Atualização em Informática na Educação. https://doi.org/10.5753/sbc.5627.6.4
Freitas, E., Fonseca, F., Garcia, V., Falcão, T. P. d. R., Marques, E., & Ferreira Mello, R. (2020). Evaluación de un modelo de madurez para la adopción de learning analytics en instituciones de educación superior. Revista Latinoamericana de Tecnología Educativa—RELATEC, 19(2), 101–113. https://doi.org/10.17398/1695-288X.19.2.101
Freitas, E., Fonseca, F., Garcia, V., Ferreira Mello, R., & Gašević, D. (2020). Towards a maturity model for learning analytics adoption. An overview of its levels and areas. In Proceedings of the 2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT 2020), 6–9 July 2020, Tartu, Estonia (pp. 122–126). IEEE. https://doi.org/10.1109/ICALT49669.2020.00059
Garcia, S., Marques, E. C. M., Mello, R. F., Gašević, D., Rodrigues, R. L., & Falcão, T. P. (2021). Adopting learning analytics in a Brazilian higher education institution: Ideal and predicted expectations. In Proceedings of the IV Latin American Conference on Learning Analytics (p. 8-17). Retrieved from https://ceur-ws.org/Vol-3059/paper1.pdf
Garcia, V. C. (2010). Rise reference model for software reuse adoption in Brazilian companies [Ph.D. Thesis]. Federal University of Pernambuco. Retrieved from https://repositorio.ufpe.br/handle/123456789/1729
Gašević, D., Tsai, Y., Dawson, S., & Pardo, A. (2019). How do we start? an approach to learning analytics adoption in higher education. International Journal of Information and Learning Technology, 36(4), 342-353. doi: https://doi.org/10.1108/IJILT-02-2019-0024
Govindarajan, K., Kumar, V. S., Boulanger, D., & Kinshuk. (2015). Learning analytics solution for reducing learners’ course failure rate. In IEEE Seventh International Conference on Technology for Education (T4E) (p. 83-90). doi: https://doi.org/10.1109/T4E.2015.14
Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Educational Technology Society, 15(3), 42–57.
Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2009). Multivariate analysis of data. (análise multivariada de dados) (6th ed.). Bookman Editora.
Halper, F., & Stodder, D. (2014). TDWI analytics maturity model guide. TDWI Research. Retrieved from https://tdwi.org/~/media/545E06D7CE184B19B269E929B0903D0C
Hanaei, E. H. A., & Rashid, A. (2014). DF-C2M2: A Capability Maturity Model for Digital Forensics Organisations. In IEEE Security and Privacy Workshops (p. 57-60). doi: https://doi.org/10.1109/SPW.2014.17
Hausladen, I., & Schosser, M. (2020). Towards a maturity model for big data analytics in airline network planning. Journal of Air Transport Management, 82, 101721. https://doi.org/10.1016/j.jairtraman.2019.101721
Hilliger, I., Pérez-Sanagustín, M., Pérez-Álvarez, R., Henríquez, V., Guerra, J., Zuñiga-Prieto, M. Á ., Ortiz-Rojas, M., Tsai, Y.-S., ,Gašević, D., Muñoz-Merino, P. J., Broos, T., & De Laet, T. (2020). Leadership and maturity: How do they affect learning analytics adoption in Latin America? In D. Ifenthaler & D. Gibson (Eds.), Adoption of data analytics in higher education learning and teaching (pp. 305–326). Cham: Springer International Publishing. doi: https://doi.org/10.1007/978-3-030-47392-1_16
Johnson, R. A., & Wichern, D. W. (2008). Applied multivariate statistical analysis (6th ed.). Pearson.
Kelvin, T., Leandro, F., Fagundes, R., & Freitas, E. (2021). Políticas para adoção de learning analytics: Uma proposta baseada nas opiniões dos estudantes. In Anais do xxxii simpósio brasileiro de informática na educação (pp. 885–896). Porto Alegre, RS, Brasil: SBC. doi: https://doi.org/10.5753/sbie.2021.218599
Keystone Strategy. (2016). Data analytics maturity model business impact. white paper. Keystone Strategy. Retrieved from https://info.microsoft.com/rs/157-GQE-382/images/EN-CNTNT-SQL-Data%20Analytics%20Maturity%20Model-en-us.pdf
Khalil, M., Prinsloo, P., & Slade, S. (2022). A comparison of learning analytics frameworks: A systematic review. In Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK 2022), 21–25 March 2022, online (pp. 152–163). ACM. https://doi.org/10.1145/3506860.3506878
Klimko, G. (2001). Knowledge management and maturity models: building common understanding. In D. Remenyi (Ed.), Proceedings of the Second European Conference on Knowledge Management (ECKM 2001), 8–9 November 2001, Bled, Slovenia (pp. 269–278). MCIL.
Kloke, J., & Mckean, J. W. (2015). Nonparametric statistical methods using R (1st ed.). CRC Press.
Knobbout, J., & van der Stappen, E. (2020, Mar.). A capability model for learning analytics adoption: Identifying organizational capabilities from literature on learning analytics, big data analytics, and business analytics. International Journal of Learning Analytics and Artificial Intelligence for Education (iJAI), 2(1), pp. 47–66. doi: https://doi.org/10.3991/ijai.v2i1.12793
Knobbout, J., van der Stappen, E., Versendaal, J., & van de Wetering, R. (2023). Supporting learning analytics adoption: Evaluating the learning analytics capability model in a real-world setting. Applied Sciences, 13(5). doi: https://doi.org/10.3390/app13053236
Li, M., & Smidts, C. (2003). A ranking of software engineering measures based on expert opinion. IEEE Transactions on Software Engineering, 29(9), 811-824. doi: https://doi.org/10.1109/TSE.2003.1232286
Mettler, T. (2010). Thinking in terms of design decisions when developing maturity models. International Journal of Strategic Decision Sciences, 1(10). https://doi.org/10.4018/jsds.2010100105
Molsey, M., Brackett, M., Earley, S., & Henderson, D. (2009). The DAMA guide to the data management body of knowledge (DAMA-DMBOK, 1st ed.). Technics Publications.
Pedhazur, E. J., & Schmelkin, L. P. (2013). Measurement, design, and analysis: An integrated approach. Psychology Press.
Pérez-Sanagustín, M., Hilliger, I., Maldonado-Mahauad, J., & Pérez-Álvarez, R. (2022). Building institutional capacity for learning analytics: Top-down & bottom-up initiatives. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje, 17(3), 281–289. https://doi.org/10.1109/RITA.2022.3191413
Pérez-Sanagustín, M., Hilliger, I., Maldonado-Mahauad, J., Pérez-Álvarez, R., Natera Ramirez, L., Muñoz-Merino, P., Tsai, Y.-S., Ortiz-Rojas, M., Broos, T., Ángel Zuñiga-Prieto, M., Sheihing, E., & Whitelock-Wainwright, A. (2019, April). Building capacity to use learning analytics to improve higher education in Latin America: LALA Framework (Version 2.0). European Commission.
Pontual Falcão, T., Lins Rodrigues, R., Cechinel, C., Dermeval, D., Harada Teixeira de Oliveira, E., Gasparini, I., D. Araújo, R., Primo, T., Gašević, D., & Ferreira Mello, R. (2022). A penny for your thoughts: Students and instructors’ expectations about learning analytics in Brazil. In LAK22: 12th International Learning Analytics and Knowledge Conference (p. 186–196). New York, NY, USA: Association for Computing Machinery. doi: https://doi.org/10.1145/3506860.3506886
Purwoningsih, T., Santoso, H. B., Isal, Y. K., & Hasibuan, Z. A. (2018). The pedagogy optimization with educational data mining and learning analytics for e-learning system - a review of the literature review. In Third International Conference on Informatics and Computing (ICIC) (p. 1-5). doi: https://doi.org/10.1109/IAC.2018.8780514
Scheneider, T., Cechinel, C., Mello, R., Freitas, E., Falc˜ao, T., & Ramos, V. (2022). Análise do Nível de Maturidade na Adoção de Learning Analytics em Instituições de Ensino Superior Brasileiras. In Anais do XXXII Workshop de Aplicações Práticas de Learning Analytics em Instituições de Ensino no Brasil (SBIE 2022), 22–26 November 2021, online (pp. 11–20). Sociedade Brasileira de Computação. https://doi.org/10.5753/wapla.2022.226423
Sen, A., Ramamurthy, K., & Sinha, A. P. (2012). A model of data warehousing process maturity. IEEE Transactions on Software Engineering, 38(2), 336-353. doi: https://doi.org/10.1109/TSE.2011.2
Siemens, G., & Baker, R. S. J. d. (2012). Learning analytics and educational data mining: Towards communication and collaboration. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (p. 252–254). New York, NY, USA: Association for Computing Machinery. doi: https://doi.org/10.1145/2330601.2330661
Siemens, G., Dawson, S., & Lynch, G. (2013). Improving the quality of productivity of the higher education sector: Policy and strategy for systems-level deployment of learning analytics. Solar Research. Retrieved from https://solaresearch.org/wp-content/uploads/2017/06/SoLAR_Report_2014.pdf
Tsai, Y. S., & Gašević, D. (2017a). Learning analytics in higher education—challenges and policies: A review of eight learning analytics policies. In Proceedings of the Seventh International Learning Analytics; Knowledge Conference (p. 233–242). New York, NY, USA: Association for Computing Machinery. doi: https://doi.org/10.1145/3027385.3027400
Tsai, Y. S., & Gašević, D. (2017b). The state of learning analytics in Europe – SHEILA (executive summary). Sheila Project. Retrieved from http://sheilaproject.eu/2017/04/18/the-state-of-learning-analytics-in-europe-executive-summary
Tsai, Y.-S., Kovanovic, V., & Gašević, D. (2021). Connecting the dots: An exploratory study on learning analytics adoption factors, experience, and priorities. The Internet and Higher Education, 50, 100794. doi: https://doi.org/10.1016/j.iheduc.2021.100794
Tsai, Y.-S., Moreno-Marcos, P. M., Jivet, I., Scheffel, M., Tammets, K., Kollom, K., & Gašević, D. (2018, Nov.). The SHEILA framework: Informing institutional strategies and policy processes of learning analytics. Journal of Learning Analytics, 5(3), 5–20. doi: https://doi.org/10.18608/jla.2018.53.2
Tsai, Y. S., Poquet, O., Gašević, D., Dawson, S., & Pardo, A. (2019). Complexity leadership in learning analytics: Drivers, challenges and opportunities. British Journal of Educational Technology, 50(6), 2839-2854. doi: https://doi.org/10.1111/bjet.12846
Tsai, Y.-S., Rates, D., Moreno-Marcos, P. M., Muñoz-Merino, P. J., Jivet, I., Scheffel, M., . . . Gašević, D. (2020). Learning analytics in European higher education—Trends and barriers. Computers Education, 155, 103933. doi: https://doi.org/10.1016/j.compedu.2020.103933
Viberg, O., Hatakka, M., Bälter, O., & Mavroudi, A. (2018). The current landscape of learning analytics in higher education. Computers in Human Behavior, 89, 98-110. doi: https://doi.org/10.1016/j.chb.2018.07.027
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
