Combining Checkpoint and Process Learning Analytics to Support Learning Design Decisions in Blended Learning Environments
DOI:
https://doi.org/10.18608/jla.2020.73.4Keywords:
Learning analytics, Learning design, Blended learning, Teachers, Canvas, Social network analysis, Text network analysisAbstract
Learning analytics (LA) constitutes a key opportunity to support learning design (LD) in blended learning environments. However, details as to how LA supports LD in practice and information on teacher experiences with LA are limited. This study explores the potential of LA to inform LD based on a one-semester undergraduate blended learning course at a Norwegian university. Our findings indicate that creating valuable connections between LA and LD requires a detailed analysis of student checkpoints (e.g., online logins) and process analytics (e.g., online content and interaction dynamics) to find meaningful learning behaviour patterns that can be forwarded to teachers in retrospect to support the redesign of courses. Moreover, the teachers in our study found the LA visualizations to be valuable for understanding student online learning processes, but they also requested the timely sharing of aggregated LA visualizations in a simple, easy-to-interpret format, yet detailed enough to be informative and actionable. We conclude the paper by arguing that the potential of LA to support LD is improved when multiple levels of LA are considered.
References
Agostinho, S. (2011). The use of a visual learning design representation to support the design process of teaching in higher education. Australasian Journal of Educational Technology, 27(6). https://https://dx.doi.org/10.14742/ajet.923
Alhadad, S. S., & Thompson, K. (2017). Understanding the mediating role of teacher inquiry when connecting learning analytics with design for learning. Interaction Design and Architecture(s) Journal, 33, 54–74. Retrieved from https://eprints.qut.edu.au/202493/1/62149001.pdf
Andersen, R., & Mørch, A. I. (2016). Mutual development in mass collaboration: Identifying interaction patterns in customer-initiated software product development. Computers in Human Behavior, 65, 77–91. https://dx.doi.org/10.1016/j.chb.2016.08.005
Bakharia, A., Corrin, L., de Barba, P., Kennedy, G., Gašević, D., Mulder, R., Lockyer, L. (2016). A conceptual framework linking learning design with learning analytics. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 329–338). New York, NY: ACM. https://dx.doi.org/10.1145/2883851.2883944
Bennett, S., Agostinho, S., & Lockyer, L. (2015). Technology tools to support learning design: Implications derived from an investigation of university teachers’ design practices. Computers & Education, 81, 211–220. https://dx.doi.org/10.1016/j.compedu.2014.10.016
Benesty, J., Chen, J., Huang, Y., & Cohen, I. (2009). Pearson correlation coefficient. In I. Cohen, Y. Huang, J. Chen, & J. Benesty (Eds.), Noise reduction in speech processing (pp. 1–4). Berlin, Germany: Springer. https://dx.doi.org/10.1007/978-3-642-00296-0_5
Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-centred learning analytics. Journal of Learning Analytics, 6(2), 1–9. https://dx.doi.org/10.18608/jla.2019.62.1
Bonk, C. J., & Graham, C. R. (2012). The handbook of blended learning: Global perspectives, local designs. Hoboken, NJ: John Wiley & Sons. Retrieved from https://books.google.no/books?hl=en&lr=&id=2u2TxK06PwUC&oi=fnd&pg=PT14&dq=
Conole, G. (2012). Designing for learning in an open world (Vol. 4). Leicester, UK: Springer Science & Business Media. Retrieved from https://dl.uswr.ac.ir/bitstream/Hannan/130952/1/9781441985163.pdf https://dx.doi.org/10.1007/978-1-4419-8517-0
Dobozy, E., & Cameron, L. (2018). Special issue on learning design research: Mapping the terrain. Australasian Journal of Educational Technology, 34(2). https://dx.doi.org/10.14742/ajet.4390
Eradze, M., Rodriguez Triana, M. J., & Laanpere, M. (2017). Semantically annotated lesson observation data in learning analytics datasets: A reference model. Interaction Design and Architecture(s) Journal, 33, 75–91. Retrieved from http://www.mifav.uniroma2.it/inevent/events/idea2010/doc/33_4.pdf
Fritz, J. (2016). LMS course design as learning analytics variable. In J. Greer, M. Molinaro, X. Ochoa, & T. McKay (Eds.), Proceedings of the 1st Learning Analytics for Curriculum and Program Quality Improvement Workshop (PCLA 2016), 25 April 2016, Edinburgh, UK (pp. 15–19). Retrieved from https://www.semanticscholar.org/paper/LMS-Course-Design-As-Learning-Analytics-Variable-Fritz/be7e8fe65384513355467e56cc84b36829669482
Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education, 28, 68–84. https://dx.doi.org/10.1016/j.iheduc.2015.10.002
Griffiths, D. (2017). The use of models in learning design and learning analytics. Interaction Design and Architecture(s) Journal, 33, 113–133. Retrieved from http://ubir.bolton.ac.uk/1330/1/Griffiths%20D%20IxD%26A%20article.pdf
Gunn, C., McDonald, J., Donald, C., Nichols, M., Milne, J., & Blumenstein, M. (2017). Building an evidence base for teaching and learning design using learning analytics. Wellington, Australia: Ako Aotearoa – The National Centre for Tertiary Teaching Excellence. Retrieved from https://ako.ac.nz/assets/Knowledge-centre/NPF-15-008
Haya, P. A., Daems, O., Malzahn, N., Castellanos, J., & Hoppe, H. U. (2015). Analysing content and patterns of interaction for improving the learning design of networked learning environments. British Journal of Educational Technology, 46(2), 300–316. https://dx.doi.org/10.1111/bjet.12264
Hernández‐Leo, D., Martinez‐Maldonado, R., Pardo, A., Muñoz‐Cristóbal, J. A., & Rodríguez‐Triana, M. J. (2019). Analytics for learning design: A layered framework and tools. British Journal of Educational Technology, 50(1), 139–152. https://dx.doi.org/10.1111/bjet.12645
Harel, D., & Koren, Y. (2001). A fast multi-scale method for drawing large graphs. In J. Marks (Ed.) Graph drawing. GD 2000. Notes in computer science (Vol. 1984). Berlin, Germany: Springer. https://dx.doi.org/10.1007/3-540-44541-2_18
Ifenthaler, D., Gibson, D., & Dobozy, E. (2018). Informing learning design through analytics: Applying network graph analysis. Australasian Journal of Educational Technology, 34(2), 117–132. https://dx.doi.org/10.14742/ajet.3767
Kaliisa, R., Mørch, A. I., & Kluge, A. (2019). Exploring social learning analytics to support teaching and learning decisions in online learning environments. In M. Scheffel, J. Broisin, V. Pammer-Schindler, A. Ioannou, & J. Schneider (Eds.), Transforming learning with meaningful technologies. Proceedings of the 14th European Conference on Technology Enhanced Learning (EC-TEL 2019), 16–19 September 2019, Delft, The Netherlands. Lecture Notes in Computer Science (Vol. 11722) (pp. 187–198). Cham, Switzerland: Springer. https://dx.doi.org/10.1007/978-3-030-29736-7_14
Kaliisa, R., Kluge, A., & Mørch, A. I. (in press a). Overcoming the challenges to learning analytics adoption at the practitioner level: A critical analysis of 18 learning analytics frameworks. Scandinavian Journal of Educational Research.
Kaliisa, R., Kluge, A., & Mørch, A. I.. (in press b). “My point of departure for analytics is extreme skepticism”: Implications derived from an investigation of university teachers’ learning analytics perspectives and design practices. Technology, Knowledge & Learning.
Knight, S., Shum, S. B., & Littleton, K. (2014). Epistemology, assessment, pedagogy: Where learning meets analytics in the middle space. Journal of Learning Analytics, 1(2), 23–47. https://dx.doi.org/10.18608/jla.2014.12.3
Law, N., Li, L., Herrera, L. F., Chan, A., & Pong, T.-C. (2017). A pattern language based learning design studio for an analytics informed inter-professional design community. Interaction Design and Architecture(s) Journal, 33, 92–112. Retrieved from http://www.mifav.uniroma2.it/inevent/events/idea2010/doc/33_5.pdf
Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. https://dx.doi.org/10.1177/0002764213479367
Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. https://dx.doi.org/10.1016/j.compedu.2009.09.008
Macfadyen, L. P., & Dawson, S. (2012). Numbers are not enough: Why e-learning analytics failed to inform an institutional strategic plan. Journal of Educational Technology & Society, 15(3), 149–163. Retrieved from https://www.jstor.org/stable/10.2307/jeductechsoci.15.3.149
Mangaroska, K., & Giannakos, M. N. (2018). Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, 12(4), 516–534. https://dx.doi.org/10.1109/TLT.2018.2868673
Martinez-Maldonado, R., Kay, J., Yacef, K., & Schwendimann, B. (2012). An interactive teacher’s dashboard for monitoring groups in a multi-tabletop learning environment. In S. A. Cerri, W. J. Clancey, G. Papadourakis, & K. Panourgia (Eds.), Proceedings of the 11th International Conference on Intelligent Tutoring Systems (ITS 2012), 14–18 June 2012, Chania, Greece,. Lecture notes in computer science (Vol. 7315) (pp. 482–492). Berlin, Germany: Springer. https://dx.doi.org/10.1007/978-3-642-30950-2_62
McKenney, S., & Mor, Y. (2015). Supporting teachers in data‐informed educational design. British Journal of Educational Technology, 46(2), 265–279. https://dx.doi.org/10.1111/bjet.12262
Michos, K., Hernández‐Leo, D., & Albó, L. (2018). Teacher‐led inquiry in technology‐supported school communities. British Journal of Educational Technology, 49(6), 1077–1095. https://dx.doi.org/10.1111/bjet.12696
Mor, Y., Ferguson, R., & Wasson, B. (2015). Learning design, teacher inquiry into student learning and learning analytics: A call for action. British Journal of Educational Technology, 46(2), 221–229. https://dx.doi.org/10.1111/bjet.12273
Melero, J., Hernández‐Leo, D., Sun, J., Santos, P., & Blat, J. (2015). How was the activity? A visualization support for a case of location‐based learning design. British Journal of Educational Technology, 46(2), 317–329. https://dx.doi.org/10.1111/bjet.12238
Nguyen, Q., Huptych, M., & Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 141–150). New York, NY: ACM. https://dx.doi.org/10.1145/3170358.3170398
Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., & Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior, 76, 703–714. https://dx.doi.org/10.1016/j.chb.2017.03.028
Paranyushkin, D. (2019). InfraNodus: Generating insight using text network analysis. Proceedings of the 28th International Conference on World Wide Web (WWW ’19), 13–17 May 2019, San Francisco, CA, USA (pp. 3584–3589). New York, NY: ACM. https://dx.doi.org/10.1145/3308558.3314123
Persico, D., & Pozzi, F. (2015). Informing learning design with learning analytics to improve teacher inquiry. British Journal of Educational Technology, 46(2), 230–248. https://dx.doi.org/10.1111/bjet.12207
Rienties, B., Cross, S., & Zdrahal, Z. (2017). Implementing a learning analytics intervention and evaluation framework: What works? In D. Kei Daniel (Ed.) Big data and learning analytics in higher education (pp. 147–166). Cham, Switzerland: Springer. https://dx.doi.org/10.1007/978-3-319-06520-5_10
Rienties, B., & Toetenel, L. (2016). The impact of learning design on student behaviour, satisfaction and performance: A cross-institutional comparison across 151 modules. Computers in Human Behavior, 60, 333–341. https://dx.doi.org/10.1016/j.chb.2016.02.074
Rodríguez‐Triana, M. J., Martínez‐Monés, A., Asensio‐Pérez, J. I., & Dimitriadis, Y. (2015). Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL situations. British Journal of Educational Technology, 46(2), 330–343. https://dx.doi.org/10.1111/bjet.12198
Samuelsen, J., Chen, W., & Wasson, B. (2019). Integrating multiple data sources for learning analytics: Review of literature. Research and Practice in Technology Enhanced Learning, 14(11), 1-20. https://dx.doi.org/10.1186/s41039-019-0105-4
Shibani, A., Knight, S., & Shum, S. B. (2019). Contextualizable learning analytics design: A generic model and writing analytics evaluations. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (pp. 210–219). New York, NY: ACM. https://dx.doi.org/10.1145/3303772.3303785
Smith, M. A., Shneiderman, B., Milic-Frayling, N., Mendes Rodrigues, E., Barash, V., Dunne, C., & Gleave, E. (2009). Analyzing (social media) networks with NodeXL. Proceedings of the 4th International Conference on Communities and Technologies (C&T ’09), 25–27 June 2009, University Park, PA, USA (pp. 255–264). New York, NY: ACM. https://dx.doi.org/10.1145/1556460.1556497
Wise, A. F., & Jung, Y. (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://dx.doi.org/10.18608/jla.2019.62.4
Wise, A. F., & Shaffer, D. W. (2015). Why theory matters more than ever in the age of big data. Journal of Learning Analytics, 2(2), 5–13. https://dx.doi.org/10.18608/jla.2015.22.2
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2020 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
