A Learning Analytics Approach to Monitoring the Quality of Online One-to-One Tutoring
DOI:
https://doi.org/10.18608/jla.2022.7411Keywords:
online one-to-one tutoring, learning analytics, sequential pattern mining, decision trees, research paperAbstract
One-to-one online tutoring provided by human tutors can improve students’ learning outcomes. However, monitoring the quality of such tutoring is a significant challenge. In this paper, we propose a learning analytics approach to monitoring online one-to-one tutoring quality. The approach analyzes teacher behaviours and classifies tutoring sessions into those that are effective and those that are not effective. More specifically, we use sequential behaviour pattern mining to analyze tutoring sessions using the CM-SPAM algorithm and classify tutoring sessions into effective and less effective using the J-48 and JRIP decision tree classifiers. To show the feasibility of the approach, we analyzed data from 2,250 minutes of online one-to-one primary math tutoring sessions with 44 tutors from eight schools. The results showed that the approach can classify tutors’ effectiveness with high accuracy (F measures of 0.89 and 0.98 were achieved). The results also showed that effective tutors present significantly more frequent hint provision and proactive planning behaviours than their less-effective colleagues in these online one-to-one sessions. Furthermore, effective tutors sequence their monitoring actions with appropriate pauses and initiations of students’ self-correction behaviours. We conclude that the proposed approach is feasible for monitoring the quality of online one-to-one primary math tutoring sessions.
References
Amiri, M., & Ghonsooly, B. (2015). The relationship between English learning anxiety and the students’ achievement on examinations. Journal of Language Teaching and Research, 6(4), 855–865. http://dx.doi.org/10.17507/jltr.0604.20
Anania, J. (1983). The influence of instructional conditions on student learning and achievement. Evaluation in Education: An International Review Series, 7, 3–76. https://doi.org/10.1016/0191-765X(83)90002-2
Andres, J. M. A. L., Ocumpaugh, J., Baker, R. S., Slater, S., Paquette, L., Jiang, Y., … Biswas, G. (2019). Affect sequences and learning in Betty’s Brain. In Proceedings of the Ninth International Conference on Learning Analytics & Knowledge (LAK 2019), 4–8 March 2019, Tempe, Arizona, USA (pp. 383–390). New York: ACM. https://doi.org/10.1145/3303772.3303807
Bell, A. (1993). Principles for the design of teaching. Educational Studies in Mathematics, 24(1), 5–34. https://doi.org/10.1007/BF01273293
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. https://doi.org/10.1080/0969595980050102
Boroujeni, M. S., & Dillenbourg, P. (2019). Discovery and temporal analysis of MOOC study patterns. Journal of Learning Analytics, 6(1), 16–33. https://doi.org/10.18608/jla.2019.61.2
Butler, A. C., Karpicke, J. D., & Roediger III, H. L. (2007). The effect of type and timing of feedback on learning from multiple-choice tests. Journal of Experimental Psychology: Applied, 13(4), 273. https://doi.org/10.1037/1076-898X.13.4.273
Camacho, V. L., de la Guía, E., Olivares, T., Flores, M. J., & Orozco-Barbosa, L. (2020). Data capture and multimodal learning analytics focused on engagement with a new wearable IoT approach. IEEE Transactions on Learning Technologies, 13(4), 704–717. https://doi.org/10.1109/TLT.2020.2999787
Carter Jr., R. A., Rice, M., Yang, S., & Jackson, H. A. (2020). Self-regulated learning in online learning environments: strategies for remote learning. Information and Learning Sciences, 121(5/6), 321–329. https://doi.org/10.1108/ILS-04-2020-0114
Chi, M. T., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145–182. https://doi.org/10.1207/s15516709cog1302_1
Chi, M. T., De Leeuw, N., Chiu, M.-H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18(3), 439–477. https://doi.org/10.1016/0364-0213(94)90016-7
Cukurova, M., Mavrikis, M., Luckin, R., Clark, J., & Crawford, C. (2017). Interaction analysis in online maths human tutoring: The case of third space learning. In Proceedings of the International Conference on Artificial Intelligence In Education (AIED 2017), 28 June–1 July 2017, Wuhan, China (pp. 636-643). Cham: Springer. https://doi.org/10.1007/978-3-319-61425-0_80
Cullinane, C., & Montacute, R. (2020). COVID-19 and Social Mobility Impact Brief #1: School Shutdown. London, UK: Sutton Trust. Retrieved from https://eric.ed.gov/?id=ED605807
D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157. https://doi.org/10.1016/j.learninstruc.2011.10.001
Donald, C., Blake, A., Girault, I., Datt, A., & Ramsay, E. (2009). Approaches to learning design: Past the head and the hands to the HEART of the matter. Distance Education, 30(2), 179–199. https://doi.org/10.1080/01587910903023181
Fournier-Viger, P., Gomariz, A., Campos, M., & Thomas, R. (2014). Fast vertical mining of sequential patterns using co-occurrence information. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD 2014), 13–16 May 2014, Tainan, Taiwan (pp. 40–52). Cham: Springer. https://doi.org/10.1007/978-3-319-06608-0_4
Gitinabard, N., Heckman, S., Barnes, T., & Lynch, C. F. (2019). What will you do next? A sequence analysis on the student transitions between online platforms in blended courses. arXiv:1905.00928. Retrieved from https://arxiv.org/abs/1905.00928
Graesser, A. C. (1993). Questioning Mechanisms during Tutoring, Conversation, and Human-Computer Interaction (Tech. Rep.). Memphis State University Department of Psychology. https://apps.dtic.mil/sti/pdfs/ADA266420.pdf
Graesser, A. C., Person, N., Harter, D., & Tutoring Research Group. (2000). Teaching tactics in autotutor. Modelling human teaching tactics and strategies. International Journal of Artificial Intelligence in Education, 11, 1020–1029. http://users.sussex.ac.uk/~bend/its2000/graesser.pdf
Graesser, A. C., Person, N. K., & Magliano, J. P. (1995). Collaborative dialogue patterns in naturalistic one-to-one tutoring. Applied Cognitive Psychology, 9(6), 495–522. https://doi.org/10.1002/acp.2350090604
Hacker, D. J., Dole, J. A., Ferguson, M., Adamson, S., Roundy, L., & Scarpulla, L. (2015). The short-term and maintenance effects of self-regulated strategy development in writing for middle school students. Reading & Writing Quarterly, 31(4), 351–372. https://doi.org/10.1080/10573569.2013.869775
Hastie, M., Chen, N.-S., & Kuo, Y.-H. (2007). Instructional design for best practice in the synchronous cyber classroom. Journal of Educational Technology & Society, 10(4), 281–294. Retrieved from https://www.j-ets.net/collection/published-issues/10_4
Hodges, C., Moore, S., Lockee, B., Trust, T., & Bond, A. (2020). The difference between emergency remote teaching and online learning. Educause Review, 27. Retrieved from https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remote-teaching-and-online-learning
Hwang, G.-J., Hsu, T.-C., Lai, C.-L., & Hsueh, C.-J. (2017). Interaction of problem-based gaming and learning anxiety in language students’ English listening performance and progressive behavioral patterns. Computers & Education, 106, 26–42. https://doi.org/10.1016/j.compedu.2016.11.010
Johnson, G. M., & Bratt, S. E. (2009). Technology education students: E‐tutors for school children. British Journal of Educational Technology, 40(1), 32-41. https://doi.org/10.1111/j.1467-8535.2007.00805.x
Khan-Galaria, M., Cukurova, M., & Luckin, R. (2020). A framework for exploring the impact of tutor practices on learner self-regulation in online environments. In Proceedings of the International Conference on Artificial Intelligence in Education (AIED 2020), 6–10 July 2020, Ifrane, Morocco (pp. 135–139). Cham: Springer. https://doi.org/10.1007/978-3-030-52240-7_25
Kinnebrew, J. S., Loretz, K. M., & Biswas, G. (2013). A contextualized, differential sequence mining method to derive students’ learning behavior patterns. Journal of Educational Data Mining, 5(1), 190–219. https://doi.org/10.5281/zenodo.3554617
Kopp, B., Matteucci, M. C., & Tomasetto, C. (2012). E-tutorial support for collaborative online learning: An explorative study on experienced and inexperienced e-tutors. Computers & Education, 58(1), 12–20. https://doi.org/10.1016/j.compedu.2011.08.019
Kulik, J. A., & Kulik, C.-L. C. (1988). Timing of feedback and verbal learning. Review of Educational Research, 58(1), 79–97. https://doi.org/10.3102/00346543058001079
Lefevre, D., & Cox, B. (2017). Delayed instructional feedback may be more effective, but is this contrary to learners’ preferences? British Journal of Educational Technology, 48(6), 1357–1367. https://doi.org/10.1111/bjet.12495
Lepper, M. R., & Woolverton, M. (2002). The wisdom of practice: Lessons learned from the study of highly effective tutors. In J. Aronson (ed.), Improving Academic Achievement (pp. 135–158). Elsevier. https://doi.org/10.1016/B978-012064455-1/50010-5
Lin, X., Schwartz, D. L., & Hatano, G. (2005). Toward teachers’ adaptive metacognition. Educational Psychologist, 40(4), 245–255. https://doi.org/10.1207/s15326985ep4004_6
Malekian, D., Bailey, J., & Kennedy, G. (2020). Prediction of students’ assessment readiness in online learning environments: The sequence matters. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 382–391). New York: ACM. https://doi.org/10.1145/3375462.3375468
Martinez-Maldonado, R., Yacef, K., Kay, J., Kharrufa, A., & Al-Qaraghuli, A. (2011). Analysing frequent sequential patterns of collaborative learning activity around an interactive tabletop. In Proceedings of the Fourth International Educational Data Mining Conference (EDM 2011), 6–8 July 2011, Eindhoven, Netherlands. Retrieved from https://educationaldatamining.org/EDM2011/wp-content/uploads/proc/edm11_proceedings.pdf
Matcha, W., Gasevic, D., Jovanovic, J., Pardo, A., Lim, L., Maldonado-Mahauad, J., … Tsai, Y.-S. (2020). Analytics of learning strategies: Role of course design and delivery modality. Journal of Learning Analytics, 7(2), 45–71. https://doi.org/10.18608/jla.2020.72.3
Mathan, S. A., & Koedinger, K. R. (2002). An empirical assessment of comprehension fostering features in an intelligent tutoring system. In Proceedings of the International Conference on Intelligent Tutoring Systems (ITS 2002), 2–7 June 2002, Biarritz, France, and San Sebastian, Spain (pp. 330–343). Springer. https://doi.org/10.1007/3-540-47987-2_37
Merrill, D. C., Reiser, B. J., Ranney, M., & Trafton, J. G. (1992). Effective tutoring techniques: A comparison of human tutors and intelligent tutoring systems. The Journal of the Learning Sciences, 2(3), 277–305. https://doi.org/10.1207/s15327809jls0203_2
Narciss, S., & Huth, K. (2004). How to design informative tutoring feedback for multimedia learning. In H. M. Niegemann, D. Leutner, and R. Brünken (eds.), Instructional Design for Multimedia Learning, pp. 181–195. Kornwestheim, Germany: Waxmann Buecher. Retrieved from https://studierplatz2000.tu-dresden.de/toolkit/presentations/CD/Literatur/Publikationen/ID_nahu.pdf
Ozdagoglu, G., Oztas, G. Z., & Cagliyangil, M. (2019). An application framework for mining online learning processes through event-logs. Business Process Management Journal, 25(5), 860–886. https://doi.org/10.1108/BPMJ-10-2017-0279
Pardos, Z. A., Baker, R. S., San Pedro, M. O., Gowda, S. M., & Gowda, S. M. (2013). Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK 2013), 8–12 April 2013, Leuven, Belgium (pp. 117–124). New York: ACM. https://doi.org/10.1145/2460296.2460320
Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist, 37(2), 91–105. https://doi.org/10.1207/S15326985EP3702_4
Plass, J. L., Heidig, S., Hayward, E. O., Homer, B. D., & Um, E. (2014). Emotional design in multimedia learning: Effects of shape and color on affect and learning. Learning and Instruction, 29, 128–140. https://doi.org/10.1016/j.learninstruc.2013.02.006
Prieto, L. P., Sharma, K., Dillenbourg, P., & Jesús, M. (2016). Teaching analytics: Towards automatic extraction of orchestration graphs using wearable sensors. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 148–157). New York: ACM. https://doi.org/10.1145/2883851.2883927
Prieto, L. P., Sharma, K., Kidzinski, Ł., Rodríguez‐Triana, M. J., & Dillenbourg, P. (2018). Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning, 34(2), 193–203. https://doi.org/10.1111/jcal.12232
Rapanta, C., Botturi, L., Goodyear, P., Guàrdia, L., & Koole, M. (2020). Online university teaching during and after the Covid-19 crisis: Refocusing teacher presence and learning activity. Postdigital Science and Education, 2, 923–945. https://doi.org/10.1007/s42438-020-00155-y
Riel, J., Lawless, K. A., & Brown, S. W. (2018). Timing matters: Approaches for measuring and visualizing behaviours of timing and spacing of work in self-paced online teacher professional development courses. Journal of Learning Analytics, 5(1), 25–40. https://doi.org/10.18608/jla.2018.51.3
Ritter, S., Anderson, J. R., Koedinger, K. R., & Corbett, A. (2007). Cognitive tutor: Applied research in mathematics education. Psychonomic Bulletin & Review, 14, 249–255. https://doi.org/10.3758/BF03194060
Ryan, R. M., & Deci, E. L. (2009). Promoting self-determined school engagement: Motivation, learning, and well-being. In K. R. Wenzel & A. Wigfield (Eds.), Educational Psychology Handbook Series. Handbook of Motivation at School (pp. 171–195). Routledge/Taylor & Francis Group. https://psycnet.apa.org/record/2009-24219-009
Saint, J., Gasevic, D., Matcha, W., Uzir, N. A., & Pardo, A. (2020). Combining analytic methods to unlock sequential and temporal patterns of self-regulated learning. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 402–411). New York: ACM. https://doi.org/10.1145/3375462.3375487
Schmidt, H. G., Dolmans, D., Gijselaers, W. H., & Des Marchais, J. E. (1995). Theory-guided design of a rating scale for course evaluation in problem-based curricula. Teaching and Learning in Medicine: An International Journal, 7(2), 82–91. https://doi.org/10.1080/10401339509539719
Shield, M., & Dole, S. (2013). Assessing the potential of mathematics textbooks to promote deep learning. Educational Studies in Mathematics, 82, 183–199. https://doi.org/10.1007/s10649-012-9415-9
Shorrocks-Taylor, D., & Hargreaves, M. (1999). Making it clear: A review of language issues in testing with special reference to the national curriculum mathematics tests at key stage 2. Educational Research, 41(2), 123–136. https://doi.org/10.1080/0013188990410201
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795
Sleeman, D., Kelly, A. E., Martinak, R., Ward, R. D., & Moore, J. L. (1989). Studies of diagnosis and remediation with high school algebra students. Cognitive Science, 13(4), 551–568. https://doi.org/10.1016/0364-0213(89)90023-2
Torgerson, C., Ainsworth, H., Buckley, H., Hampden-Thompson, G., Hewitt, C., Humphry, D., … Torgerson, D. (2016). Affordable Online Maths Tuition: Evaluation Report and Executive Summary. Education Endowment Foundation. Retrieved from https://eric.ed.gov/?id=ED581116
Walsh, J. A., & Sattes, B. D. (2016). Quality Questioning: Research-Based Practice to Engage Every Learner. Corwin Press. ISBN: 1 74101 442 5. https://books.google.co.uk/books?id=lG51DQAAQBAJ&lr=
Wang, Y., Li, T., Geng, C., & Wang, Y. (2019). Evaluating student learning effect based on process mining. In Proceedings of the Second International Conference on Applied Informatics (ICAI 2019), 7–9 November 2019, Madrid, Spain (pp. 59–72). Springer. https://doi.org/10.1007/978-3-030-32475-9_5
Wittwer, J., Nückles, M., Landmann, N., & Renkl, A. (2010). Can tutors be supported in giving effective explanations? Journal of Educational Psychology, 102(1), 74. https://doi.org/10.1037/a0016727
Wittwer, J., Nückles, M., & Renkl, A. (2008). Is underestimation less detrimental than overestimation? The impact of experts’ beliefs about a layperson’s knowledge on learning and question asking. Instructional Science, 36, 27–52. https://doi.org/10.1007/s11251-007-9021-x
Wong, J., Khalil, M., Baars, M., de Koning, B. B., & Paas, F. (2019). Exploring sequences of learner activities in relation to self-regulated learning in a massive open online course. Computers & Education, 140, 103595. https://doi.org/10.1016/j.compedu.2019.103595
Wylie, R., & Chi, M. T. (2014). 17—The self-explanation principle in multimedia learning. In R. E. Mayer (ed.), The Cambridge Handbook of Multimedia Learning (pp. 413–432). Cambridge University Press. https://doi.org/10.1017/CBO9781139547369.021
Zheng, J., Xing, W., & Zhu, G. (2019). Examining sequential patterns of self- and socially shared regulation of STEM learning in a CSCL environment. Computers & Education, 136, 34–48. https://doi.org/10.1016/j.compedu.2019.03.005
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
