Real-Time Prediction of Students’ Activity Progress and Completion Rates
DOI:
https://doi.org/10.18608/jla.2020.72.2Keywords:
instructional timing, progress predictions, classroom orchestrationAbstract
In classrooms, some transitions between activities impose (quasi-)synchronicity, meaning there is a need for learners to move between activities at the same time. To make real-time decisions about when to move to the next activity, teachers need to be able to balance the progress of their students as they work at different paces. In this paper, we present a set of estimators that can be used in real time to predict the progress and completion rates of students working on computer-supported activities that can be divided into sequential subtasks. With our estimators, we investigate what effect the average progress rate of the class, a given number of previous steps, or weighting the proportion of progress assigned to each subtask has on predictions of students’ progress. We find that accounting for the average class progress rate near the beginning of the activity can improve predictions over baseline. Additionally, weighted subtasks decrease prediction accuracy for activities where the behaviour of faster students diverges from the average behaviour of the class. This paper contributes to our ability to provide accurate student progress predictions and to understand the behaviour of students as they progress through the activity. These real-time predictions can enable teachers to optimize learning time in their classrooms.
References
Alavi, H. S., & Dillenbourg, P. (2012). An ambient awareness tool for supporting supervised collaborative problem solving.
IEEE Transactions on Learning Technologies, 5(3), 264–274. https://dx.doi.org/10.1109/TLT.2012.7
Baker, R., Corbett, A. T., & Koedinger, K. R. (2004). Detecting student misuse of intelligent tutoring systems. In 7th International Conference on Intelligent Tutoring Systems (ITS2004), 30 August–3 September 2004, Macei´o, Alagoas, Brazil (pp. 531–540). Springer. https://dx.doi.org/10.1007/978-3-540-30139-450
Baker, R., & Siemens, G. (2014). Educational data mining and learning analytics. In R. K. Sawyer (Ed.), Cambridge Handbook of the Learning Sciences. Cambridge, UK: Cambridge University Press. https://dx.doi.org/10.1017/CBO9781139519526.016
Beck, J. E. (2005). Engagement tracing: Using response times to model student disengagement. In C. Looi, G. I. McCalla, B. Bredeweg, & J. Breuker (Eds.), Artificial Intelligence in Education—Supporting Learning through Intelligent and Socially Informed Technology, Proceedings of the 12th International Conference on Artificial Intelligence in Education, AIED 2005, 18–22 July 2005, Amsterdam, Netherlands (Vol. 125, pp. 88–95). IOS Press. Retrieved from http://www.booksonline.iospress.nl/Content/View.aspx?piid=1296
Boroujeni, M. S., Sharma, K., Kidzi´nski, Ł., Lucignano, L., & Dillenbourg, P. (2016). How to quantify student’s regularity? In Proceedings of the 11th European Conference on Technology Enhanced Learning (EC-TEL 2016), 13–16 September 2016, Lyon, France (pp. 277–291). Springer. https://dx.doi.org/10.1007/978-3-319-45153-421
Boyer, S., & Veeramachaneni, K. (2015). Transfer learning for predictive models in massive open online courses. In Proceedings of the 17th International Conference on Artificial Intelligence in Education (AIED 2015), 22–26 June 2015, Madrid, Spain (pp. 54–63). Springer. https://dx.doi.org/10.1007/978-3-319-19773-96
Campbell, S., & Skinner, C. H. (2004). Combining explicit timing with an interdependent group contingency program to decrease transition times: An investigation of the timely transitions game. Journal of Applied School Psychology, 20(2), 11–27. https://dx.doi.org/10.1300/J370v20n0202
Charleer, S., Santos, J. L., Klerkx, J., & Duval, E. (2014). Improving teacher awareness through activity, badge and content visualizations. In Proceedings of the 13th International Conference on Web-Based Learning (ICWL 2014), 14–17 August 2014, Tallinn, Estonia (pp. 143–152). Springer. https://dx.doi.org/10.1007/978-3-319-13296-916
Clark, D., & Linn, M. C. (2003). Designing for knowledge integration: The impact of instructional time. The Journal of the Learning Sciences, 12(4), 451–493. https://dx.doi.org/10.1207/S15327809JLS12041
Corbett, A. T., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4), 253–278. https://dx.doi.org/10.1007/BF01099821
Crossley, S., Paquette, L., Dascalu, M., McNamara, D. S., & Baker, R. S. (2016). Combining click-stream data with NLP tools to better understand MOOC completion. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (LAK 2016), 25–29 April 2015, Edinburgh, United Kingdom (pp. 6–14). New York, USA: ACM. https://dx.doi.org/10.1145/2883851.2883931
Dekker, G. W., Pechenizkiy, M., & Vleeshouwers, J. M. (2009). Predicting students drop out: A case study. In T. Barnes, M. Desmarais, C. Romero, & S. Ventura (Eds.), Proceedings of the Second International Conference on Educational Data Mining (EDM ’09), 1–3 July 2009, Cordoba, Spain. Retrieved from http://www.educationaldatamining.org/EDM2009/uploads/proceedings/dekker.pdf
Dillenbourg, P. (2013). Design for classroom orchestration. Computers & Education, 69, 485–492. https://dx.doi.org/10.1016/j.compedu.2013.04.013
Dillenbourg, P. (2015). Orchestration Graphs. Lausanne, Switzerland: EPFL Press. Retrieved from https://www.epflpress.org/produit/735/9782940222841/orchestration-graphs
Dillenbourg, P., Li, N., & Kidzinski, L. (2016). The Complications of the Orchestration Clock (Tech. Rep.). Portland Press. Retrieved from https://portlandpress.com/DocumentLibrary/Umbrella/Wenner%20Gren/Vol%2088/PPLWennerCh02.pdf
Dillenbourg, P., Prieto, L. P., & Olsen, J. K. (2018). Classroom orchestration. In F. Fischer, C. E. Hmelo-Silver, S. R. Goldman, & P. Reimann (Eds.), International Handbook of the Learning Sciences (pp. 180–190). Routledge. https://dx.doi.org/10.4324/9781315617572
Dillenbourg, P., Zufferey, G., Alavi, H. S., Jermann, P., Do, L. H. S., Bonnard, Q., . . . Kaplan, F. (2011). Classroom orchestration: The third circle of usability. In Proceedings of the Ninth Annual Conference on Computer-Supported Collaborative Learning (CSCL2011), 4–8 July 2011, Hong Kong, China (Vol. 1, pp. 510–517). International Society of the Learning Sciences. Retrieved from https://infoscience.epfl.ch/record/168741?ln=en
D’mello, S. K., Craig, S. D., Witherspoon, A., Mcdaniel, B., & Graesser, A. (2008). Automatic detection of learner’s affect from conversational cues. User Modeling and User-Adapted Interaction, 18(1-2), 45–80. https://dx.doi.org/10.1007/s11257-0079037-6
Feng, M., Heffernan, N., & Koedinger, K. (2009). Addressing the assessment challenge with an online system that tutors as it assesses. User Modeling and User-Adapted Interaction, 19(3), 243–266. https://dx.doi.org/10.1007/s11257-009-9063-7
Fiel, J., Lawless, K. A., & Brown, S. W. (2018). Timing matters: Approaches for measuring and visualizing behaviours of timing and spacing of work in self-paced online teacher professional development courses. Journal of Learning Analytics, 5(1), 25–40. https://dx.doi.org/10.18608/jla.2018.51.3
Fraser, B. J.,Walberg, H. J.,Welch, W.W., & Hattie, J. A. (1987). Syntheses of educational productivity research. International Journal of Educational Research, 11(2), 147–252. https://dx.doi.org/10.1016/0883-0355(87)90035-8
Gettinger, M., & Seibert, J. K. (2002). Best practices in increasing academic learning time. Best Practices in School Psychology, IV, 773–787. Retrieved from https://psycnet.apa.org/record/2006-03715-049
Haklev, S., Faucon, L., Hadzilacos, T., & Dillenbourg, P. (2017). Orchestration graphs: Enabling rich social pedagogical scenarios in MOOCs. In Proceedings of the Fourth ACM Conference on Learning@Scale (L@S 2017), 20–21 April 2017, Cambridge, Massachusetts, USA (pp. 261–264). New York, USA: ACM. https://dx.doi.org/10.1145/3051457.3054000
Hine, J. F., Ardoin, S. P., & Foster, T. E. (2015). Decreasing transition times in elementary school classrooms: Using computer-assisted instruction to automate intervention components. Journal of Applied Behavior Analysis, 48(3), 495–510. https://dx.doi.org/10.1002/jaba.233
Holstein, K., Hong, G., Tegene, M., McLaren, B. M., & Aleven, V. (2018). The classroom as a dashboard: Co-designing wearable cognitive augmentation for K–12 teachers. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 5–9 March 2018, Sydney, Australia (pp. 79–88). New York, USA: ACM. https://dx.doi.org/10.1145/3170358.3170377
Hutt, S., Mills, C., Bosch, N., Krasich, K., Brockmole, J., & D’Mello, S. (2017). Out of the fr-eye-ing pan: Towards gaze-based models of attention during learning with technology in the classroom. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization (UMAP 2017), 9–12 July 2017, Bratislava, Slovakia (pp. 94–103). New York, USA: ACM. https://dx.doi.org/10.1145/3079628.3079669
Jovanovi´c, J., Gaˇsevi´c, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet and Higher Education, 33(4), 74–85. Retrieved from https://psycnet.apa.org/doi/10.1016/j.iheduc.2017.02.001
Karweit, N., & Slavin, R. E. (1981). Measurement and modeling choices in studies of time and learning. American Educational Research Journal, 18(2), 157–171. https://dx.doi.org/10.3102F00028312018002157
Klerkx, J., Verbert, K., & Duval, E. (2017). Learning analytics dashboards. Handbook of Learning Analytics, 1, 143–150. https://dx.doi.org/10.18608/hla17.012
Kollar, I., & Fischer, F. (2013). Orchestration is nothing without conducting—But arranging ties the two together!: A response to Dillenbourg (2011). Computers & Education, 69, 507–509. https://dx.doi.org/10.1016/j.compedu.2013.04.008
Loboda, T. D., Guerra, J., Hosseini, R., & Brusilovsky, P. (2014). Mastery grids: An open source social educational progress visualization. In Proceedings of the Ninth European Conference on Technology Enhanced Learning (EC-TEL 2014), 16–19 September 2014, Graz, Austria (pp. 235–248). Springer. https://dx.doi.org/10.1007/978-3-319-11200-818
Malone, T. W. (1981). Toward a theory of intrinsically motivating instruction. Cognitive Science, 5(4), 333–369. https://dx.doi.org/10.1207/s15516709cog05042
Martinez-Maldonado, R., Clayphan, A., & Kay, J. (2015). Deploying and visualising teacher’s scripts of small group activities in a multi-surface classroom ecology: A study in-the-wild. Computer Supported Cooperative Work (CSCW), 24(2-3), 177–221. https://dx.doi.org/10.1007/s10606-015-9217-6
Ming, N., & Ming, V. (2012). Predicting student outcomes from unstructured data. In Workshop and Poster Proceedings of the 20th Conference on User Modeling, Adaptation, and Personalization (UMAP 2012), 16–20 July 2012, Montreal, Canada. Retrieved from http://ceur-ws.org/Vol-872/pale2012paper2:pdf
Molenaar, I., & Knoop-van Campen, C. (2017). Teacher dashboards in practice: Usage and impact. In Proceedings of the 12th
European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 125–138). Springer. https://dx.doi.org/10.1007/978-3-319-66610-510
Pavlik Jr., P. I., Cen, H., & Koedinger, K. R. (2009). Performance factors analysis—A new alternative to knowledge tracing. (online submission) Retrieved from http://pact.cs.cmu.edu/koedinger/pubs/AIED%202009%20final%20Pavlik%20Cen%20Keodinger%20corrected.pdf
Pel´anek, R. (2015). Metrics for evaluation of student models. Journal of Educational Data Mining, 7(2), 1–19. Retrieved
from https://jedm.educationaldatamining.org/index.php/JEDM/article/download/JEDM087/pdf12
Phiri, L., Meinel, C., & Suleman, H. (2016). Streamlined orchestration: An orchestration workbench framework for effective teaching. Computers & Education, 95, 231–238. https://dx.doi.org/10.1016/j.compedu.2016.01.011
Prieto, L. P., Dlab, M. H., Guti´errez, I., Abdulwahed, M., & Balid, W. (2011). Orchestrating technology enhanced learning: A literature review and a conceptual framework. International Journal of Technology Enhanced Learning, 3(6), 583. https://dx.doi.org/10.1504/IJTEL.2011.045449
Rhymer, K. N., Skinner, C. H., Jackson, S., McNeill, S., Smith, T., & Jackson, B. (2002). The 1-minute explicit timing intervention: The influence of mathematics problem difficulty. Journal of Instructional Psychology, 29(4), 305–311. Retrieved from https://psycnet.apa.org/record/2002-11085-011
Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., . . . Dillenbourg, P. (2016). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30–41. https://dx.doi.org/10.1109/TLT.2016.2599522
Slotta, J. D., Tissenbaum, M., & Lui, M. (2013). Orchestrating of complex inquiry: Three roles for learning analytics in a smart classroom infrastructure. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK 2013), 8–12 April 2013, Leuven, Belgium (pp. 270–274). New York, USA: ACM. https://dx.doi.org/10.1145/2460296.2460352
Song, Y., Wong, L.-H., & Looi, C.-K. (2012). Fostering personalized learning in science inquiry supported by mobile technologies. Educational Technology Research and Development, 60(4), 679–701. https://dx.doi.org/10.1007/s11423-012-9245-6
Taylor, C., Veeramachaneni, K., & O’Reilly, U.-M. (2014). Likely to stop? Predicting stopout in massive open online courses. arXiv preprint arXiv:1408.3382.
Van Houten, R., Hill, S., & Parsons, M. (1975). An analysis of a performance feedback system: The effects of timing and feedback, public posting, and praise upon academic performance and peer interaction 1. Journal of Applied Behavior Analysis, 8(4), 449–457. https://dx.doi.org/10.1901/jaba.1975.8-449
Van Houten, R., & Thompson, C. (1976). The effects of explicit timing on math performance. Journal of Applied Behavior Analysis, 9(2), 227. https://dx.doi.org/10.1901%2Fjaba.1976.9-227
van Leeuwen, A., Rummel, N., & van Gog, T. (2019). What information should CSCL teacher dashboards provide to help teachers interpret CSCL situations? International Journal of Computer-Supported Collaborative Learning, 14, 261–289. https://dx.doi.org/10.1007/s11412-019-09299-x
van Leeuwen, A., van Wermeskerken, M., Erkens, G., & Rummel, N. (2017). Measuring teacher sense making strategies of learning analytics: A case study. Learning: Research and Practice, 3(1), 42–58. https://dx.doi.org/10.1080/23735082.2017.1284252
Vanlehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265. https://dx.doi.org/10.5555/1435351.1435353
Verbert, K., Govaerts, S., Duval, E., Santos, J. L., Assche, F., Parra, G., & Klerkx, J. (2014). Learning dashboards: An overview and future research opportunities. Personal and Ubiquitous Computing, 18(6), 1499–1514. https://dx.doi.org/10.1007/s00779-013-0751-2
Wachtler, J., Khalil, M., Taraghi, B., & Ebner, M. (2016). On using learning analytics to track the activity of interactive MOOC videos. In Proceedings of the LAK 2016 Workshop on Smart Environments and Analytics in Video-Based Learning, 26 April 2016, Edinburgh, United Kingdom (pp. 8–17). Retrieved from http://ceur-ws.org/Vol-1579/paper3.pdf
Wang, P., Tchounikine, P., & Quignard, M. (2017). Chao: A framework for the development of orchestration technologies for technology-enhanced learning activities using tablets in classrooms. International Journal of Technology Enhanced Learning, 10(1/2), 1–21. https://dx.doi.org/10.5555/3202163.3202164
Whitehill, J., Mohan, K., Seaton, D., Rosen, Y., & Tingley, D. (2017). Delving deeper into MOOC student dropout prediction. arXiv preprint arXiv:1702.06404.
Xu, D., Wang, H., & Wang, M. (2005). A conceptual model of personalized virtual learning environments. Expert Systems with Applications, 29(3), 525–534. https://dx.doi.org/10.1016/j.eswa.2005.04.028
Yarbrough, J. L., Skinner, C. H., Lee, Y. J., & Lemmons, C. (2004). Decreasing transition times in a second grade classroom: Scientific support for the timely transitions game. Journal of Applied School Psychology, 20(2), 85–107. https://dx.doi.org/10.1300/J370v20n0206
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2020 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
