Markers of Cognitive Quality in Student Contributions to Online Course Discussion Forums
DOI:
https://doi.org/10.18608/jla.2022.7250Keywords:
discussion forum, participation, engagement, quality, community of inquiry, cognitive presence, ICAP, feature analysis, research paperAbstract
By participating in asynchronous course discussion forums, students can work together to refine their ideas and construct knowledge collaboratively. Typically, some messages simply repeat or paraphrase course content, while others bring in new material, demonstrate reasoning, integrate concepts, and develop solutions. Through the messages they send, students thus display different levels of intellectual engagement with the topic and the course. We refer to this as cognitive quality. The work presented here used two widely studied frameworks for assessing critical discourse and cognitive engagement: the ICAP and Community of Inquiry frameworks. The constructs of the frameworks were used as proxy measures for cognitive quality. Predictive classifiers were trained for both frameworks on the same data in order to discover which attributes of the dialogue were most informative and how those attributes were correlated with framework constructs. We found that longer and more complex messages were associated with indicators of greater quality in both frameworks, and that the threaded reply structure mattered more than chronological order. By including the framework labels as additional model features, we also assessed the links between frameworks. The empirical results provide evidence that the two frameworks measure different aspects of student behaviour relating to cognitive quality.
References
Atapattu, T., Thilakaratne, M., Vivian, R., & Falkner, K. (2019). Detecting cognitive engagement using word embeddings within an online teacher professional development community. Computers & Education. doi: 10.1016/j.compedu.2019.05.020
Barbosa, G., Camelo, R., Cavalcanti, A. P., Miranda, P., Mello, R. F., Kovanovi´c, V., & Gaševi´c, D. (2020). Towards automatic cross-language classification of cognitive presence in online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 605–614). New York, NY, USA: ACM. doi: 10.1145/3375462.3375496
Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32. doi: 10.1023/A:1010933404324
Chawla, N. V., Bowyer, K.W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16, 321–357. doi: 10.1613/jair.953
Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science, 13(2), 145–182. doi: 10.1207/s15516709cog1302 1
Chi, M. T. H., & Wylie, R. (2014). The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes. Educational Psychologist, 49(4), 219–243. doi: 10.1080/00461520.2014.965823
Corich, S., Kinshuk, & Hunt, L. M. (2006). Measuring critical thinking within discussion forums using a computerised content analysis tool. In Proceedings of the 5th International Conference on Networked Learning (pp. 1–8). Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.102.8598&rep=rep1&type=pdf
Farrow, E., Moore, J., & Gaševi´c, D. (2020). Dialogue attributes that inform depth and quality of participation in course discussion forums. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 129–134). New York, NY, USA: ACM. doi: 10.1145/3375462.3375481
Farrow, E., Moore, J., & Gaševi´c, D. (2019). Analysing discussion forum data: A replication study avoiding data contamination. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 170–179). New York, NY, USA: ACM. doi: 10.1145/3303772.3303779
Ferguson, R., Wei, Z., He, Y., & Buckingham Shum, S. (2013). An evaluation of learning analytics to identify exploratory dialogue in online discussions. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 85–93). New York, New York, USA: ACM Press. doi: 10.1145/2460296.2460313
Ferreira, M., Rolim, V., Mello, R. F., Lins, R. D., Chen, G., & Gaševi´c, D. (2020). Towards automatic content analysis of social presence in transcripts of online discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 141–150). New York, NY, USA: ACM. doi: 10.1145/3375462.3375495
Ferreira, R., Kovanovi´c, V., Gaševi´c, D., & Rolim, V. (2018). Towards Combined Network and Text Analytics of Student Discourse in Online Discussions. In Proceedings of the International Conference on Artificial Intelligence in Education (Vol. 10331, pp. 111–126). Springer International Publishing. doi: 10.1007/978-3-319-93843-1 9
Gardner, J., Brooks, C., Andres, J. M., & Baker, R. (2018). Replicating MOOC predictive models at scale. In Proceedings of the 5th Annual ACM Conference on Learning at Scale (pp. 1–10). doi: 10.1145/3231644.3231656
Garrison, D. R. (2011). E-learning in the 21st century: A framework for research and practice (2nd ed.). New York and London: Routledge. doi: 10.4324/9780203838761
Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2), 87–105. doi: 10.1016/S1096-7516(00)00016-6
Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7–23. doi: 10.1080/08923640109527071
Gaševi´c, D., Adesope, O., Joksimovi´c, S., & Kovanovi´c, V. (2015). Externally-facilitated regulation scaffolding and role assignment to develop cognitive presence in asynchronous online discussions. The Internet and Higher Education, 24, 53–65.
doi: 10.1016/j.iheduc.2014.09.006
Hu, Y., Donald, C., Giacaman, N., & Zhu, Z. (2020). Towards automated analysis of cognitive presence in MOOC discussions. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 135–140). New York, NY,
USA: ACM. doi: 10.1145/3375462.3375473
Kovanovi´c, V., Joksimovi´c, S., Gaševi´c, D., & Hatala, M. (2014). Automated cognitive presence detection in online discussion transcripts. CEUR Workshop Proceedings, 1137. Retrieved from http://ceur-ws.org/Vol-1137/LA machinelearning submission 1.pdf
Kovanovi´c, V., Joksimovi´c, S., Waters, Z., Gaševi´c, D., Kitto, K., Hatala, M., & Siemens, G. (2016). Towards Automated Content Analysis of Discussion Transcripts: A Cognitive Presence Case. In Proceedings of the Sixth International Conference
on Learning Analytics & Knowledge (pp. 15–24). New York, NY, USA: ACM. doi: 10.1145/2883851.2883950
Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159–174. doi: 10.2307/2529310
Le, Q. V., & Mikolov, T. (2014). Distributed Representations of Sentences and Documents. In Proceedings of the 31st International Conference on Machine Learning (Vol. 32, p. 9). JMLR.org. Retrieved from http://proceedings.mlr.press/v32/le14.pdf
McKlin, T. (2004). Analyzing Cognitive Presence in Online Courses Using an Artificial Neural Network (Doctoral dissertation, Georgia State University). Retrieved from http://scholarworks.gsu.edu/msit diss/1
McKlin, T., Harmon, S. W., Evans, W., & Jones, M. G. (2001). Cognitive presence in web-based learning: A content analysis of students’ online discussions. In Annual proceedings of selected research and development and practice papers presented
at the National Convention of the Association for Educational Communications and Technology (Vol. 1-2, pp. 272–277). Retrieved from https://files.eric.ed.gov/fulltext/ED470101.pdf
McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated Evaluation of Text and Discourse with Coh-Metrix. New York: Cambridge University Press. doi: 10.1017/CBO9780511894664
Neto, V., Rolim, V., Ferreira, R., Kovanovi´c, V., Gaševi´c, D., Dueire Lins, R., & Lins, R. (2018). Automated Analysis of Cognitive Presence in Online Discussions Written in Portuguese. In Proceedings of the 13th European Conference on Technology Enhanced Learning (pp. 245–261). Cham: Springer International Publishing. doi: 10.1007/978-3-319-98572-5_19
Taskin, Y., Hecking, T., Hoppe, H. U., Dimitrova, V., & Mitrovic, A. (2019). Characterizing comment types and levels of engagement in video-based learning as a basis for adaptive nudging. In Proceedings of the 14th European Conference on Technology Enhanced Learning (pp. 362–376). Cham: Springer International Publishing. doi: 10.1007/978-3-030-29736-7_27
Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54. doi: 10.1177/0261927X09351676
Wang, X., Wen, M., & Ros´e, C. P. (2016). Towards triggering higher-order thinking behaviors in MOOCs. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 398–407). New York, NY, USA: ACM. doi: 10.1145/2883851.2883964
Wang, X., Yang, D., Wen, M., Koedinger, K., & Ros´e, C. P. (2015). Investigating how student’s cognitive behavior in MOOC discussion forums affect learning gains. In Proceedings of the 8th International Conference on Educational
Data Mining (pp. 226–233). Massachusetts, USA: International Educational Data Mining Society. Retrieved from http://www.educationaldatamining.org/EDM2015/proceedings/full226-233.pdf
Waters, Z., Kovanovi´c, V., Kitto, K., & Gaševi´c, D. (2015). Structure Matters: Adoption of Structured Classification Approach in the Context of Cognitive Presence Classification. In Information Retrieval Technology (Vol. 9460, pp. 227–238). Cham:
Springer International Publishing. doi: 10.1007/978-3-319-28940-3 18
Wise, A. F., & Cui, Y. (2018). Unpacking the relationship between discussion forum participation and learning in MOOCs. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 330–339). New York, New
York, USA: ACM Press. doi: 10.1145/3170358.3170403
Wise, A. F., Cui, Y., & Vytasek, J. (2016). Bringing order to chaos in MOOC discussion forums with content-related thread identification. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 188–197). New York, New York, USA: ACM Press. doi: 10.1145/2883851.2883916
Yogev, E., Gal, K., Karger, D., Facciotti, M. T., & Igo, M. (2018). Classifying and visualizing students’ cognitive engagement in course readings. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (pp. 1–10). New York, NY, USA: ACM. doi: 10.1145/3231644.3231648
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
