Towards Reflective Writing Analytics: Rationale, Methodology and Preliminary Results
DOI:
https://doi.org/10.18608/jla.2017.41.5Keywords:
learning analytics, education, writing analytics, reflection, natural language processing, reflective move, rhetoricAbstract
When used effectively, reflective writing tasks can deepen learners’ understanding of key concepts, help them critically appraise their developing professional identity, and build qualities for lifelong learning. As such, reflecting writing is attracting substantial interest from universities concerned with experiential learning, reflective practice, and developing a holistic conception of the learner. However, reflective writing is for many students a novel genre to compose in, and tutors may be inexperienced in its assessment. While these conditions set a challenging context for automated solutions, natural language processing may also help address the challenge of providing real time, formative feedback on draft writing. This paper reports progress in designing a writing analytics application, detailing the methodology by which informally expressed rubrics are modelled as formal rhetorical patterns, a capability delivered by a novel web application. Preliminary tests on an independently human-annotated corpus are encouraging, showing improvements from the first to second version, but with much scope for improvement. We discuss a range of issues: the prevalence of false positives in the tests, areas for futures technical improvements, the risks of gaming the system, and the participatory design process that has enabled work across disciplinary boundaries to develop the prototype to its current state.
References
Aït-Mokhtar, S., Chanod, J-P., & Roux, C. (2002). Robustness beyond shallowness: Incremental deep parsing. Natural Language Engineering, 8(2/3), 121–144. http://dx.doi.org/10.1017/S1351324902002887
Bass, R., & Eynon, B. (2016). Open and integrative: Designing liberal education for the new digital ecosystem. Association of American Colleges and Universities. https://www.aacu.org/publications-research/publications/open-and-integrative-designing-liberal-education-new-digital
Boud, D., Keogh, R., & Walker, D. (1985). Reflection: Turning experience into learning. London: Routledge, Abingdon, Oxon.
Boud, D., & Walker, D. (1998). Promoting reflection in professional courses: The challenge of context. Studies in Higher Education, 23(2), 191–206. http://dx.doi.org/10.1080/03075079812331380384
Brun, C., & Hagège, C. (2003). Normalization and paraphrasing using symbolic methods. Proceedings of the 2nd International Workshop on Paraphrasing (PARAPHRASE ’03), Vol. 16, 11 July 2003, Sapporo, Japan (pp. 41–48). Stroudsburg, PA: Association for Computational Linguistics. http://dx.doi.org/10.3115/1118984.1118990
Brun, C., & Hagège, C. (2004). Intertwining deep syntactic processing and named entity detection. Advances in natural language processing (pp. 195–206). Berlin: Springer. http://dx.doi.org/10.1007/978-3-540-30228-5_18
Brun, C., Popa, D. N., & Roux, C. (2014). XRCE: Hybrid classification for aspect-based sentiment analysis. Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), 23–24 August 2014, Dublin, Ireland (pp. 838–842). Association for Computational Linguistics. http://www.aclweb.org/anthology/S14-2149
Buckingham Shum, S., & Deakin Crick, R. (2016). Learning analytics for 21st century competencies. Journal of Learning Analytics, 3(2), 6–21. http://dx.doi.org/10.18608/jla.2016.32.2
Buckingham Shum, S., Sándor, Á, Goldsmith, R., Wang, X., Bass, R., & McWilliams, M. (2016). Reflecting on reflective writing analytics: Assessment challenges and iterative evaluation of a prototype tool. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 213–222). New York: ACM. http://dx.doi.org/10.1145/2883851.2883955
Burnett, M. M., & Scaffidi, C. (2011). End-user development. In M. Soegaard & R. F. Dam (Eds.), The encyclopedia of human–computer interaction, 2nd ed. Interaction Design Foundation. https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed
De Liddo, A., Sándor. Á., & Buckingham Shum, S. (2012). Contested collective intelligence: Rationale, technologies, and a human–machine annotation study. Computer Supported Cooperative Work, 21(4–5), 417–448. http://dx.doi.org/10.1007/s10606-011-9155-x
Gibson, A., & Kitto, K. (2015). Analysing reflective text for learning analytics: An approach using anomaly recontextualisation. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 275–279). New York: ACM. http://dx.doi.org/10.1145/2723576.2723635
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C., & Knight, S. (2017). Reflective writing analytics for actionable feedback. Proceedings of the 7th International Conference on Learning Analytics and Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 153–162). New York: ACM. http://dx.doi.org/10.1145/3027385.3027436
Gibson, A., Kitto, K., & Bruza, P. (2016). Towards the discovery of learner metacognition from reflective writing. Journal of Learning Analytics, 3(2), 22–36. http://dx.doi.org/10.18608/jla.2016.32.3
Hatton N., & Smith, D. (1995, January). Reflection in teacher education: Towards definition and implementation. Teaching & Teacher Education, 11(1), 33–49. http://dx.doi.org/10.1016/0742-051X(94)00012-U
Huang, Z., ten Teije, A., van Harmelen, F., & Aït-Mokhtar, S. (2014). Semantic representation of evidence-based clinical guidelines. Proceedings of the 6th International Workshop on Knowledge Representation for Health Care (KR4HC 2014), 21 July 2014, Vienna, Austria (pp. 78–94). http://dx.doi.org/10.1007/978-3-319-13281-5_6
Hulsman, R. L., & van der Vloodt, J. (2015). Self-evaluation and peer-feedback of medical students’ communication skills using a web-based video annotation system. Exploring content and specificity. Patient Education and Counseling, 98(3), 356–363. http://dx.doi.org/10.1016/j.pec.2014.11.007
Knight, S., Buckingham Shum, S., Ryan, P., Sándor, Á., & Wang, X. (in press). Designing academic writing analytics for civil law student self-assessment. International Journal of Artificial Intelligence in Education (Special Issue on Multidisciplinary Approaches to Reading and Writing Integrated with Disciplinary Education, Eds. D. McNamara, S. Muresan, R. Passonneau & D. Perin). http://dx.doi.org/10.1007/s40593-016-0121-0
Knight, S., Martinez-Maldonado, R., Gibson, A., & Buckingham-Shum, S. (2017). Towards mining sequences and dispersion of rhetorical moves in student written texts. Proceedings of the 7th International Conference on Learning Analytics and Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 228–232). New York: ACM. http://dx.doi.org/10.1145/3027385.3027433
Lisacek, F., Chichester, C., Kaplan, A., & Sandor, Á. (2005). Discovering paradigm shift patterns in biomedical abstracts: Application to neurodegenerative diseases. Proceedings of the 1st International Symposium on Semantic Mining in Biomedicine (SMBM), 11–13 April 2005, Cambridge, United Kingdom (pp. 41–50). http://www.xrce.xerox.com/Research-Development/Publications/2005-006
MacLean, M., Carter, K., Lövstrand, L., & Moran, T. (1990). User-tailorable systems: Pressing the issues with buttons. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’90), 1–5 April 1990, Seattle, WA, USA (pp. 175–182). New York: ACM. http://dx.doi.org/10.1145/97243.97271
Moon, J. (2004). A handbook of reflective and experiential learning: Theory and practice. London: RoutledgeFalmer.
Moon, J. (2010). Reflective learning workshop (Handout 10/07), University of Worcester, UK. http://worc.ac.uk/edu/documents/Jenny_Moon_RefLearnlong07.doc
Reidsema, C., Goldsmith, R., & Mort, P. (2010). Writing to learn: Reflective practice in engineering design. Proceedings of the 9th Annual ASEE Global Colloquium (ASEE 2010), 18–21 October 2010, Singapore. American Society for Engineering Education. https://www.academia.edu/501660/Writing_to_Learn_Reflective_Practice_in_Engineering_Design
Rodgers, C. (2002a). Defining reflection: Another look at John Dewey and reflective thinking. Teachers College Record, 104(4), 842–866. http://www.tcrecord.org/content.asp?contentid=10890
Rodgers, C. (2002b). Voices inside schools. Harvard Educational Review, 72(2), 230–254.
Ryan, M. (2010). The 4 R’s model of reflective thinking, version 1.5. Developing Reflective Approaches to Writing (DRAW) Project, Queensland University of Technology. http://www.citewrite.qut.edu.au/write/4Rs-for-students-page1-v1.5.pdf
Ryan, M. (2011). Improving reflective writing in higher education: A social semiotic perspective, Teaching in Higher Education, 16(1), 99–111. Http://dx.doi.org/10.1080/13562517.2010.507311
Ryan, M., & Ryan, M. (2012). Developing a systematic, cross-faculty approach to teaching and assessing reflection in higher education. Australian Government, Office of Learning and Teaching. http://www.olt.gov.au/system/files/resources/PP9_1327_Ryan_report_2012.pdf
Sándor, Á. (2007). Modeling metadiscourse conveying the author’s rhetorical strategy in biomedical research abstracts. Revue Française de Linguistique Appliquée, 12(2), 97–108. http://www.xrce.xerox.com/Research-Development/Publications/2007-029
Sándor, Á., & Vorndran, A. (2009). Detecting key sentences for automatic assistance in peer reviewing research articles in educational sciences. Proceedings of the Workshop on Text and Citation Analysis for Scholarly Digital Libraries, 47th Annual Meeting of the Association for Computational Linguistics, 2–7 August 2009, Singapore. http://www.xrce.xerox.com/Research-Development/Publications/2009-039
Scouller, K. (1998). The influence of assessment methods on students’ learning approaches: Multiple choice question examination versus assignment essay. Higher Education, 35(4), 453–472. http://dx.doi.org/10.1023/A:1003196224280
Simsek, D., Buckingham Shum, S., Sándor, Á., De Liddo, A., & Ferguson, R. (2013). XIP dashboard: Visual analytics from automated rhetorical parsing of scientific metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics (DCLA13), 8 April 2013, Leuven, Belgium. http://oro.open.ac.uk/37391
Simsek, D., Sándor, Á., Buckingham Shum, S., Ferguson, R., De Liddo, A., & Whitelock, D. (2015). Correlations between automated rhetorical analysis and tutors’ grades on student essays. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 355–359). New York: ACM. http://dx.doi.org/10.1145/2723576.2723603
Sumsion, J., & Fleet, A. (1996). Reflection: Can we assess it? Should we assess it? Assessment & Evaluation in Higher Education, 21(2), 121–130. http://dx.doi.org/10.1080/0260293960210202
Ullmann, T. D., Wild, F., & Scott, P. (2012). Comparing automatically detected reflective texts with human judgements. Proceedings of the 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR-TEL ʼ12), 18 September 2013, Saarbrucken, Germany (pp. 101–116). http://ceur-ws.org/Vol-931/paper8.pdf
Webster-Wright, A. (2013). The eye of the storm: A mindful inquiry into reflective practices in higher education. Reflective Practice, 14(4), 556–567. http://dx.doi.org/10.1080/14623943.2013.810618
Downloads
Published
How to Cite
Issue
Section
License
TEST
