Are You Being Rhetorical? A Description of Rhetorical Move Annotation Tools and Open Corpus of Sample Machine-Annotated Rhetorical Moves
DOI:
https://doi.org/10.18608/jla.2020.73.10Keywords:
Writing analytics, Corpus analysis, Rhetorical moves, Open dataAbstract
Writing analytics has emerged as a sub-field of learning analytics, with applications including the provision of formative feedback to students in developing their writing capacities. Rhetorical markers in writing have become a key feature in this feedback, with a number of tools being developed across research and teaching contexts. However, there is no shared corpus of texts annotated by these tools, nor is it clear how the tool annotations compare. Thus, resources are scarce for comparing tools for both tool development and pedagogic purposes. In this paper, we conduct such a comparison and introduce a sample corpus of texts representative of the particular genres, a subset of which has been annotated using three rhetorical analysis tools (one of which has two versions). This paper aims to provide both a description of the tools and a shared dataset in order to support extensions of existing analyses and tool design in support of writing skill development. We intend the description of these tools, which share a focus on rhetorical structures, alongside the corpus, to be a preliminary step to enable further research, with regard to both tool development and tool interaction.
References
Aït-Mokhtar, S., Chanod, J.-P., & Roux, C. (2002). Robustness beyond shallowness: Incremental deep parsing. Natural Language Engineering, 8(2–3), 121–144. https://dx.doi.org/10.1017/S1351324902002887
Allen, L. K., Jacovina, M. E., & McNamara, D. S. (2016). Computer-based writing instruction. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of Writing Research (pp. 316–329). New York, NY: Guilford Press. Retrieved from https://eric.ed.gov/?id=ED586512
Anthony, L. (1999). Writing research article introductions in software engineering: How accurate is a standard model? IEEE Transactions on Professional Communication, 42(1), 38–46. https://dx.doi.org/10.1109/47.749366
Anthony, L., & Lashkia, G. V. (2003). Mover: A machine learning tool to assist in the reading and writing of technical papers. IEEE Transactions on Professional Communication, 46(3), 185–193. https://dx.doi.org/10.1109/TPC.2003.816789
Buckingham Shum, S., Knight, S., McNamara, D., Allen, L. K., Betik, D., & Crossley, S. (2016). Critical perspectives on writing analytics. In Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ’16), 25–29 April 2016, Edinburgh, UK (pp. 481–483). New York, NY: ACM. https://dx.doi.org/10.1145/2883851.2883854
Buckingham Shum, S., Sándor, Á., Goldsmith, R., Wang, X., Bass, R., & McWilliams, M. (2016). Reflecting on reflective writing analytics: Assessment challenges and iterative evaluation of a prototype tool. Proceedings of the 6th International Conference on Learning Analytics & Knowledge, 25–29 April 2016, Edinburgh, UK (pp. 213–222). New York, NY: ACM. https://dx.doi.org/10.1145/2883851.2883955
Burstein, J., Chodorow, M., & Leacock, C. (2003). CriterionSM online essay evaluation: An application for automated evaluation of student essays. In J. Riedl & R. W. Hill Jr. (Eds.), Proceedings of the Fifteenth Conference on Innovative Applications of Artificial Intelligence (IAAI 2003), 12–14 August 2003, Acapulco, Mexico (pp. 3–10). Retrieved from https://dblp.org/db/conf/iaai/iaai2003.html
Cai, (L.) J. (2016). An exploratory study on an integrated genre-based approach for the instruction of academic lexical phrases. Journal of English for Academic Purposes, 24, 58–74. https://dx.doi.org/10.1016/J.JEAP.2016.09.002
Chang, C.-F., & Kuo, C.-H. (2011). A corpus-based approach to online materials development for writing research articles. English for Specific Purposes, 30(3), 222–234. https://dx.doi.org/10.1016/j.esp.2011.04.001
Cope, B., & Kalantzis, M. (1993). Introduction: How a genre approach to literacy can transform the way writing is taught. In B. Cope & M. Kalantzis (Eds.), The powers of literacy: A genre approach to teaching writing (pp. 1–21). London, UK: Taylor & Francis. Retrieved from https://www.taylorfrancis.com/books/9780203149812/chapters/10.4324/9780203149812-9
Cotos, E. (2011). Potential of automated writing evaluation feedback. CALICO Journal, 28(2), 420–459. https://dx.doi.org/10.11139/cj.28.2.420-459
Cotos, E. (2014). Enhancing writing pedagogy with learner corpus data. ReCALL, 26(2), 202–224. https://dx.doi.org/10.1017/S0958344014000019
Cotos, E. (2016). Computer-assisted research writing in the disciplines. In S. A. Crossley & D. S. McNamara (Eds.), Adaptive educational technologies for literacy instruction (pp. 225–242). London, UK: Taylor & Francis. Retrieved from https://www.taylorfrancis.com/books/9781315647500/chapters/10.4324/9781315647500-15
Cotos, E. (2018). Move analysis. In C. A. Chapelle (Ed.), The encyclopedia of applied linguistics (pp. 1–8). Oxford, UK: John Wiley & Sons. https://dx.doi.org/10.1002/9781405198431.wbeal1485
Cotos, E., Huffman, S., & Link, S. (2015). Furthering and applying move/step constructs: Technology-driven marshalling of Swalesian genre theory for EAP pedagogy. Journal of English for Academic Purposes, 19, 52–72. https://dx.doi.org/10.1016/j.jeap.2015.05.004
Cotos, E., Huffman, S., & Link, S. (2017). A move/step model for methods sections: Demonstrating rigour and credibility. English for Specific Purposes, 46, 90–106. https://dx.doi.org/10.1016/j.esp.2017.01.001
Cotos, E., Huffman, S., & Link, S. (2020). Understanding graduate writers’ interaction with and impact of the Research Writing Tutor during revision. Journal of Writing Research, 12(1), 187–232. https://dx.doi.org/10.17239/jowr-2020.12.01.07
Cotos, E., Link, S., & Huffman, S. R. (2016). Studying disciplinary corpora to teach the craft of discussion. Writing and Pedagogy, 87(1), 33-64. https://dx.doi.org/10.1558/wap.v8i1.27661
Cotos, E., Link, S., & Huffman, S. (2017). Effects of DDL technology on genre learning. Language Learning & Technology, 21(3), 104–130. https://dx.doi.org/10125/44623
Cotos, E., & Pendar, N. (2016). Discourse classification into rhetorical functions for AWE feedback. Calico Journal, 33(1), 92-116. https://dx.doi.org/10.1558/cj.v33i1.27047
Cotos, E., & Sándor, Á. (2018). Testing an integrated method for the automated analysis of rhetorical intent in academic genres. Paper presented at AAAL Conference 2018, 24–27 March 2018, Chicago, IL, USA.
De Liddo, A., Sándor, Á., & Buckingham Shum, S. (2012). Contested collective intelligence: Rationale, technologies, and a human-machine annotation study. Computer Supported Cooperative Work (CSCW), 21(4), 417–448. https://dx.doi.org/10.1007/s10606-011-9155-x
Elsevier Labs. (2015). OA STM Corpus. Elsevier Labs Github. Retrieved from http://elsevierlabs.github.io/OA-STM-Corpus/
Flower, L. S., & Hayes, J. R. (1977). Problem-solving strategies and the writing process. College English, 39(4), 449–461. https://dx.doi.org/10.2307/375768
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C., Knight, S., …, Jarvis, W. (2017). Reflective writing analytics for actionable feedback in authentic assessment. In Proceedings of the 7th International Conference on Learning Analytics & Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 153–162). New York, NY: ACM. https://dx.doi.org/10.1145/3027385.3027436
Hyland, K. (2007). Genre pedagogy: Language, literacy and L2 writing instruction. Journal of Second Language Writing, 16(3), 148–164. https://dx.doi.org/10.1016/j.jslw.2007.07.005
Knight, S., Abel, S., Shibani, A., Yoong Kuan, G., Conijn, R., Gibson, A., …, Buckingham Shum, S. (2020). Are you being rhetorical? An open dataset of machine annotated rhetorical moves. Stash. https://dx.doi.org/10.26195/5f336ead59a43
Knight, S., Allen, L. K., Gibson, A., McNamara, D., & Buckingham Shum, S. (2017). Writing analytics literacy—Bridging from research to practice. In Proceedings of the 7th International Conference on Learning Analytics & Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 496–497). New York, NY: ACM. https://dx.doi.org/10.1145/3027385.3029425
Knight, S., Buckingham Shum, S., Ryan, P., Sándor, Á., & Wang, X. (2018). Academic writing analytics for civil law: Participatory design through academic and student engagement. International Journal of Artificial Intelligence in Education, 28(1), 1–28. https://dx.doi.org/10.1007/s40593-016-0121-0
Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., …, Buckingham Shum, S. (2020). AcaWriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12(1), 141–186. https://dx.doi.org/10.17239/jowr-2020.12.01.06
Kuteeva, M., & Negretti, R. (2016). Graduate students’ genre knowledge and perceived disciplinary practices: Creating a research space across disciplines. English for Specific Purposes, 41, 36–49. https://dx.doi.org/10.1016/J.ESP.2015.08.004
Liakata, M., Saha, S., Dobnik, S., Batchelor, C., & Rebholz-Schuhmann, D. (2012). Automatic recognition of conceptualization zones in scientific articles and two life science applications. Bioinformatics, 28(7), 991–1000. https://dx.doi.org/10.1093/bioinformatics/bts071
Lisacek, F., Chichester, C., Kaplan, A., & Sándor, Á. (2005). Discovering paradigm shift patterns in biomedical abstracts: Application to neurodegenerative diseases. In Proceedings of the First International Symposium on Semantic Mining in Biomedicine (SMBM), 11–13 April 2005, Cambridge, UK (pp. 41–50). Retrieved from https://www.researchgate.net/publication/228635443_Discovering_Paradigm_Shift_Patterns_in_Biomedical_Abstracts_Application_to_Neurodegenerative_Diseases
Mann, W. C., & Thompson, S. A. (1987). Rhetorical structure theory: Description and construction of text structures. In G. Kempen (Ed.), Natural language generation (Vol. 135) (pp. 85–95). Dordrecht, The Netherlands: Springer. https://dx.doi.org/10.1007/978-94-009-3645-4_7
McDonald, J., Moskal, A. C. M., Gunn, C., & Donald, C. (2018). Text analytic tools to illuminate student learning. In J. M. Lodge, J. C. Horvath, & L. Corrin (Eds.), Learning analytics in the classroom: Translating learning analytics for teachers. London, UK: Taylor & Francis. https://dx.doi.org/10.4324/9781351113038-11
Mizumoto, A., Hamatani, S., & Imao, Y. (2017). Applying the bundle–move connection approach to the development of an online writing support tool for research articles. Language Learning, 67(4), 885–921. https://dx.doi.org/10.1111/lang.12250
National Commission On Writing. (2003). Report of the National Commission on writing in America’s schools and colleges: The neglected “R,” the need for a writing revolution. The College Board. Retrieved from https://archive.nwp.org/cs/public/print/resource/2523
Nesi, H., Gardner, S., Thompson, P., & Wickens, P. (2004). British academic written English corpus. Oxford Text Archive. Oxford, UK: University of Oxford. Retrieved from http://purl.ox.ac.uk/ota/2539
PMC. (n.d.). Open Access Subset. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/
Sándor, Á. (2007). Modeling metadiscourse conveying the author’s rhetorical strategy in biomedical research abstracts. Revue française de linguistique appliquée, 12(2), 97–108. https://dx.doi.org/10.3917/rfla.122.0097
Sándor, Á., Kaplan, A., & Rondeau, G. (2006). Discourse and citation analysis with concept-matching. In Proceedings of the International Symposium: Discourse and Document (ISDD), 15–16 June 2006, Caen, France (pp. 147–152). Retrieved from https://www.researchgate.net/publication/240828357_Discourse_and_citation_analysis_with_concept-matching
Sándor, Á., & Vorndran, A. (2009). Detecting key sentences for automatic assistance in peer reviewing research articles in educational sciences. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries, 7 August 2009, Singapore (pp. 36–44). Stroudsburg, PA, USA: Association for Computational Linguistics. https://dx.doi.org/10.3115/1699750.1699757
Scardamalia, M., & Bereiter, C. (1987). Knowledge telling and knowledge transforming in written composition. Advances in Applied Psycholinguistics, 2, 142–175.
Shermis, M. D., & Burstein, J. (Eds.). (2013). Handbook of automated essay evaluation: Current applications and new directions. London, UK: Routledge/Taylor & Francis. Retrieved from https://psycnet.apa.org/record/2013-15323-000
Shibani, A. (2018a). AWA-Tutor: A platform to ground automated writing feedback in robust learning design. In S. Buckingham Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Companion Proceedings of the 8th International Learning Analytics and Knowledge Conference (LAK ’18), 5–9 March 2018, Sydney, Australia. Society for Learning Analytics Research (SoLAR). Retrieved from http://bit.ly/lak18-companion-proceedings
Shibani, A. (2018b). Developing a learning analytics intervention design and tool for writing instruction. In S. Buckingham Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Companion Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK ’18), 5–9 March 2018, Sydney, Australia. Society for Learning Analytics Research (SoLAR). Retrieved from http://bit.ly/lak18-companion-proceedings
Shibani, A., Abel, S., Gibson, A., & Knight, S. (2018). Turning the TAP on writing analytics. In S. Buckingham Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Companion Proceedings of the 8th International Conference on Learning Analytics & Knowledge (LAK ’18), 5–9 March 2018, Sydney, Australia. Society for Learning Analytics Research (SoLAR). Retrieved from http://bit.ly/lak18-companion-proceedings
Shibani, A., Knight, S., & Buckingham Shum, S. (2019). Contextualizable learning analytics design: A generic model and writing analytics evaluations. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2018, Tempe, AZ, USA (pp. 210–219). New York, NY: ACM. https://dx.doi.org/10.1145/3303772.3303785
Shibani, A., Knight, S., & Buckingham Shum, S. (2020). Educator perspectives on learning analytics in classroom practice. Internet and Higher Education, 46, 100730. https://dx.doi.org/10.1016/j.iheduc.2020.100730
Shibani, A., Knight, S., Buckingham Shum, S., & Ryan, P. (2017). Design and implementation of a pedagogic intervention using writing analytics. In W. Chen, J.-C. Yang, A. F. Mohd Ayub, S. L. Wong, & A. Mitrovic (Eds.), Proceedings of the 25th International Conference on Computers in Education, 4–8 December 2017, Christchurch, New Zealand (pp. 306–315). Asia-Pacific Society for Computers in Education. Retrieved from https://www.apsce.net/icce/icce2017/140.115.135.84/icce/icce2017/proceedings_main.html
Simsek, D., Sándor, Á., Buckingham Shum, S., Ferguson, R., De Liddo, A., & Whitelock, D. (2015). Correlations between automated rhetorical analysis and tutors’ grades on student essays. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ’15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 355–359). New York, NY: ACM. https://dx.doi.org/10.1145/2723576.2723603
Swales, J. (1981). Aspects of article introductions (Aston ESP reports No. 1). Birmingham, UK: The University of Aston in Birmingham.
Swales, J. (1990). Genre analysis: English in academic and research settings. Cambridge, UK: Cambridge University Press.
Teufel, S., & Moens, M. (2002). Summarizing scientific articles: Experiments with relevance and rhetorical status. Computational Linguistics, 28(4), 409–445. https://dx.doi.org/10.1162/089120102762671936
Warschauer, M., & Grimes, D. (2008). Automated writing assessment in the classroom. Pedagogies: An International Journal, 3(1), 22–36. https://dx.doi.org/10.1080/15544800701771580
Wingate, U. (2012). Using Academic Literacies and genre-based models for academic writing instruction: A “literacy” journey. Journal of English for Academic Purposes, 11(1), 26–37. https://dx.doi.org/10.1016/j.jeap.2011.11.006
Yan, D., Rupp, A. A., & Foltz, P. W. (Eds.). (2020). Handbook of automated scoring: Theory into practice. London, UK: Chapman and Hall/CRC. Retrieved from https://www.routledge.com/Handbook-of-Automated-Scoring-Theory-into-Practice/Yan-Rupp-Foltz/p/book/9781138578272
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2020 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
