Automating the Detection of Reflection-on-Action
DOI:
https://doi.org/10.18608/jla.2017.42.15Keywords:
student reflection, reflection-on-action, epistemic frame theory, virtual internshipAbstract
Learning to solve complex problems — problems whose solutions require the application of more than basic facts and skills — is critical to meaningful participation in the economic, social, and cultural life of the digital age. In this paper, we use a theoretical understanding of how professionals use reflection-in-action to solve complex problems to investigate how students learn this critical 21st-century skill and how we can develop and automate learning analytic techniques to assess that learning. We present a preliminary study examining the automated detection of reflective discourse during collaborative, complex problem solving. We analyze student reflection-on-action in a virtual learning environment, focusing on both reflection in individual discourse and collaborative reflection among students. Our results suggest that it is possible to detect student reflection on complex problems in virtual learning environments, but that different models may be appropriate depending on students’ prior domain experience.
References
Andrzejewski, D., Zhu, X., & Craven, M. (2009). Incorporating domain knowledge into topic modeling via Dirichlet forest priors. Proceedings of the 26th Annual International Conference on Machine Learning (ICML ’09), 14–18 June 2009, Montreal, Quebec, Canada (pp. 25–32). New York: ACM. http://dx.doi.org/10.1145/1553374.1553378
Arastoopour, G., Chesler, N. C., & Shaffer, D. W. (2014). Epistemic persistence: A simulation-based approach to increasing participation of women in engineering. Journal of Women and Minorities in Science and Engineering, 20(3), 211–234.
http://dx.doi.org/10.1615/JWomenMinorScienEng.2014007317
Arastoopour, G., & Shaffer, D. W. (2015). Epistemography and professional CSCL environment design. Exploring the material conditions of learning: The Computer Supported Collaborative Learning (CSCL) Conference 2015, 7–11 June 2015, Gothenberg, Sweden (Vol. 1, pp. 2014–211). International Society of the Learning Sciences.
Arts, J. A. R., Gijselaers, W. H., & Boshuizen, H. P. A. (2006). Understanding managerial problem-solving, knowledge use and information processing: Investigating stages from school to the workplace. Contemporary Educational Psychology, 31(4), 387–410. https://doi.org/10.1016/j.cedpsych.2006.05.005
Autor, D. H., Levy, F., & Murnane, R. J. (2003a). Computer-based technological change and skill demands: Reconciling the perspectives of economists and sociologists. In E. Applebaum, A. D. Bernhardt, & R. J. Murnane, Low-wage America: How employers are reshaping opportunity in the workplace (pp. 121–154). New York: Russell Sage Foundation.
Autor, D. H., Levy, F., & Murnane, R. J. (2003b). The skill content of recent technological change: An empirical exploration. Quarterly Journal of Economics, 118(4), 1279–1333. https://doi.org/10.1162/003355303322552801
Bagley, E. (2010). The epistemography of an urban and regional planning practicum: Appropriation in the face of resistance. WCER Working Paper 2010-8. Wisconsin Center for Education Research, University of Wisconsin–Madison.
Blei, D. M., & Lafferty, J. D. (2009). Topic models. In A. N. Srivastava & M. Sahami (Eds.), Text mining: Classification, clustering, and applications (pp. 71–94). Boca Raton, FL: CRC Press.
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42. https://doi.org/10.3102/0013189X018001032
Campione, J. C., Shapiro, A. M., & Brown, A. L. (1995). Forms of transfer in a community of learners: Flexible learning and understanding. In A. McKeough, J. Lupart, & A. Marini (Eds.), Teaching for transfer: Fostering generalization in learning (pp. 35–69). Mahwah, NJ: Erlbaum
Califf, M. E., & Mooney, R. J. (2003). Bottom-up relational learning of pattern matching rules for information extraction. The Journal of Machine Learning Research, 4, 177–210.
Chesler, N. C., Ruis, A. R., Collier, W., Swiecki, Z., Arastoopour, G., & Shaffer, D. W. (2015). A novel paradigm for engineering education: Virtual internships with individualized mentoring and assessment of engineering thinking. Journal of Biomechanical Engineering, 137(2), 024701:1-8. http://dx.doi.org/10.1115/1.4029235
Coutinho, S., Wiemer-Hastings, K., Skowronski, J. J., & Britt, M. A. (2005). Metacognition, need for cognition and use of explanations during ongoing learning and problem solving. Learning and Individual Differences, 15(4), 321–337. https://doi.org/10.1016/j.lindif.2005.06.001
Desautel, D. (2009). Becoming a thinking thinker: Metacognition, self-reflection, and classroom practice. The Teachers College Record, 111(8), 1997–2020.
DiSessa, A. A. (1988). Knowledge in pieces. In G. Forman & P. Pufall (Eds.), Constructivism in the computer age (pp. 47–70). Hillsdale, NJ: Erlbaum.
Dodman, D., McGranahan, G., & Dalal-Clayton, B. (2013). Integrating the environment in urban planning and management: Key principles and approaches for cities in the 21st century. Geneva, Switzerland: United Nations Environmental Programme.
Dorogovtsev, S. N., & Mendes, J. F. (2003). Evolution of networks: From biological nets to the Internet and WWW. Oxford, UK: Oxford University Press.
Dumais, S., Furnas, G., Landauer, T., Deerwester, S., & Harshman, R. (1988). Using latent semantic analysis to improve access to textual information. In J. J. O’Hare (Ed.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ98), 4–9 April 2009, Boston, MA, USA (pp. 281–285). Washington, DC: Human Factors in Computing Systems. http://dx.doi.org/10.1145/57167.57214
Eraut, M. (1994). Developing professional knowledge and competence. London: Falmer.
Fogarty, R. (1994). The mindful school: How to teach for metacognitive reflection. Palatine, IL: Skylight Publishing.
Friedman, T. L. (2006). The world is flat [updated and expanded]: A brief history of the twenty-first century. New York: Macmillan.
Gama, C. (2004). Metacognition in interactive learning environments: The reflection assistant model. In J. C. Lester, R. M. Vicari, & F. Paraguaçu (Eds.), Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS 2004), 30 August–3 September 2004, Maceió, Alagoas, Brazil (pp. 668–677). Springer: Berlin Heidelberg. http://dx.doi.org/10.1007/b100137
Gibson, A., & Kitto, K. (2015). Analysing reflective text for learning analytics: An approach using anomaly recontextualisation. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 275–279). New York: ACM. http://dx.doi.org/10.1145/2723576.2723635
Goodwin, C. (1994). Professional vision. American Anthropologist, 96(3), 606–633.
Graesser, A. C., Foltz, P. W., Rosen, Y., Shaffer, D. W., Forsyth, C., & Germany, M. (in press). Challenges of assessing collaborative problem solving. In E. Care, P. Griffin, & M. Wilson (Eds.), Assessment and Teaching of 21st Century Skills. Springer.
Grant, A. M. (2001). Rethinking psychological mindedness: Metacognition, self-reflection, and insight. Behaviour Change, 18(1), 8–17.
Hacker, D. J., Dunlosky, J., & Graesser, A. C. (Eds.). (2009). Handbook of metacognition in education. Mahwah, NJ: Erlbaum/Taylor & Francis.
Hatfield, D. L., & Shaffer, D. W. (2010). The epistemography of journalism 335: Complexity in developing journalistic expertise. Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010), 29 June–2 July 2010, Chicago, IL, USA (Vol. 1, pp. 628–635). International Society of the Learning Sciences.
Hatfield, D., Shaffer, D. W., Bagley, E. S., Nulty, A., & Nash, P. (2008). Reflection in professional play. Proceedings of the 8th International Conference of the Learning Sciences (ICLS ’08), 24–28 June 2008, Utrecht, Netherlands (Vol. 3, pp. 245–252). International Society of the Learning Sciences.
Howard, D. V. (1983). Cognitive psychology: Memory, language, and thought. New York: Macmillan.
i Cancho, R. F., & Solé, R. V. (2001). The small world of human language. Proceedings of the Royal Society of London. Series B: Biological Sciences, 268(1482), 2261–2265. http://dx.doi.org/10.1098/rspb.2001.1800
Kim, Y. R., Park, M. S., Moore, T. J., & Varma, S. (2013). Multiple levels of metacognition and their elicitation through complex problem-solving tasks. The Journal of Mathematical Behavior, 32(3), 377–396. https://doi.org/10.1016/j.jmathb.2013.04.002
Kitchener, K. S. (1983). Cognition, metacognition, and epistemic cognition: A three level model of cognitive processing. Human Development, 26, 222–232. https://doi.org/10.1159/000272885
Landauer, T. K., McNamara, D. S., Dennis, S., & Kintsch, W. (2007). Handbook of latent semantic analysis. Mahwah, NJ: Erlbaum.
Lave, J. (1988). The culture of acquisition and the practice of understanding. (IRL report 88-00087). Palo Alto, CA: Institute for Research on Learning.
Levy, F., & Murnane, R. J. (2004). The new division of labor. Princeton, NJ: Princeton University Press.
Linn, M. C., Eylon, B.-S., & Davis, E. A. (2004). The knowledge integration perspective on learning. In M. C. Linn, E. A. Davis, & P. Bell (Eds.), Internet environments for science education (pp. 29–46). Mahwah, NJ: Erlbaum.
Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2), 203–208. http://dx.doi.org/10.3758/BF03204766
McAleese, R. (1998). The knowledge arena as an extension to the concept map: Reflection in action. Interactive Learning Environments, 6(3), 251–272.
Madani, A., Vassiliou, M. C., Watanabe, Y., Al-Halabi, B., Al-Rowais, M. S., Deckelbaum, D. L., … Feldman, L. S. (2017). What are the principles that guide behaviors in the operating room? Creating a framework to define and measure performance. Annals of Surgery, 265(2), 255–267. http://dx.doi.org/10.1097/SLA.0000000000001962
Miller, T. M., & Geraci, L. (2011). Training metacognition in the classroom: The influence of incentives and feedback on exam predictions. Metacognition and Learning, 6(3), 303–314. http://dx.doi.org/10.1007/s11409-011-9083-7
Murnane, R. J., & Levy, F. (1993). Why today’s high-school-educated males earn less than their fathers did: The problem and an assessment of responses. Harvard Educational Review, 63(1), 1–20. https://doi.org/10.17763/haer.63.1.7585v420548725x0
Nash, P., & Shaffer, D. W. (2013). Epistemic trajectories: Mentoring in a game design practicum. Instructional Science, 41(4), 745–771. http:/dx.doi.org/10.1007/s11251-012-9255-0
Newman, M. E. (2004). Analysis of weighted networks. Physical Review E, 70(5), 056131. https://doi.org/10.1103/PhysRevE.70.056131
Novak, J. D., & Cañas, A. J. (2006). The theory underlying concept maps and how to construct them. Pensacola, FL: Florida Institute for Human and Machine Cognition.
Palinscar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 1(2), 117–175. http://dx.doi.org/10.1207/s1532690xci0102_1
Powell, W. W., & Snellman, K. (2004). The knowledge economy. Annual Review of Sociology, 199–220.
Ruckelshaus, C., & Leberstein, S. (2014). Manufacturing low pay: Declining wages in the jobs that built America’s middle class. http://www.nelp.org/page/-/Justice/2014/Manufacturing-Low-Pay-Declining-Wages-Jobs-Built-Middle-Class.pdf?nocdn=1
Rupp, A. A., Gushta, M., Mislevy, R. J., & Shaffer, D. W. (2010). Evidence-centered design of epistemic games: Measurement principles for complex learning environments. The Journal of Technology, Learning and Assessment, 8(4). https://ejournals.bc.edu/ojs/index.php/jtla/article/view/1623
Schön, D. A. (1983). The reflective practitioner: How professionals think in action. London: Temple Smith.
Schön, D. A. (1984). The architectural studio as an exemplar of education for reflection-in-action. Journal of Architectural Education, 38(1), 2–9. http://dx.doi.org/10.1080/10464883.1984.10758345
Schön, D. A. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. San Francisco, CA: Jossey-Bass.
Schön, D. A. (1992). The crisis of professional knowledge and the pursuit of an epistemology of practice. Journal of Interprofessional Care, 6(1), 49–63. http://dx.doi.org/10.3109/13561829209049595
Schön, D. A. (1995). Knowing-in-action: The new scholarship requires a new epistemology. Change, November/December, 27–34. http://dx.doi.org/10.1080/00091383.1995.10544673
Shaffer, D. W. (2006). Epistemic frames for epistemic games. Computers and Education, 46(3), 223–234. http://dx.doi.org/10.1016/j.compedu.2005.11.003
Shaffer, D. W. (2012). Models of situated action: Computer games and the problem of transfer. In C. Steinkuehler, K. Squire, & S. Barab (Eds.), Games, learning, and society: Learning and meaning in the digital age (pp. 403–433). Cambridge, UK: Cambridge University Press
Shaffer, D. W., Hatfield, D., Svarovsky, G., Nash, P. Nulty, A., Bagley, E., Franke, K., Rupp, A. A., & Mislevy, J. R. (2009). Epistemic network analysis: A prototype for 21st century assessment of learning. The International Journal of Learning and Media, 1(2), 33–53. http://dx.doi.org/10.1162/ijlm.2009.0013
Shaffer, D. W., Rogers, B., Eagan, B. R., & Marquart, C. (2016). rhoR: Rho for inter-rater reliability [Computer software]. https://CRAN.R-project.org/package=rhoR.
Siebert-Evenstone, A. L., Arastoopour, G., Collier, W., Swiecki, Z., Ruis, A. R., & Shaffer, D. W. (2016). In search of conversational grain size: Modeling semantic structure using moving stanza windows. In C.‑K. Looi, J. Polman, U. Cress, & P. Reimann (Eds.), Transforming learning, empowering learners: Proceedings of the 12th International Conference of the Learning Sciences (ICLS ’16), 20–24 June 2016, Singapore (Vol. 1, pp. 631–638).
Southavilay, V., Yacef, K., Reimann, P., & Calvo, R. A. (2013). Analysis of collaborative writing processes using revision maps and probabilistic topic models. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 38–47). New York: ACM. http://dx.doi.org/10.1145/2460296.2460307
Spector, J. M., Merrill, M. D., Elen, J., & Bishop, M. J. (2013). Handbook of research on educational communications and technology. Springer Science & Business.
Svarovsky, G. N., & Shaffer, D. W. (2006). Design meetings and design notebooks as tools for reflection in the engineering design course. Proceedings of the 36th Annual Frontiers in Education Conference (FIE 2006), 27 October–1 November 2006, San Diego, CA, USA (pp. 7–12). IEEE.
Tang, J., Meng, Z., Nguyen, X., Mei, Q., & Zhang, M. (2014). Understanding the limiting factors of topic modeling via posterior contraction analysis. Proceedings of the 31st International Conference on Machine Learning, PMLR, 32(1), 190–198.
Ullmann, T. D., Wild, F., & Scott, P. (2012). Comparing automatically detected reflective texts with human judgements. Proceedings of the 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR-TEL ʼ12), 18 September 2013, Saarbrucken, Germany (pp. 101–116).
Usher, R., & Bryant, I. (1997). Adult education and the postmodern challenge. London: Routledge.
Velardi, P., Fabriani, P., & Missikoff, M. (2001). Using text processing techniques to automatically enrich a domain ontology. Proceedings of the International Conference on Formal Ontology in Information Systems (FOIS ’01), 17–19 October 2001, Ogunquit, Maine, USA (pp. 270–284). New York: ACM. http://dx.doi.org/10.1145/505168.505194
Veenman, M. V., Van Hout-Wolters, B. H., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1(1), 3–14. http://dx.doi.org/10.1007/s11409-006-6893-0
Weeber, M., Klein, H., de Jong‐van den Berg, L., & Vos, R. (2001). Using concepts in literature‐based discovery: Simulating Swanson’s Raynaud–fish oil and migraine–magnesium discoveries. Journal of the American Society for Information Science and Technology, 52(7), 548–557. http://dx.doi.org/10.1002/asi.1104
White, S., & Feiner, S. (2009). SiteLens: Situated visualization techniques for urban site visits. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ09), 4–9 April 2009, Boston, MA, USA (pp. 1117–1120). New York: ACM. http://dx.doi.org/10.1145/1518701.1518871
Wood, D., Bruner, J., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Child Psychiatry, 17, 89−100. http://dx.doi.org/10.1111/j.1469-7610.1976.tb00381.x
Wood, P. K. (1983). Inquiring systems and problem structures: Implications for cognitive development. Human Development, 26, 249–265. http://dx.doi.org/10.1159/000137808
Xun, G. E., & Land, S. M. (2004). A conceptual framework for scaffolding III-structured problem-solving processes using question prompts and peer interactions. Educational Technology Research and Development, 52(2), 5–22. http://dx.doi.org/10.1007/BF02504836
Published
How to Cite
Issue
Section
License
Copyright (c) 2017 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
