Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity
DOI:
https://doi.org/10.18608/jla.2019.62.3Keywords:
real-time analytics, co-design, cognitive augmentation, orchestration, awareness, design methods, prototyping methods, k-12, classrooms, needfinding, wearables, AI, adaptive learning technologies, intelligent tutoring systems, automation, augmentation, teachers, participatory designAbstract
Involving stakeholders throughout the creation of new educational technologies can help ensure their usefulness and usability in real-world contexts. However, given the complexity of learning analytics (LA) systems, it can be challenging to meaningfully involve non-technical stakeholders throughout their design and development. This article reports on the iterative co-design, development, and classroom evaluation of Konscia, a wearable, real-time awareness tool for teachers working in AI-enhanced K-12 classrooms. In the process, we argue that the co-design of LA systems requires new kinds of prototyping methods. We introduce one of our own prototyping methods, REs, to address unique challenges of co-prototyping LA tools. This work presents the first end-to-end demonstration of how non-technical stakeholders can participate throughout the whole design process for a complex LA system—from early generative phases to the selection and tuning of analytics to evaluation in real-world contexts. We conclude by providing methodological recommendations for future LA co-design efforts.
References
Aguilar, S. J. (2018). Examining the relationship between comparative and self-focused academic data visualizations in at-risk college students’ academic motivation. Journal of Research on Technology in Education, 50(1), 84–103. http://dx.doi.org/10.1080/15391523.2017.1401498
Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2016). Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 26(1), 205–223. http://dx.doi.org/10.1007/s40593-015-0089-1
Baker, R. S. (2016). Stupid tutoring systems, intelligent humans. International Journal of Artificial Intelligence in Education, 26(2), 600–614. http://dx.doi.org/10.1007/s40593-016-0105-0
Baumer, E. P. (2017). Toward human-centered algorithm design. Big Data & Society, 4(2). http://dx.doi.org/10.1177/2053951717718854
Beck, J. E., & Gong, Y. (2013). Wheel-spinning: Students who fail to master a skill. In H. C. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Proceedings of the 16th International Conference on Artificial Intelligence in Education (AIED ʼ13), 9–13 July 2013, Memphis, TN, USA. (pp. 431–440). Springer, Berlin, Heidelberg. http://dx.doi.org/10.1007/978-3-642-39112-5_44
Beyer, H., & Holtzblatt, K. (1997). Contextual design: Defining customer-centered systems. Amsterdam, Netherlands: Elsevier.
Black, P., & Harrison, C. (2001). Feedback in questioning and marking: The science teacher’s role in formative assessment. School Science Review, 82(301), 55–61.
Bonsignore, E., DiSalvo, B., DiSalvo, C., & Yip, J. (2017). Introduction to participatory design in the learning sciences. In B. DiSalvo, J. Yip, E. Bonsignore, & C. DiSalvo (Eds.), Participatory Design for Learning (pp. 15–18). Abingdon-on-Thames, UK: Routledge.
Buchenau, M., & Suri, J. F. (2000). Experience prototyping. In Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (DIS ’00), 17–19 August 2000, New York City, NY, USA (pp. 424–433). New York: ACM. http://dx.doi.org10.1145/347642.347802
Bull, S., & Kay, J. (2016). SMILI☺: A framework for interfaces to learning data in open learner models, learning analytics and related fields. International Journal of Artificial Intelligence in Education, 26(1), 293–331. http://dx.doi.org/10.1007/s40593-015-0090-8
Cairns, P., & Cox, A. L. (Eds.). (2008). Research methods for human–computer interaction (Vol. 12). Cambridge, UK: Cambridge University Press.
Davidoff, S., Lee, M. K., Dey, A. K., & Zimmerman, J. (2007). Rapidly exploring application design through speed dating. In J. Krumm, G. D. Abowd, A. Seneviratne, & T. Strang (Eds.), Proceedings of the International Conference on Ubiquitous Computing (UbiComp 2007), Lecture Notes in Computer Science, vol. 4717 (pp. 429–446). Springer, Berlin, Heidelberg. http://dx.doi.org/10.1007/978-3-540-74853-3_25
Dennerlein, S., Kowald, D., Pammer-Schindler, V., Lex, E., & Ley, T. (2018). Simulation-based co-creation of algorithms. In CEUR Workshop Proceedings (Vol. 2190). RWTH Aachen University.
Desmarais, M. C., & Baker, R. S. (2012). A review of recent advances in learner and skill modeling in intelligent learning environments. User Modeling and User-Adapted Interaction, 22(1–2), 9–38. http://dx.doi.org/10.1007/s11257-011-9106-8
Diana, N., Eagle, M., Stamper, J. C., Grover, S., Bienkowski, M. A., & Basu, S. (2017). Automatic peer tutor matching: Data-driven methods to enable new opportunities for help. In X. Hu, T. Barnes, A. Hershkovitz, & L. Paquette (Eds.), Proceedings of the 10th International Conference on Educational Data Mining (EDM2017), 25–28 June 2017, Wuhan, China (pp. 372–373). International Educational Data Mining Society.
DiSalvo, B., & DiSalvo, C. (2014). Designing for democracy in education: Participatory design and the learning sciences. In J. L. Polman, E. A. Kyza, D. K. O’Neill, I. Tabak, W. R. Penuel, A. S. Jurow, & L. D’Amico (Eds.), Learning and Becoming in Practice: Proceedings of the International Conference of the Learning Sciences (ICLS ’14), 23–27 June 2014, Boulder, CO, USA (Vol. 2, pp. 793–799). International Society of the Learning Sciences. https://dx.doi.org/10.22318/icls2014.793
Dollinger, M., & Lodge, J. M. (2018). Co-creation strategies for learning analytics. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 97–101). New York: ACM. http://dx.doi.org/10.1145/3170358.3170372
Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017). UX design innovation: Challenges for working with machine learning as a design material. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 6–11 May 2017, Denver, Colorado, USA (pp. 278–288). New York: ACM. http://dx.doi.org/10.1145/3025453.3025739
Evenson, S. (2006). Directed storytelling: Interpreting experience for design. In A. Bennett (Ed.), Design Studies: Theory and research in graphic design (pp. 231–240). Hudson, NY: Princeton Architectural Press.
Halloran, J., Hornecker, E., Fitzpatrick, G., Weal, M., Millard, D., Michaelides, D., Cruickshank, D., & De Roure, D. (2006). Unfolding understandings: Co-designing UbiComp in situ, over time. In Proceedings of the 6th Conference on Designing Interactive Systems (DIS ’06), 26–28 June 2006, University Park, PA, USA (pp. 109–118). New York: ACM. http://dx.doi.org/ 10.1145/1142405.1142423
Hanington, B., & Martin, B. (2012). Universal methods of design: 100 ways to research complex problems, develop innovative ideas, and design effective solutions. London: Rockport Publishers.
Heffernan, N. T., & Heffernan, C. L. (2014). The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education, 24(4), 470–497. http://dx.doi.org/10.1007/s40593-014-0024-x
Hoadley, C. (2017). How participatory design has influenced the learning sciences. In B. DiSalvo, J. Yip, E. Bonsignore, & C. DiSalvo (Eds.), Participatory design for learning (pp. 34–39). Abingdon-on-Thames, UK: Routledge.
Holstein, K., Hong, G., Tegene, M., McLaren, B. M., & Aleven, V. (2018). The classroom as a dashboard: Co-designing wearable cognitive augmentation for K–12 teachers. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 79–88). New York: ACM. http://dx.doi.org/10.1145/3170358.3170377
Holstein, K., McLaren, B. M., & Aleven, V. (2017a). SPACLE: Investigating learning across virtual and physical spaces using spatial replays. In Proceedings of the 7th International Conference on Learning Analytics and Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 358–367). New York: ACM. http://dx.doi.org/10.1145/3027385.3027450
Holstein, K., McLaren, B. M., & Aleven, V. (2017b). Intelligent tutors as teachers’ aides: Exploring teacher needs for real-time analytics in blended classrooms. In Proceedings of the 7th International Conference on Learning Analytics and Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 257–266). New York: ACM. http://dx.doi.org/10.1145/3027385.3027451
Holstein, K., McLaren, B. M., & Aleven, V. (2018a). Informing the design of teacher awareness tools through causal alignment analysis. In J. Kay & R. Luckin (Eds.), Rethinking Learning in the Digital Age: Making the Learning Sciences Count. Proceedings of the 13th International Conference of the Learning Sciences (ICLS ’18), 23–27 June 2018, London, UK (Vol. 1, pp. 104–111). International Society of the Learning Sciences.
Holstein, K., McLaren, B. M., & Aleven, V. (2018b). Student learning benefits of a mixed-reality teacher awareness tool in AI-enhanced classrooms. In C. Penstein Rosé, R. Martínez-Maldonado, U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, & B. du Boulay (Eds.), Proceedings of the 19th International Conference on Artificial Intelligence in Education (AIED 2018), 27–30 June 2018, London, UK. (pp. 154–168). Springer, Cham. http://dx.doi.org/10.1007/978-3-319-93843-1_12
Holstein, K., Wortman Vaughan, J., Daumé, H. III, Dudík, M., & Wallach, H. (2019). Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 4–9 May 2019, Glasgow, Scotland, UK, Paper No. 600. New York: ACM. https://dx.doi.org/10.1145/3290605.3300830
Holstein, K., Yu, Z., Sewall, J., Popescu, O., McLaren, B. M., & Aleven, V. (2018). Opening up an intelligent tutoring system development environment for extensible student modeling. In C. Penstein Rosé, R. Martínez-Maldonado, U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, & B. du Boulay (Eds.), Proceedings of the 19th International Conference on Artificial Intelligence in Education (AIED 2018), 27–30 June 2018, London, UK (pp. 169–183). Springer, Cham. http://dx.doi.org/10.1007/978-3-319-93843-1_13
Kai, S., Almeda, M. V., Baker, R. S., Heffernan, C., & Heffernan, N. (2018). Decision tree modeling of wheel-spinning and productive persistence in skill builders. Journal of Educational Data Mining, 10(1), 36–71. https://jedm.educationaldatamining.org/index.php/JEDM/article/view/210
Kamar, E. (2016). Directions in hybrid intelligence: Complementing AI systems with human intelligence. In S. Kambhampati (Ed.), Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI-16), 9–15 July 2016, New York, NY, USA (pp. 4070–4073). Palo Alto, CA: AAAI Press/International Joint Conferences on Artificial Intelligence.
Käser, T., Klingler, S., & Gross, M. (2016). When to stop? Towards universal instructional policies. In Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 289–298). New York: ACM. http://dx.doi.org/10.1145/2883851.2883961
Khachatryan, G. A., Romashov, A. V., Khachatryan, A. R., Gaudino, S. J., Khachatryan, J. M., Guarian, K. R., & Yufa, N. V. (2014). Reasoning mind genie 2: An intelligent tutoring system as a vehicle for international transfer of instructional methods in mathematics. International Journal of Artificial Intelligence in Education, 24(3), 333–382.
Koedinger, K. R., Baker, R. S., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J. (2010). A data repository for the EDM community: The PSLC DataShop. In C. Romero, S. Ventura, M. Pechenizkiy, & R. S. J. d. Baker (Eds.), Handbook of educational data mining (pp. 43–56). Boca Raton, FL: Chapman & Hall/CRC.
Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: A meta-analytic review. Review of Educational Research, 86(1), 42–78. http://dx.doi.org/10.3102/0034654315581420
Lee, M. K., & Baykal, S. (2017). Algorithmic mediation in group decisions: Fairness perceptions of algorithmically mediated vs. discussion-based social division. In Proceedings of the 20th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW 2017) 25 February–1 March 2017, Portland, OR, USA (pp. 1035–1048). New York: ACM. http://dx.doi.org/10.1145/2998181.2998230
Lee, M. K., Kusbit, D., Kahng, A., Kim, J. T., Yuan, X., Chan, A., Noothigattu, R., See, D., Lee, S., Psomas, C. A., & Procaccia, A. (2018). WeBuildAI: Participatory framework for fair and efficient algorithmic governance. Pre-print.
Lipton, Z. C. (2016). The mythos of model interpretability. Paper presented at the 2016 ICML Workshop on Human Interpretability in Machine Learning (WHI 2016), 23 June 2016, New York, NY, USA. arXiv preprint arXiv:1606.03490.
Martinez-Maldonado, R., Clayphan, A., Yacef, K., & Kay, J. (2015). MTFeedback: Providing notifications to enhance teacher awareness of small group work in the classroom. IEEE Transactions on Learning Technologies, 8(2), 187–200. http://dx.doi.org/10.1109/TLT.2014.2365027
Martinez-Maldonado, R. M., Kay, J., Yacef, K., & Schwendimann, B. (2012). An interactive teacher’s dashboard for monitoring groups in a multi-tabletop learning environment. In S. A. Cerri, W. J. Clancey, G. Papadourakis, & K.-K.
Panourgia (Eds.), Proceedings of the 11th International Conference on Intelligent Tutoring Systems (ITS 2012), 14–18 June 2012, Chania, Greece (pp. 482–492). Springer, Berlin, Heidelberg. http://dx.doi.org/10.1007/978-3-642-30950-2_62
Martinez-Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., & Clayphan, A. (2016). LATUX: An iterative workflow for designing, validating and deploying learning analytics visualisations. Journal of Learning Analytics, 2(3), 9–39. http://dx.doi.org/10.1007/978-3-642-30950-2_62
Mavrikis, M., Gutierrez-Santos, S., Geraniou, E., Noss, R., & Poulovassilis, A. (2013). Iterative context engineering to inform the design of intelligent exploratory learning environments for the classroom. In R. Luckin, S. Puntambekar, P. Goodyear, B. Grabowski, J. Underowood, & N. Winters (Eds.), Handbook of design in educational technology (pp. 80–92). Abingdon-on-Thames, UK: Routledge.
Mavrikis, M., Gutierrez-Santos, S., & Poulovassilis, A. (2016). Design and evaluation of teacher assistance tools for exploratory learning environments. In Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 168–172). New York: ACM. http://dx.doi.org/10.1145/2883851.2883909
Moraveji, N., Li, J., Ding, J., O’Kelley, P., & Woolf, S. (2007). Comicboarding: Using comics as proxies for participatory design with children. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ07), 28 April–3 May 2007, San Jose, CA (pp. 1371–1374). New York: ACM.
Odom, W., Zimmerman, J., Davidoff, S., Forlizzi, J., Dey, A. K., & Lee, M. K. (2012). A fieldwork of the future with user enactments. In Proceedings of the Designing Interactive Systems Conference (DIS ’12) 11–15 June 2012, Newcastle Upon Tyne, UK (pp. 338–347). New York: ACM. http://dx.doi.org/10.1145/2317956.2318008
Ogan, A., Walker, E., Baker, R. S., Rebolledo Mendez, G., Jimenez Castro, M., Laurentino, T., & De Carvalho, A. (2012). Collaboration in cognitive tutor use in Latin America: Field study and design recommendations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ12), 5–10 May 2012, Austin, TX, USA (pp. 1381–1390). New York: ACM. http://dx.doi.org/10.1145/2207676.2208597
Ogan, A., Walker, E., Baker, R., Rodrigo, M. M. T., Soriano, J. C., & Castro, M. J. (2015). Towards understanding how to assess help-seeking behavior across cultures. International Journal of Artificial Intelligence in Education, 25(2), 229–248. http://dx.doi.org/10.1007/s40593-014-0034-8
Olsen, J. K. (2017). Orchestrating Combined Collaborative and Individual Learning in the Classroom. Unpublished doctoral dissertation, Carnegie Mellon University.
Oulasvirta, A., Kurvinen, E., & Kankainen, T. (2003). Understanding contexts by being there: Case studies in bodystorming. Personal and Ubiquitous Computing, 7(2), 125–134.
Penuel, W. R., Roschelle, J., & Shechtman, N. (2007). Designing formative assessment software with teachers: An analysis of the co-design process. Research and Practice in Technology Enhanced Learning, 2(1), 51–74. http://dx.doi.org/10.1142/S1793206807000300
Penuel, W. R., & Yarnall, L. (2005). Designing handheld software to support classroom assessment: Analysis of conditions for teacher adoption. The Journal of Technology, Learning and Assessment, 3(5). https://ejournals.bc.edu/index.php/jtla/article/view/1658
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018). Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810.
Prieto-Alvarez, C. G., Martinez-Maldonado, R., & Anderson, T. (2018). Co-designing learning analytics tools with learners. Learning analytics in the classroom: Translating learning analytics research for teachers. Abingdon-on-Thames, UK: Routledge.
Rau, M. A. (2015). Why do the rich get richer? A structural equation model to test how spatial skills affect learning with representations. In O. C. Santos et al. (Eds.), Proceedings of the 8th International Conference on Educational Data Mining (EDM2015), 26–29 June 2015, Madrid, Spain (pp. 350–357). International Educational Data Mining Society.
Reich, J., & Ito, M. (2017). From good intentions to real outcomes: Equity by design in learning technologies. Irvine, CA: Digital Media and Learning Research Hub.
Ritter, S., Carlson, R., Sandbothe, M., & Fancsali, S. E. (2015). Carnegie Learning’s adaptive learning products. In O. C. Santos et al. (Eds.), Proceedings of the 8th International Conference on Educational Data Mining (EDM2015), 26–29 June 2015, Madrid, Spain (pp. 633–634). International Educational Data Mining Society.
Ritter, S., Yudelson, M., Fancsali, S., & Berman, S. R. (2016). Towards integrating human and automated tutoring systems. In T. Barnes et al. (Eds.), Proceedings of the 9th International Conference on Educational Data Mining (EDM2016), 29 June–2 July 2016, Raleigh, NC, USA (pp. 626–627). International Educational Data Mining Society.
Rodriguez-Triana, M. J., Prieto Santos, L. P., Vozniuk, A., Shirvani Boroujeni, M., Schwendimann, B. A., Holzer, A. C., & Gillet, D. (2017). Monitoring, awareness and reflection in blended technology enhanced learning: A systematic review. International Journal of Technology Enhanced Learning, 9, 126–150. http://dx.doi.org/10.1504/IJTEL.2017.084489
Schofield, J. W., Eurich-Fulcer, R., & Britt, C. L. (1994). Teachers, computer tutors, and teaching: The artificially intelligent tutor as an agent for classroom change. American Educational Research Journal, 31(3), 579–607. http://dx.doi.org/10.2307/1163227
Schuler, D., & Namioka, A. (Eds.). (1993). Participatory design: Principles and practices. Boca Raton, FL: CRC Press.
Shepard, L. A. (1997). Insights gained from a classroom-based assessment project. Center for the Study of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing, Graduate School of Education & Information Studies, University of California, Los Angeles.
Sujan, M., & Pasquini, A. (1998). Allocating tasks between humans and machines in complex systems. In Proceedings of the 4th International Conference on Achieving Quality in Software (AQuIS ’98), 30 March–2 April 1998, Venice, Italy (pp. 173–184).
Tohidi, M., Buxton, W., Baecker, R., & Sellen, A. (2006). User sketches: A quick, inexpensive, and effective way to elicit more reflective user feedback. In Proceedings of the 4th Nordic Conference on Human–Computer Interaction: Changing Roles (NordiCHI 2006), 14–18 October 2006, Oslo, Norway (pp. 105–114). New York: ACM. http://dx.doi.org/10.1145/1182475.1182487
Veitch, J., Salmon, J., & Ball, K. (2007). Children’s active free play in local neighborhoods: A behavioral mapping study. Health Education Research, 23(5), 870–879. http://dx.doi.org/10.1093/her/cym074
Wickens, C. D., Gordon, S. E., Liu, Y., & Lee, J. (1998). An introduction to human factors engineering. New York: Longman.
Wright, P., Dearden, A., & Fields, B. (2000). Function allocation: A perspective from studies of work practice. International Journal of Human–Computer Studies, 52(2), 335–355. http://dx.doi.org/10.1006/ijhc.1999.0292
Yacef, K. (2002). Intelligent teaching assistant systems. In Proceedings of the International Conference on Computers in Education (ICCE 2002), 3–6 December 2002, Auckland, New Zealand (pp. 136–140). IEEE Computer Society. http://dx.doi.org/10.1109/CIE.2002.1185885
Zhu, H., Yu, B., Halfaker, A., & Terveen, L. (2018, November). Value-sensitive algorithm design: Method, case study, and lessons. Proceedings of the ACM on Human–Computer Interaction, CSCW issue, Vol. 2, article #194. http://dx.doi.org/10.1145/3274463
Downloads
Published
How to Cite
Issue
Section
License
TEST
