The Science Student Electronic Exit Ticket (SEET) System
Visualizations to Help Teachers Notice and Reflect on Classroom Inequalities
DOI:
https://doi.org/10.18608/jla.2024.8199Keywords:
learning analytic dashboards, equity visualizations, studies of teacher adoption and use, learning sciences, science education, research paperAbstract
This study examined the ways in which an equity analytics tool — the SEET system — supported middle school science teachers’ reflections on the experiences of diverse students in their classrooms. The tool provides teachers with “equity visualizations” — disaggregated classroom data by gender and race/ethnicity — designed to support teachers to notice and reflect on inequitable patterns in student participation in classroom knowledge-building activities, as well as “whole class visualizations” that enable teachers to look at participation patterns. The visualizations were based on survey data collected from students reflecting on the day’s lessons, responding to questions aligned with three theoretical constructs indicative of equitable participation in science classrooms: coherence, relevance, and contribution. The study involved 42 teachers, divided into two cohorts, participating in a two-month professional learning series. Diary studies and semi-structured interviews were used to probe teachers’ perceptions of the visualizations’ usability, usefulness, and utility for supporting their reflections on student experiences and instructional practices. A key result is that only the “equity visualizations” prompted teacher reflections on diverse student experiences. However, despite the support equity visualizations provided for this core task, the teachers consistently ranked the whole class visualizations as more usable and useful.
References
Ahn, J., Campos, F., Hays, M., & Digiacomo, D. (2019). Designing in context: Reaching beyond usability in learning analytics dashboard design. Journal of Learning Analytics, 6(2), 70–85. https://doi.org/10.18608/jla.2019.62.5
Arthurs, L. A., & Kreager, B. Z. (2017). An integrative review of in-class activities that enable active learning in college science classroom settings. International Journal of Science Education, 39(15), 2073–2091. https://doi.org/10.1080/09500693.2017.1363925
Arthars, N., & Liu, D. Y.-T. (2020). How and why faculty adopt learning analytics. In D. Ifenthaler & D. Gibson (Eds.), Adoption of data analytics in higher education learning and teaching (pp. 201–220). Springer, Cham. https://doi.org/10.1007/978-3-030-47392-1_11
Ali, L., Asadi, M., Gašević, D., Jovanović, J., & Hatala, M. (2013). Factors influencing beliefs for adoption of a learning analytics tool: An empirical study. Computers & Education, 62, 130–148. https://doi.org/10.1016/j.compedu.2012.10.023
Brown, B. A. (2021). Science in the city: Culturally relevant STEM education. Harvard Education Press.
Basu, S. J., & Barton, A. C. (2007). Developing a sustained interest in science among urban minority youth. Journal of Research in Science Teaching, 44(3), 466–489. https://doi.org/10.1002/tea.20143
Banilower, E. R., Smith, P. S., Malzahn, K. A., Plumley, C. L., Gordon, E. M., & Hayes, M. L. (2018). Report of the 2018 NSSME+. Horizon Research, Inc. https://files.eric.ed.gov/fulltext/ED598121.pdf
Carlone, H. B. (2004). The cultural production of science in reform‐based physics: Girls’ access, participation, and resistance. Journal of Research in Science Teaching, 41(4), 392–414. https://doi.org/10.1002/tea.20006
Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. SAGE Publications.
Campos, F. C., Ahn, J., DiGiacomo, D. K., Nguyen, H., & Hays, M. (2021). Making sense of sensemaking: Understanding how K–12 teachers and coaches react to visual analytics. Journal of Learning Analytics, 8(3), 60–80. https://doi.org/10.18608/jla.2021.7113
Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research (3rd ed.). SAGE Publications.
Dourado, R. A., Rodrigues, R. L., Ferreira, N., Mello, R. F., Gomes, A. S., & Verbert, K. (2021). A teacher-facing learning analytics dashboard for process-oriented feedback in online learning. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 482–489). ACM Press. https://doi.org/10.1145/3448139.3448187
Dazo, S. L., Stepanek, N. R., Chauhan, A., & Dorn, B. (2017). Examining instructor use of learning analytics. Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’17), 6–11 May 2017, Denver, CO, USA (pp. 2504–2510). ACM Press. https://doi.org/10.1145/3027063.3053256
Eccles, J. S. (1983). Expectancies, values and academic behaviors. In J. T. Spence (Ed.), Achievement and achievement motives (pp. 75–146). W. H. Freeman.
Heer, J., Bostock, M., & Ogievetsky, V. (2010). A tour through the visualization zoo: A survey of powerful visualization techniques, from the obvious to the obscure. Queue, 8(5), 20–30. https://doi.org/10.1145/1794514.1805128
Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. https://doi.org/10.1177/0002764213479367
Li, Q., Jung, Y., & Wise, A. F. (2021). Beyond first encounters with analytics: Questions, techniques and challenges in instructors’ sensemaking. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 344–353). ACM Press. https://doi.org/10.1145/3448139.3448172
Larson, R., & Csikszentmihalyi, M. (2014). The experience sampling method. In M. Csikszentmihalyi (Ed.), Flow and the foundations of positive psychology (pp. 21–34). Springer, Dordrecht. https://doi.org/10.1007/978-94-017-9088-8_2
Molenaar, I., & Knoop-van Campen, C. A. N. (2019). How teachers make dashboard information actionable. IEEE Transactions on Learning Technologies, 12(3), 347–355. https://doi.org/10.1109/TLT.2018.2851585
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). SAGE Publications.
National Academies of Sciences, Engineering, and Medicine. (2018). How people learn II: Learners, contexts, and cultures. The National Academies Press. https://doi.org/10.17226/24783
Nguyen, H., Campos, F., & Ahn, J. (2021). Discovering generative uncertainty in learning analytics dashboards. In M. Sahin & D. Ifenthaler (Eds.), Visualizations and dashboards for learning analytics (pp. 457–475). Springer, Cham. https://doi.org/10.1007/978-3-030-81222-5_21
Ochoa, X., Knight, S., & Wise, A. F. (2020). Learning analytics impact: Critical conversations on relevance and social responsibility. Journal of Learning Analytics, 7(3), 1–5. https://doi.org/10.18608/jla.2020.73.1
Office for Civil Rights. (2018). 2015–16 civil rights data collection: STEM course taking. US Department of Education, Office for Civil Rights. https://civilrightsdata.ed.gov/assets/downloads/stem-course-taking.pdf
Peck, E. M., Ayuso, S. E., & El-Etr, O. (2019). Data is personal: Attitudes and perceptions of data visualization in rural Pennsylvania. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 4–9 May 2019, Glasgow, Scotland, UK (Paper No. 244). ACM Press. https://doi.org/10.1145/3290605.3300474
PERTS. (2010). Project for Education Research that Scales. Retrieved 15 September 2022 from https://www.perts.net/
Penuel, W. R., Van Horne, K., Jacobs, J. K., & Turner, M. (2018). Developing a validity argument for practical measures of student experience in projectbased science classrooms. Annual Meeting of the American Educational Research Association, New York, NY, USA. http://learndbir.org/uploads/Resources/Validity-Argument-for-Practical-Measures-of-Student-Experience-in-Project.pdf
Penuel, W. R., & Watkins, D. A. (2019). Assessment to promote equity and epistemic justice: A use-case of a research-practice partnership in science education. The ANNALS of the American Academy of Political and Social Science, 683(1), 201–216. https://doi.org/10.1177/0002716219843249
Reinholz, D., & Shah, N. (2021). Equity and equality: How data visualizations mediate teacher sensemaking about racial and gender inequity. Contemporary Issues in Technology and Teacher Education, 21(3), 646–669. https://www.learntechlib.org/primary/p/218443/
Reinholz, D. L., & Shah, N. (2018). Equity analytics: A methodological approach for quantifying participation patterns in mathematics classroom discourse. Journal for Research in Mathematics Education, 49(2), 140–177. https://doi.org/10.5951/jresematheduc.49.2.0140
Reiser, B. J., Novak, M., & McGill, T. A. W. (2017). Coherence from the students’ perspective: Why the vision of the framework for K–12 science requires more than simply “combining” three dimensions of science learning. Paper presented to the Board on Science Education Workshop “Instructional Materials for the Next Generation Science Standards,” 27 June 2017. The National Academy of Science.
Reiser, B. J., Novak, M., McGill, T. A. W., & Penuel, W. R. (2021). Storyline units: An instructional model to support coherence from the students’ perspective. Journal of Science Teacher Education, 32(7), 805–829. https://doi.org/10.1080/1046560X.2021.1884784
Raza, A., Penuel, W. R., Jacobs, J., & Sumner, T. (2020). Supporting equity in schools: Using visual learning analytics to understand learners’ classroom experiences. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user experience research: An introduction for the field of learning design & technology (pp. 277–296). EdTech Books. https://edtechbooks.org/ux/supporting_school_equity
Raza, A., Penuel, W. R., Allen, A.-R., Sumner, T., & Jacobs, J. K. (2021). “Making it culturally relevant”: A visual learning analytics system supporting teachers to reflect on classroom equity. Proceedings of the 15th International Conference of the Learning Sciences (ICLS ’21), 8–11 June 2021, Bochum, Germany (pp. 442–449). International Society of the Learning Sciences. https://repository.isls.org//handle/1/7501
Rieman, J. (1993). The diary study: A workplace-oriented research tool to guide laboratory efforts. Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems (INTERCHI ’93), 24–29 April 1993, Amsterdam, The Netherlands (pp. 321–326). ACM Press.
Schwarz, C. V., Passmore, C., & Reiser, B. J. (2017). Moving beyond “knowing” science to making sense of the world. In C. V. Schwarz, C. Passmore, & B. J. Reiser (Eds.), Helping students make sense of the world using next generation science and engineering practices (pp. 3–21). NSTA Press.
Stroupe, D. (2014). Examining classroom science practice communities: How teachers and students negotiate epistemic agency and learn science‐as‐practice. Science Education, 98(3), 487–516. https://doi.org/10.1002/sce.21112
Shah, N., & Coles, J. A. (2020). Preparing teachers to notice race in classrooms: Contextualizing the competencies of preservice teachers with antiracist inclinations. Journal of Teacher Education, 71(5), 584–599. https://doi.org/10.1177/0022487119900204
Shah, N., & Lewis, C. M. (2019). Amplifying and attenuating inequity in collaborative learning: Toward an analytical framework. Cognition and Instruction, 37(4), 423–452. https://doi.org/10.1080/07370008.2019.1631825
Theobald, E. J., Eddy, S. L., Grunspan, D. Z., Wiggins, B. L., & Crowe, A. J. (2017). Student perception of group dynamics predicts individual performance: Comfort and equity matter. PLOS ONE, 12(7), e0181336. https://doi.org/10.1371/journal.pone.0181336
Tai, R. H., Liu, C. Q., Maltese, A. V., & Fan, X. (2006). Planning early for careers in science. Science, 312(5777), 1143–1144. https://doi.org/10.1126/science.1128690
Vieira, C., Parsons, P., & Byrd, V. (2018). Visual learning analytics of educational data: A systematic literature review and research agenda. Computers & Education, 122, 119–135. https://doi.org/10.1016/j.compedu.2018.03.018
Verbert, K., Ochoa, X., De Croon, R., Dourado, R. A., & De Laet, T. (2020). Learning analytics dashboards: The past, the present and the future. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 35–40). ACM Press. https://doi.org/10.1145/3375462.3375504
Wise, A. F., & Jung, Y. (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://doi.org/10.18608/jla.2019.62.4
Wise, A. F., Sarmiento, J. P., & Boothe, M., Jr. (2021). Subversive learning analytics. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 639–645). ACM Press. https://doi.org/10.1145/3448139.3448210
Williamson, K., & Kizilcec, R. F. (2021). Learning analytics dashboard research has neglected diversity, equity and inclusion. Proceedings of the 8th ACM Conference on Learning @ Scale (L@S 2021), 22–25 June 2021, Virtual Event Germany (pp. 287–290). ACM Press. https://doi.org/10.1145/3430895.3460160
Williamson, K., & Kizilcec, R. (2022). A review of learning analytics dashboard research in higher education: Implications for justice, equity, diversity, and inclusion. Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK ’22), 21–25 March 2022, Online (pp. 260–270). ACM Press. https://doi.org/10.1145/3506860.3506900
Wright, C., Wendell, K. B., & Paugh, P. P. (2018). “Just put it together to make no commotion:” Re-imagining urban elementary students’ participation in engineering design practices. International Journal of Education in Mathematics, Science and Technology, 6(3), 285–301. https://www.ijemst.net/index.php/ijemst/article/view/198/145
Warren, B., & Rosebery, A. S. (2011). Navigating interculturality: African American male students and the science classroom. Journal of African American Males in Education, 2(1), 98–115. https://jaamejournal.scholasticahq.com/article/18415-navigating-interculturality-african-american-male-students-and-the-science-classroom
Zirkel, S., Garcia, J. A., & Murphy, M. C. (2015). Experience-sampling research methods and their potential for education research. Educational Researcher, 44(1), 7–16. https://doi.org/10.3102/0013189X14566879
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
