Towards Analytics for Wholistic School Improvement: Hierarchical Process Modelling and Evidence Visualization
DOI:
https://doi.org/10.18608/jla.2017.42.13Keywords:
Learning analytics, academic analytics, leadership, decision support, complex systems, educational values, visualization, dashboard, uncertainty, risk, surveysAbstract
Central to the mission of most educational institutions is the task of preparing the next generation of citizens to contribute to society. Schools, colleges, and universities value a range of outcomes — e.g., problem solving, creativity, collaboration, citizenship, service to community — as well as academic outcomes in traditional subjects. Often referred to as “wider outcomes,” these are hard to quantify. While new kinds of monitoring technologies and public datasets expand the possibilities for quantifying these indices, we need ways to bring that data together to support sense-making and decision-making. Taking a systems perspective, the hierarchical process modelling (HPM) approach and the “Perimeta” visual analytic provides a dashboard that informs leadership decision-making with heterogeneous, often incomplete evidence. We report a prototype of Perimeta modelling from education, aggregating wider outcomes data across a network of schools, and calculating their cumulative contribution to key performance indicators, using the visual analytic of the Italian flag to make explicit not only the supporting evidence, but also the challenging evidence, as well as areas of uncertainty. We discuss the nature of the modelling decisions and implicit values involved in quantifying these kinds of educational outcomes.
References
Assessment Reform Group. (1999). Assessment for learning: Beyond the black box. Cambridge, UK: University of Cambridge School of Education.
Assessment Reform Group. (2002). Testing, motivation and learning. Cambridge, UK: Assessment Reform Group.
Beer, S. (1984). The viable system model: Its provenance, development, methodology and pathology. Journal of the Operational Research Society, 35(1), 7–25. http://dx.doi.org/10.1057/jors.1984.2
Beer, S. (1985). Diagnosing the system for organisations. Hoboken, NJ: John Wiley.
Behrens, J., & DiCerbo, K. (2014). Harnessing the currents of the digital ocean. In J. Larussen & B. White (Eds.), Learning analytics: From research to practice. New York: Springer. http://dx.doi.org/10.1007/978-1-4614-3305-7_3
Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78(1), 246–263. http://dx.doi.org/10.1111/j.1467-8624.2007.00995.x
Blockley, D. (2010). The importance of being process. Civil Engineering and Environmental Systems, 27(3), 189–199. http://dx.doi.org/10.1080/10286608.2010.482658
Blockley, D., & Godfrey, P. (2000). Doing it differently: Systems for rethinking construction. London: Telford.
Bottery, M. (2004). The challenges of educational leadership. London: Paul Chapman.
Bower, D. F. (2004). Leadership and the self-organizing school. Paper presented at the Complexity Science and Educational Research Conference, Kingston, Ontario, Canada.
Bryk, A., & Gomez, L. (2008). Ruminations on Reinventing an R&D Capacity for Educational Improvement. Paper presented at the The Supply Side of School Reform and the Future of Educational Entrepreneurship, Stanford, CA.
Bryk, A. S., Sebring, P. B., Allensworth, E., Luppescu, S., & Easton, J. Q. (2009). Organizing schools for improvement: Lessons from Chicago: University of Chicago Press.
Bryk, A., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: Building networked improvement communities in education. In M. T. Hallinan (Ed.), Frontiers in Sociology of Education (pp. 127–162). Netherlands: Springer. http://dx.doi.org/10.1007/978-94-007-1576-9_7
Bryk, A., Gomez, L., Grunow, A., & LeMahieu, P. (2015). Learning to improve: How America’s schools can get better at getting better. Cambridge, MA: Harvard Educational Press.
Bryk, A., Sebring, P., Allensworth, E., Luppescu, S., & Easton, J. (2010). Organizing schools for improvement: Lessons from Chicago. Chicago, IL: University of Chicago Press.
Buckingham Shum, S., & Deakin Crick, R. (2012). Learning dispositions and transferable competencies: Pedagogy, modelling and learning analytics. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ʼ12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 92–101). New York: ACM. http://dx.doi.org/10.1145/2330601.2330629
Buckingham Shum, S., & Deakin Crick, R. (2016). Learning analytics for 21st century competencies. Journal of Learning Analytics and Knowledge, 3(2). http://dx.doi.org/10.18608/jla.2016.32.2
Campbell, J., DeBlois, P., & Oblinger, D. (2007). Academic analytics: A new tool for a new era. Educause Review, 42(4), 40–57.
Checkland, P. (1999). Systems thinking systems practice. Chichester, UK: John Wiley.
Checkland, P., & Scholes, J. (1999). Soft systems methodology in action. Chichester, UK: John Wiley.
Claxton, G. (2008). Expanding the capacity to learn: A new end for education? Opening Keynote Address, British Educational Research Association Annual Conference, 6 September 2006, Warwick University.
Cui, W., & Blockley, D. (1990). Interval probability theory for evidential support. International Journal of Intelligent Systems, 5(2), 183–192. http://dx.doi.org/10.1002/int.4550050204
Daniel, B. (2015). Big Data and analytics in higher education: Opportunities and challenges. British Journal of Educational Technology, 46(5), 904–920. http://dx.doi.org/10.1111/bjet.12230
Darling-Hammond, L., & Bransford, J. (2005). Preparing teachers for a changing world: What teachers should learn and be able to do. San Francisco, CA: Jossey-Bass.
Davis, J., MacDonald, A., & White, L. (2010). Problem-structuring methods and project management: an example of stakeholder involvement using Hierarchical Process Modelling methodology. Journal of the Operational Research Society,, 61, 893-904. http://dx.doi.org/10.1057/jors.2010.12
Davis, B., & Sumara, D. (2006). Complexity and education inquiries into learning, teaching and research. London: Routledge
Davis, J., & Fletcher, P. (2000). Managing assets under uncertainty. Society of Petroleum Engineers Asia Pacific Conference on Integrated Modelling for Asset Management (APCIMAM 2000), 25-26 April 2000, Yokohama, Japan.
Deakin Crick, R. (2017). Learning analytics: Layers, loops and processes in a virtual learning infrastructure. In G. Siemens & C. Lang (Eds.), Handbook of learning analytics & educational data mining. Society of Learning Analytics Research (SoLAR). http://dx.doi.org/10.18608/hla17.025
Deakin Crick, R., Barr, S., Green, H., & Pedder, D. (2015). Evaluating the wider outcomes of schools: Complex systems modelling. Educational Management Administration and Leadership, 45(4). http://dx.doi.org/10.1177/1741143215597233
Deakin Crick, R., Huang, S., Ahmed Shafi, A., & Goldspink, C. (2015). Developing resilient agency in learning: The internal structure of learning power. British Journal of Educational Studies, 63(2), 121–160. http://dx.doi.org/10.1080/00071005.2015.1006574
Dempster, A. (1969). Elements of continuous multivariate analysis. Reading, MA: Addison-Wesley.
Delors, J. (2000). Learning: The treasure within. http://www.teacherswithoutborders.org/pdf/Delors.pdf
Dweck, C. S. (2000). Self-theories: Their role in motivation, personality, and development. New York: Psychology Press.
European Commission. (2007). Key competences for lifelong learning: European reference framework. Luxembourg: Publications Office of the European Union.
Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304–317. https://doi.org/10.1504/IJTEL.2012.051816
Ferguson, R., Clow, D., Macfadyen, L., Essa, A., Dawson, S., & Alexander, S. (2014). Setting learning analytics in context: Overcoming the barriers to large-scale adoption. Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 251–253). New York: ACM. http://dx.doi.org/10.1145/2567574.2567592
Goldspink, C. (2015). Communities making a difference national partnerships SA teaching for effective learning pedagogy research project 2010–2013. Adelaide: South Australia Department for Education and Child Development.
Goldspink, C., & Foster, M. (2013). A conceptual model and set of instruments for measuring student engagement in learning. Cambridge Journal of Education, 43(3), 291–311. http://dx.doi.org/10.1080/0305764x.2013.776513
Gunter, H. (2001). Leaders and leadership in education. London: Paul Chapman.
Hall, J., Le Masurier, J., Baker-Langman, E., Davis, J., & Taylor, C. (2004). Decision-support methodology for performance-based asset management. Journal of Civil Engineering and Environmental Systems, 21, 51–75. http://dx.doi.org/10.1080/1028660031000135086
Hall, J., Blockley, D., & Davis, J. (1998). Uncertain inference using interval probability theory. International Journal of Approximate Reasoning, 19, 247–264. https://dx.doi.org/10.1016/S0888-613X(98)10010-5
Harlen, W., & Deakin Crick, R. (2003). Testing and motivation for learning. Assessment in Education, 10(2), 169–207. http://dx.doi.org/10.1080/0969594032000121270
James, M., & Gipps, C. (1998). Broadening the basis of assessment to prevent the narrowing of learning. The Curriculum Journal, 9(3), 285–297. http://dx.doi.org/10.1080/0958517970090303
James, M., McCormick, R., Black, P., Carmichael, P., Drummond, M., Fox, A., Wiliam, D. (2007). Improving learning how to learn: Classrooms, schools and networks. London: Routledge.
Joldersma, C., & Deakin Crick, R. (2010). Citizenship, discourse ethics and an emancipatory model of lifelong learning. London: Routledge.
Jackson, M. C. (2000). Systems approaches to management. London: Kluwer Academic.
Knight, S., Buckingham Shum, S., & Littleton, K. (2014). Epistemology, assessment, pedagogy: Where learning meets analytics in the middle space. Journal of Learning Analytics, 1(2), 23–47. http://dx.doi.org/10.18608/jla.2014.12.3
Norris, D., Baer, L., & Offerman, M. (2009). A national agenda for action analytics, White Paper. National Symposium on Action Analytics, 21–23 September 2009, St. Paul, Minnesota, USA.
MacBeath, J., & Cheng, Y. (2008). Leadership for learning: International perspectives. London: Routledge.
MacBeath, J., & McGlynn, A. (2002). Self-evaluation: What’s in it for schools? London: RoutledgeFalmer.
Mason, M. (Ed.). (2008). Complexity theory and the philosophy of education. Chichester, UK: Wiley-Blackwell.
McCombs, B., & Miller, L. (2008). A school leader’s guide to creating learner-centered education: From complexity to simplicity. New York: Corwin Press.
McCombs, B. L., Daniels, D. H., & Perry, K. E. (2008). Children’s and teachers’ perceptions of learner-centered practices, and student motivation: Implications for early schooling. The Elementary School Journal, 109(1), 16–35. https://dx.doi.org/10.1086/592365
Macfadyen, L., Dawson, S., Abelardo, P., & Gašević, D. (2014). Embracing big data in complex educational systems: The learning analytics imperative and the policy challenge. Research & Practice in Assessment, 9, 17–28. http://www.rpajournal.com/dev/wp-content/uploads/2014/10/A2.pdf
McKinsey Report. (2007). How the world’s best performing school systems come out on top. http://www.smhc-cpre.org/wp-content/uploads/2008/07/how-the-worlds-best-performing-school-systems-come-out-on-top-sept-072.pdf
OECD. (2001). Investing in competencies for all. Meeting of OECD Education Ministers. Retrieved from http://www.oecd.org/innovation/research/1870557.pdf
Opfer, V., & Pedder, D. (2011). The lost promise of teacher professional development in England. European Journal of Teacher Education, 34(1), 3–24. http://dx.doi.org/10.1080/02619768.2010.534131
Pedder, D. (2006). Organizational conditions that foster successful classroom promotion of learning how to learn. Research Papers in Education, 21(2), 171–200. http://dx.doi.org/10.1080/02671520600615687
Pedder, D. (2007). Profiling teachers’ professional learning practices and values: Differences between and within schools. The Curriculum Journal, 18(3), 231–252. http://dx.doi.org/10.1080/09585170701589801
Pedder, D., James, M., & MacBeath, J. (2005). How teachers value and practise professional learning. Research Papers in Education, 20(3), 209–243. http://dx.doi.org/10.1080/02671520500192985
Piety, P. J., Hickey, D. T., & Bishop, M. J. (2014). Educational data sciences: Framing emergent practices for analytics of learning, organizations, and systems. Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK ʼ14), 24–28 March 2014, Indianapolis, IN, USA (pp. 193–202). New York: ACM. http://dx.doi.org/10.1145/2567574.2567582
Reay, D. W. D. (1999). ‘I'll be a nothing’: Structure, agency and the construction of identity through assessment. British Educational Research Journal, 25(3), 343–354. http://dx.doi.org/10.1080/0141192990250305
Ritchie, R., & Deakin Crick, R. (2007). Distributing leadership for personalising learning. London: Continuum.
Robinson, V., Lloyd, C., & Rowe, K. (2009). The impact of leadership on student outcomes: An analysis of the differential effects of leadership types. Educational Administration Quarterly, 45(3), 515–520. http://dx.doi.org/10.1177/0013161x09335140
Rychen, D., & Salagnik, L. (2001). Definition and selection of key competencies. Theoretical and Conceptual Foundations (DeSeCo) Background paper, Swiss Federal Statistical Office, OECD, ESSI, Neuchatel.
Selvin, A., & Buckingham Shum, S. (2014). Constructing knowledge art: An experiential perspective on crafting participatory representations. Williston, VT: Morgan & Claypool.
Shafer, G. (1976). A mathematical theory of evidence. Princeton, NJ: Princeton University Press.
Silins, H., & Mulford, B. (2004). Schools as learning organisations: Effects on teacher leadership and student outcomes. School Effectiveness and School Improvement, 15, 443–466. http://dx.doi.org/10.1080/09243450512331383272
Sillitto, H. (2015). Architecting systems: Concepts, principles and practice. London: College Publications Systems Series.
Snowden, D., & Boone, M. E. (2007, November). A leader’s framework for decision making. Harvard Business Review. https://hbr.org/2007/11/a-leaders-framework-for-decision-making
Thomas, D., & Seely Brown, J. (2009). Learning for a world of constant change: Homo sapiens, Homo faber & Homo ludens revisited. Paper presented at the 7th Glion Colloquium, June 2009, University of Southern California. http://www.johnseelybrown.com/Learning%20for%20a%20World%20of%20Constant%20Change.pdf
Thomas, D., & Seely Brown, J. (2011). A new culture of learning cultivating the imagination for a world of constant change. CreateSpace Independent Publishing Platform.
West-Burnham, J. (2005). Leadership for personalizing learning. In S. De Frietas & C. Yapp (Eds.), Personalizing learning in the 21st century. Stafford, UK: Network Educational Press.
White, L. 2006. Evaluating problem-structuring methods: Developing an approach to show the value and effectiveness of PSMs. Journal of the Operational Research Society, 57, 842–845. http://dx.doi.org/10.1057/palgrave.jors.2602149
Yeager, D., Bryk, A., Muhich, J., Hausman, H., & Morales, L. (2015). Practical measurement. San Francisco, CA: Carnegie Foundation for the Advancement of Teaching.
Zohar, D. (1997). Rewiring the corporate brain. San Francisco, CA: Berrett-Koehler Publishers.
Published
How to Cite
Issue
Section
License
Copyright (c) 2017 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
