Measuring Mastery Behaviors at Scale: The Persistence, Effort, Resilience and Challenge-Seeking Task (PERC)
DOI:
https://doi.org/10.18608/jla.2020.71.2Keywords:
performance tasks, persistence, effort, resilience, growth mindset, achievementAbstract
Mastery behaviours — seeking out challenging tasks and continuing to work on them despite difficulties — are integral to achievement but difficult to measure with precision. The current study reports on the development and validation of the computer-based persistence, effort, resilience, and challenge-seeking (PERC) task in two demographically diverse samples of adolescents (total N = 3,188). We present evidence for convergent validity with self-reported mastery behaviours and learning mindsets, discriminant validity from theoretically unrelated constructs, and incremental predictive validity for grade point average (GPA). Results suggest that the PERC task may be a useful tool for assessing mastery behaviours at scale.
References
Abrahams, L., Pancorbo, G., Primi, R., Santos, D., Kyllonen, P., John, O. P., & De Fruyt, F. (2019). Social-emotional skill assessment in children and adolescents: Advances and challenges in personality, clinical, and educational contexts. Psychological Assessment, 31(4), 460–473. https://dx.doi.org/10.1037/pas0000591
Arendasy, M., Sommer, M., Herle, M., Schützhofer, B., & Inwanschitz, D. (2011). Modeling effects of faking on an objective personality test. Journal of Individual Differences, 32, 210–218. https://dx.doi.org/10.1027/1614-0001/a000053
Aron, A., Aron, E. N., & Smollan, D. (1992). Inclusion of other in the self scale and the structure of interpersonal closeness. Journal of Personality and Social Psychology, 63, 596–612. https://dx.doi.org/10.1037/0022-3514.63.4.596
Bettinger, E., Ludvigsen, S., Rege, M., Solli, I. F., & Yeager, D. (2018). Increasing perseverance in math: Evidence from a field experiment in Norway. Journal of Economic Behavior & Organization, 146, 1–15. https://dx.doi.org/10.1016/j.jebo.2017.11.032
Blackwell, L., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78(1), 246–263. https://dx.doi.org/10.1111/j.1467-8624.2007.00995.x
Brown, E. D. (2009). Persistence in the face of academic challenge for economically disadvantaged preschool children. Journal of Early Childhood Research, 7(2), 175–186. https://dx.doi.org/10.1177/1476718X09102650
Burnette, J. L., O’Boyle, E. H., VanEpps, E. M., Pollack, J. M., & Finkel, E. J. (2013). Mind-sets matter: A meta-analytic review of implicit theories and self-regulation. Psychological Bulletin, 139(3), 655–701. https://dx.doi.org/10.1037/a0029531
Caspi, A., Elder, G. H., & Bem, D. J. (1988). Moving away from the world: Life-course patterns of shy children. Developmental Psychology, 24(6), 824–831. https://dx.doi.org/10.1037/0012-1649.24.6.824
Choudhury, N., & Gorman, K. S. (1999). The relation between reaction time and psychometric intelligence in a rural Guatemalan adolescent population. International Journal of Psychology, 34(4), 209–217. https://dx.doi.org/10.1080/002075999399855
Clark, L. A., & Watson, D. (2019). Constructing validity: New developments in creating objective measuring instruments. Psychological Assessment, 31(12), 1412–1427. https://dx.doi.org/10.1037/pas0000626
Cohen, P., Cohen, J., Aiken, L. S., & West, S. (1999). The problem of units and the circumstance for POMP. Multivariate Behavioral Research. 34(3), 315–346. https://dx.doi.org/10.1207/S15327906MBR3403_2
Costenbader, V., & Ngari, S. M. (2001). A Kenyan standardization of the Raven’s coloured progressive matrices. School Psychology International, 22(3), 258–268. https://dx.doi.org/10.1177/0143034301223003
Cutumisu, M., Blair, K. P., Chin, B., & Schwartz, D. L. (2015). Posterlet: A game-based assessment of children’s choices to seek feedback and to revise. Journal of Learning Analytics, 2(1), 49–71. https://dx.doi.org/10.18608/jla.2015.21.4
Diener, C. I., & Dweck, C. S. (1978). An analysis of learned helplessness: Continuous changes in performance, strategy, and achievement cognitions following failure. Journal of Personality and Social Psychology, 36(5), 451–462. https://dx.doi.org/10.1037/0022-3514.36.5.451
Diener, C. I., & Dweck, C. S. (1980). An analysis of learned helplessness: II. The processing of success. Journal of Personality and Social Psychology, 39(5), 940–952. https://dx.doi.org/10.1037/0022-3514.39.5.940
Diener, E., Ingelhart, R. F., & Tay, L. (2013). Theory and validity of life satisfaction scales. Social Indicators Research, 112(5), 497–527. https://dx.doi.org/10.1007/s11205-012-0076-y
Duckworth, A. L., & Kern, M. L. (2011). A meta-analysis of the convergent validity of self-control measures. Journal of Research in Personality, 45(3), 259–268. https://dx.doi.org/10.1016/j.jrp.2011.02.004
Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237–251. https://doi.org/10.3102/0013189X15584327
Dweck, C. S. (2000). Self-theories: Their role in motivation, personality, and development. New York: Psychology Press, Taylor & Francis Group. https://doi.org/10.4324/9781315783048
Dweck, C. S., Chiu, C., & Hong, Y. (1995). Implicit theories and their role in judgments and reactions: A world from two perspectives. Psychological Inquiry, 6(4), 267–285. https://dx.doi.org/10.1207/s15327965pli0604_1
Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95(2), 256–273. https://dx.doi.org/10.1037/0033-295X.95.2.256
Eisinga, R., te Grotenhuis, M., & Pelzer, B. (2013). The reliability of a two-item scale: Person, Cronbach, or Spearman-Brown? International Journal of Public Health, 58(4), 637–642. https://dx.doi.org/10.1007/s00038-012-0416-3
Elliot, A., McGregor, H., & Gable, S. (1999). Achievement goals, study strategies, and exam performance: A mediational analysis. Journal of Educational Psychology, 91, 549–563. https://dx.doi.org/10.1037/0022-0663.91.3.549
Enkavi, Z. A, Eisenberg, I. W., Bissett, P. G., Mazza, G. L., MacKinnon, D. P., Marsch, L. A., & Poldrack, R. A. (2019). Large-scale analysis of test–retest reliabilities of self-regulation measures. Proceedings of the National Academy of Sciences of the United States of America, 116(12), 5472–5477. https://dx.doi.org/10.1073/pnas.1818430116
Funder, D. C., & Ozer, D. J. (2019). Evaluating effect sizes in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168. https://dx.doi.org/10.1177/2515245919847202
Galla, B. M., Plummer, B. D., White, R. E., Meketon, D., D’Mello, S. K., & Duckworth, A. L. (2014). The academic diligence task (ADT): Assessing individual differences in effort on tedious but important schoolwork. Contemporary Education Psychology, 39(4), 314–325. https://dx.doi.org/10.1016/j.cedpsych.2014.08.001
Gibson, A., Kitto, K., & Bruza, P. (2016). Towards the discovery of learner metacognition from reflective writing. Journal of Learning Analytics, 3(2), 22–36. https://dx.doi.org/10.18608/jla.2016.32.3
Gottfried, A. E. (1985). Academic intrinsic motivation in elementary and junior high school students. Journal of Educational Psychology, 77(6), 631–645. https://dx.doi.org/10.1037/0022-0663.77.6.631
Haimovitz, K., & Dweck, C. S. (2016). Parents’ views of failure predict children’s fixed and growth intelligence mind-sets. Psychological Science, 27(6), 859–869. https://dx.doi.org/10.1177/0956797616639727
Hedge, C., Powell, G., & Sumner, P. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavioral Research Methods, 50(3), 1166–1186. https://dx.doi.org/10.3758/s13428-017-0935-1
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3), 61–83. https://dx.doi.org/10.1017/S0140525X0999152X
Joksimović, S., Gašević, D., Loughin, T. M., Kovanović, V., & Hatala, M. (2015). Learning at distance: Effects of interaction traces on academic achievement. Computers & Education, 87, 204–217. https://dx.doi.org/10.1016/j.compedu.2015.07.002
Knafo, A., Israel, S., Darvasi, A., Bachner-Melman, R., Uzefovsky, F., Cohen, L., …, Ebstein, R. P. (2007). Individual differences in allocation of funds in the dictator game associated with length of the arginine vasopressin 1a receptor RS3 promoter region and correlation between RS3 length and hippocampal mRNA. Genes Brain Behavior, 7(3), 266–275. https://dx.doi.org/10.1111/j.1601-183X.2007.00341.x
Kovanović, V., Gašević, D., Dawson, S., Joksimović, S., Baker, R. S., & Hatala, M. (2015). Penetrating the black box of time-on-task estimation, Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK ’15), 16–20 March 2015, Poughkeepsie, NY (pp. 184–193). New York: ACM. https://dx.doi.org/10.1145/2723576.2723623
Kraft, M. A. (2019). Interpreting effect sizes of education interventions. EdWorkingPaper No. 19-10. https://dx.doi.org/10.26300/8pjp-2z74
Krumm, A. E., Beattie, R., Takahashi, S., D’Angelo, C., Feng, M., & Cheng, B. (2016). Practical measurement and productive persistence: Strategies for using digital learning system data to drive improvement. Journal of Learning Analytics, 3(2), 116–138. https://dx.doi.org/10.18608/jla.2016.32.6
Licht, B. G., & Dweck, C. S. (1984). Determinants of academic achievement: The interaction of children’s achievement orientations with skill area. Developmental Psychology, 20(4), 628–636. https://dx.doi.org/10.1037/0012-1649.20.4.628
Lucas, R. E., & Donnellan, B. M. (2012). Estimating the reliability of single-item life satisfaction measures: Results from four national panel studies. Social Indicators Research, 105(3), 323–331. https://dx.doi.org/10.1007/s11205-011-9783-z
Mangels, J. A., Butterfield, B., Lamb, J., Good, C., & Dweck, C. (2006). Why do beliefs about intelligence influence learning success? A social cognitive neuroscience model. Social Cognitive and Affective Neuroscience, 1(2), 75–86. https://dx.doi.org/10.1093/scan/nsl013
Marcus, B. (2006). Relationships between faking, validity, and decision criteria in personnel selection. Psychology Science, 48(3), 226–246. https://psycnet.apa.org/record/2006-21801-003
Martin, C. K., Nacu, D., & Pinkard, N. (2016). Revealing opportunities for 21st century learning: An approach to interpreting user trace log data. Journal of Learning Analytics, 3(2), 37–87. https://dx.doi.org/10.18608/jla.2016.32.4
Meindl, P., Yu, A., Galla, B., Quirk, A., Haeck, C., Parker, G. J., …, Duckworth, A. L. (2019). A brief behavioral measure of frustration tolerance predicts academic achievement immediately and two years later. Emotion, 19(6), 1081–1092. https://dx.doi.org/10.1037/emo0000492
Miller, R., Greene, B., Montalvo, G., Ravindran, B., & Nichols, J. (1996). Engagement in academic work: The role of learning goals, future consequences, pleasing others, and perceived ability. Contemporary Educational Psychology, 21, 388–422. https://dx.doi.org/10.1006/ceps.1996.0028
Mischel, W., Shoda, Y., & Peake, P. K. (1988). The nature of adolescent competencies predicted by preschool delay of gratification. Journal of Personality and Social Psychology, 54, 687–696. https://dx.doi.org/10.1037/0022-3514.54.4.687
Moser, J. S., Schroder, H. S., Heeter, C., Moran, T. P., & Lee, Y. (2011). Mind your errors: Evidence for a neural mechanism linking growth mind-set to adaptive posterror adjustments. Psychological Science, 22(12), 1484–1489. https://dx.doi.org/10.1177/0956797611419520
Mueller, C. M., & Dweck, C. S. (1998). Praise for intelligence can undermine children’s motivation and performance. Journal of Personality and Social Psychology, 75(1), 33–52. https://dx.doi.org/10.1037/0022-3514.75.1.33
Raven, J., Raven, J. C., & Court, J. H. (1998). Manual for Raven’s progressive matrices and vocabulary scales. Oxford, U.K.: Oxford Psychologists Press.
Robins, R. W., & Pals, J. L. (2002). Implicit self-theories in the academic domain: Implications for goal orientation, attributions, affect, and self-esteem change. Self and Identity, 1(4), 313–336. https://dx.doi.org/10.1080/15298860290106805
Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. https://dx.doi.org/10.1177/0002764213498851
Soland, J., Zamarro, G., Cheng, A., & Hitt, C. (2019). Identifying naturally occurring direct assessments of social-emotional competencies: The promise and limitations of survey and assessment disengagement metadata. Educational Researcher, 48(7), 466–478. https://dx.doi.org/10.3102/0013189X19861356
Vazire, S., & Mehl, M. R. (2008). Knowing me, knowing you: The accuracy and unique predictive validity of self-ratings and other-ratings of daily behavior. Journal of Personality and Social Psychology, 95(5), 1202–1216. https://dx.doi.org/10.1037/a0013314
Ventura, M., & Shute, V. (2013). The validity of a game-based assessment of persistence. Computers in Human Behavior, 29, 2568–2572. https://dx.doi.org/10.1016/j.chb.2013.06.033
Ventura, M., Shute, V., & Zhao, W. (2013). The relationship between video game use and a performance-based measure of persistence. Computers & Education, 60(1), 52–58. https://dx.doi.org/10.1016/j.compedu.2012.07.003
Walton, G. M., & Cohen, G. L. (2007). A question of belonging: Race, social fit, and achievement. Journal of Personality and Social Psychology, 92(1), 82–96. https://dx.doi.org/10.1037/0022-3514.92.1.82
Willerman, L., Turner, R. G., & Peterson, M. (1976). A comparison of the predictive validity of typical and maximal personality measures. Journal of Research in Personality, 10(4), 482–492. https://dx.doi.org/10.1016/0092-6566(76)90063-5
Yeager, D. S., & Bryk, A. (2015). Practical measurement. Unpublished manuscript, University of Texas at Austin.
Yeager, D. S., Romero, C., Paunesku, D., Hulleman, C. S., Schneider, B., Hinojosa, C., …, Dweck, C. S. (2016). Using design thinking to improve psychological interventions: The case of the growth mindset during the transition to high school. Journal of Educational Psychology, 108(3), 374–391. https://dx.doi.org/10.1037/edu0000098
Ziegler, M., Schmidt-Atzert, L., Bühner, M., & Krumm, S. (2007). Fakability of different measurement methods for achievement motivation: Questionnaire, semi-projective, and objective. Psychology Science, 49, 291–307. https://psycnet.apa.org/record/2008-02884-001
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2020 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
TEST
