Název: Differential effects of additional formative assessments on student learning behaviors and outcomes
Zdrojový dokument: Studia paedagogica. 2023, roč. 28, č. 3, s. [9]-38
Rozsah
[9]-38
-
ISSN1803-7437 (print)2336-4521 (online)
Trvalý odkaz (DOI): https://doi.org/10.5817/SP2023-3-1
Trvalý odkaz (handle): https://hdl.handle.net/11222.digilib/digilib.79677
Type: Článek
Jazyk
Licence: Neurčená licence
Přístupová práva
otevřený přístup
Upozornění: Tyto citace jsou generovány automaticky. Nemusí být zcela správně podle citačních pravidel.
Abstrakt(y)
It is well-established that formative assessments with accompanying feedback can enhance learning. However, the degree to which additional formative assessments on the same material further improve learning outcomes remains an open research question. Moreover, it is unclear whether providing additional formative assessments impacts self-regulated learning behavior, and if the benefits of such assessments depend on students' self-regulated learning behavior. The current study, conducted in a real-world blended learning setting and using a Learning Analytics approach, compares 154 students who completed additional formative assessments with 154 students who did not. The results indicate that the additional formative assessments led to an improvement in learning outcomes, but also had both positive and negative effects on students' self-regulated learning behavior. Students who completed additional formative assessments performed better on the assessments but reported lower levels of subjective comprehension and devoted more time to completing exercises. Simultaneously, they devoted less effort to additional learning activities (additional investment), such as class preparation and post-processing. Furthermore, the impact of additional formative assessments on learning success depended on students' self-regulated learning behavior. It was primarily the students who invested above-average time during formative assessments (time investment) who benefited from the additional exercises. Cluster analysis revealed that high-effort students (those with above-average time investment and above-average additional investment) gained the most from the extra exercises. In contrast, low-effort students and those who achieved high performance with relatively low effort (efficient students) did not benefit from additional formative assessments. In conclusion, providing students with additional formative assessments can enhance learning, but it should be done with caution as it can alter self-regulated learning behavior in both positive and negative ways, and not all students may benefit from it equally.
Reference
[1] Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017). Rethinking the use of tests: A meta-analysis of practice testing. Review of Educational Research, 87(3), 659-701. https://doi.org/10.3102/0034654316689306
[2] Avhustiuk, M. M., Pasichnyk, I. D., & Kalamazh, R. V. (2018). The illusion of knowing in metacognitive monitoring: Effects of the type of information and of personal, cognitive, metacognitive, and individual psychological characteristics. Europe's Journal of Psychology, 14(2), 317-341. https://doi.org/10.5964/ejop.v14i2.1418
[3] Baker, R., Xu, D., Park, J., Yu, R., Li, Q., Cung, B., Fischer, Ch., Rodriguez, F., Warschauer, M., & Smyth, P. (2020). The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: Opening the black box of learning processes. International Journal of Educational Technology in Higher Education, 17(3), 1-24. https://doi.org/10.1186/s41239-020-00187-1
[4] Batsell Jr., W. R., Perry, J. L., Hanley, E., & Hostetter, A. B. (2017). Ecological validity of the testing effect: The use of daily quizzes in introductory psychology. Teaching of Psychology, 44(1), 18-23. https://doi.org/10.1177/0098628316677492
[5] Bentler, P. M., & Satorra, A. (2010). Testing model nesting and equivalence. Psychological Methods, 15(2), 111-123. https://doi.org/10.1037/a0019625
[6] Bertilsson, F., Stenlund, T., Wiklund-Hörnqvist, C., & Jonsson, B. (2021). Retrieval practice: Beneficial for all students or moderated by individual differences?. Psychology Learning & Teaching, 20(1), 21-39. https://doi.org/10.1177/1475725720973494
[7] Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417-444. https://doi.org/10.1146/annurev-psych-113011-143823
[8] Borter, N. (2016). Aufgabenkomplexität und Intelligenz [Dissertation, Universität Bern]. https://boris.unibe.ch/101386/
[9] Boston, C. (2002). The concept of formative assessment. Practical Assessment, Research, and Evaluation, 8(9). https://doi.org/10.7275/kmcq-dj31
[10] Carpenter, S. K. (2011). Semantic information activated during retrieval contributes to later retention: Support for the mediator effectiveness hypothesis of the testing effect. Journal of Experimental Psychology. Learning, Memory, and Cognition, 37(6), 1547-1552. https://doi.org/10.1037/a0024140
[11] Carpenter, S. K., Pashler, H., & Cepeda, N. J. (2009). Using tests to enhance 8th grade students' retention of U.S. history facts. Applied Cognitive Psychology, 23(6), 760-771. https://doi.org/10.1002/acp.1507
[12] Charrad, M., Ghazzali, N., Boiteau, V., & Niknafs, A. (2014). NbClust: An R package for determining the relevant number of clusters in a data set. Journal of Statistical Software, 61(6), 1-36. https://doi.org/10.18637/jss.v061.i06
[13] Chatti, M. A., Dyckhoff, A. L., Schroeder, U., & Thüs, H. (2013). A reference model for learning analytics. International Journal of Technology Enhanced Learning, 4(5-6), 318-331. https://doi.org/10.1504/IJTEL.2012.051815
[14] Clariana, R. B., & Park, E. (2021). Item-level monitoring, response style stability, and the hard-easy effect. Educational Technology Research and Development, 69, 693-710. https://doi.org/10.1007/s11423-021-09981-8
[15] Cogliano, M., Kardash, C. M., & Bernacki, M. L. (2019). The effects of retrieval practice and prior topic knowledge on test performance and confidence judgments. Contemporary Educational Psychology, 56, 117-129. https://doi.org/10.1016/j.cedpsych.2018.12.001
[16] Dalmaijer, E. S., Nord, C. L., & Astle, D. E. (2021). Statistical power for cluster analysis. arXiv. https://doi.org/10.48550/arXiv.2003.00381
[17] DiStefano, C., Zhu, M., & Mîndrilã, D. (2009). Understanding and using factor scores: Considerations for the applied researcher. Practical Assessment, Research, and Evaluation 14(20). https://doi.org/10.7275/DA8T-4G52
[18] Dodonov, Y. S., & Dodonova, Y. A. (2012). Response time analysis in cognitive tasks with increasing difficulty. Intelligence, 40(5), 379-394. https://doi.org/10.1016/j.intell.2012.07.002
[19] Dolan, C. V. (1994). Factor analysis of variables with 2, 3, 5 and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47(2), 309-326. https://doi.org/10.1111/j.2044-8317.1994.tb01039.x
[20] Dunst, B., Benedek, M., Jauk, E., Bergner, S., Koschutnig, K., Sommer, M., Ischebeck, A., Spinath, B., Arendasy, M., Bühner, M., Freudenthaler, H., & Neubauer, A. C. (2014). Neural efficiency as a function of task demands. Intelligence, 42, 22-30. https://doi.org/10.1016/j.intell.2013.09.005
[21] Eriksson, J., Kalpouzos, G., & Nyberg, L. (2011). Rewiring the brain with repeated retrieval: A parametric fMRI study of the testing effect. Neuroscience Letters, 505(1), 36-40. https://doi.org/10.1016/j.neulet.2011.08.061
[22] Fernandez, J., & Jamet, E. (2017). Extending the testing effect to self-regulated learning. Metacognition and Learning, 12, 131-156. https://doi.org/10.1007/s11409-016-9163-9
[23] Foss, D. J., & Pirozzolo, J. W. (2017). Four semesters investigating frequency of testing, the testing effect, and transfer of training. Journal of Educational Psychology, 109(8), 1067-1083. https://doi.org/10.1037/edu0000197
[24] Francis, A. P., Wieth, M. B., Zabel, K. L., & Carr, T. H. (2020). A classroom study on the role of prior knowledge and retrieval tool in the testing effect. Psychology Learning & Teaching, 19(3), 258-274. https://doi.org/10.1177/1475725720924872
[25] Goldhammer, F. (2015). Measuring ability, speed, or both? Challenges, psychometric solutions, and what can be gained from experimental control. Measurement: Interdisciplinary Research and Perspectives, 13(3-4), 133-164. https://doi.org/10.1080/15366367.2015.1100020
[26] Goldhammer, F., Hahnel, C., Kroehne, U., & Zehner, F. (2021). From byproduct to design factor. On validating the interpretation of process indicators based on log data. Large-Scale Assessments in Education, 9, 1-25. http://nbn-resolving.de/urn:nbn:de:0111-pedocs-250050
[27] Greene, R. L. (2008). Repetition and spacing effects. In H. L. Roediger (Ed.), Learning and memory: A comprehensive reference. Vol. 2: Cognitive psychology of memory (pp. 65-78). Elsevier.
[28] Greving, S., Lenhard, W., & Richter, T. (2020). Adaptive retrieval practice with multiple-choice questions in the university classroom. Journal of Computer Assisted Learning, 36(6), 799-809. https://doi.org/10.1111/jcal.12445
[29] Heitmann, S., Grund, A., Fries, S., Berthold, K., & Roelle, J. (2022). The quizzing effect depends on hope of success and can be optimized by cognitive load-based adaptation. Learning and Instruction, 77. https://doi.org/10.1016/j.learninstruc.2021.101526
[30] Ifenthaler, D. (2015). Learning Analytics. In J. M. Spector (Ed.), The SAGE encyclopedia of educational technology (Vol. 2, pp. 447-451). SAGE publication. https://madoc.bib.uni-mannheim.de/38809/
[31] Jensen, J. L., McDaniel, M. A., Kummer, T. A., Godoy, P. D. D. M., & St. Clair, B. (2020). Testing effect on high-level cognitive skills. CBE—Life Sciences Education, 19(3). 2020. https://doi.org/10.1187/cbe.19-10-0193
[32] Jost, N. S., Jossen, S. L., Rothen, N., & Martarelli, C. S. (2021). The advantage of distributed practice in a blended learning setting. Education and Information Technologies, 26, 3097-3113. https://doi.org/10.1007/s10639-020-10424-9
[33] Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet and Higher Education, 33, 74-85. https://doi.org/10.1016/j.iheduc.2017.02.001
[34] Karpicke, J., & Bauernschmidt, A. (2011). Spaced retrieval: Absolute spacing enhances learning regardless of relative spacing. Journal of Experimental Psychology. Learning, Memory, and Cognition, 37(5), 1250-1257. https://doi.org/10.1037/a0023436
[35] Karpicke, J., & Roediger, H. (2007). Expanding retrieval practice promotes short-term retention, but equally spaced retrieval enhances long-term retention. Journal of Experimental Psychology. Learning, Memory, and Cognition, 33(4), 704-719. 2007. https://doi.org/10.1037/0278-7393.33.4.704
[36] Khayi, N. A., & Rus, V. (2019). Clustering students based on their prior knowledge. In F. Collin, A. Merceron, & M. Desmarais (Eds.), Proceedings of the 12th International Conference on Educational Data Mining. Université du Québec à Montréal, Polytechnique Montréal (pp. 246-251). https://educationaldatamining.org/edm2019/proceedings/
[37] Kim, D., Yoon, M., Jo, I.-H., & Branch, R. M. (2018). Learning analytics to support self-regulated learning in asynchronous online courses: A case study at a women's university in South Korea. Computers & Education, 127, 233-251. https://doi.org/10.1016/j.compedu.2018.08.023
[38] Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). Guilford Press.
[39] Kossen, C., & Ooi, C.-Y. (2021). Trialling micro-learning design to increase engagement in online courses. Asian Association of Open Universities Journal, 16(3), 299-310. https://doi.org/10.1108/AAOUJ-09-2021-0107
[40] Kovanovic, V., Gasevic, D., Joksimovic, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74-89. https://doi.org/10.1016/j.iheduc.2015.06.002
[41] Lamotte, M., Izaute, M., & Darnon, C. (2021). Can tests improve learning in real university classrooms? Journal of Cognitive Psychology, 33(8), 974-992. https://doi.org/10.1080/20445911.2021.1956939
[42] Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education-A Literature Review. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends. Studies in systems, decision and control (pp. 1-23). Springer. https://doi.org/10.1007/978-3-319-52977-6_1
[43] Li, S., Chen, G., Xing, W., Zheng, J., & Xie, C. (2020). Longitudinal clustering of students' self-regulated learning behaviors in engineering design. Computers & Education, 153, Article 103899. https://doi.org/10.1016/j.compedu.2020.103899
[44] Li, H., Flanagan, B., Konomi, S., & Ogata, H. (2018). Measuring behaviors and identifying indicators of self-regulation in computer-assisted language learning courses. Research and Practice in Technology Enhanced Learning, 13(19), 1-12. https://doi.org/10.1186/s41039-018-0087-7
[45] Mavroudi, A., Giannakos, M., & Krogstie, J. (2018). Supporting adaptive learning pathways through the use of learning analytics: Developments, challenges and future opportunities. Interactive Learning Environments, 26(2), 206-220. https://doi.org/10.1080/10494820.2017.1292531
[46] McDaniel, M. A, Thomas, R. C., Agarwal, P. K., McDermott, K. B., & Roediger, H. L. (2013). Quizzing in middle-school science: Successful transfer performance on classroom exams. Applied Cognitive Psychology, 27(3), 360-372. https://doi.org/10.1002/acp.2914
[47] Mega, C., Ronconi, L., & De Beni, R. (2014). What makes a good student? How emotions, self-regulated learning, and motivation contribute to academic achievement. Journal of Educational Psychology, 106(1), 121-131. https://doi.org/10.1037/a0033546
[48] Minear, M., Coane, J. H., Boland, S. C., Cooney, L. H., & Albat, M. (2018). The benefits of retrieval practice depend on item difficulty and intelligence. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(9), 1474-1486. https://doi.org/10.1037/xlm0000486
[49] Moosbrugger, H., & Kelava, A. (2012). Testtheorie und Fragebogenkonstruktion. Springer. https://doi.org/10.1007/978-3-642-20072-4
[50] Ning, H. K., & Downing, K. (2015). A latent profile analysis of university students' self-regulated learning strategies. Studies in Higher Education, 40(7), 1328-1346. https://doi.org/10.1080/03075079.2014.880832
[51] Norman, G. (2010). Likert scales, levels of measurement and the "laws" of statistics. Advances in Health Sciences Education, 15(5), 625-632. https://doi.org/10.1007/s10459-010-9222-y
[52] Parpala, A., Mattsson, M., Herrmann, K. J., Bager-Elsborg, A., & Hailikari, T. (2021). Detecting the variability in student learning in different disciplines - A person-oriented approach. Scandinavian Journal of Educational Research, 66(6), 1020-1037. https://doi.org/10.1080/00313831.2021.1958256
[53] Perry, N. E., & Winne, P. H. (2006). Learning from Learning Kits: gStudy traces of students' self-regulated engagements with computerized content. Educational Psychology Review, 18, 211-228. https://doi.org/10.1007/s10648-006-9014-3
[54] Pishgar, F., Greifer, N., Leyrat, C., & Stuart, E. (2021). MatchThem: Matching and weighting after multiple imputation. The R Journal, 13(2), 294-305. https://doi.org/10.32614/RJ-2021-073
[55] R Core Team (2021). R: A language and environment for statistical computing. R foundation for statistical computing. Retrieved from https://www.R-project.org/
[56] Revelle, W. R. (2022). psych: Procedures for psychological, psychometric, and personality research. R package version 2.2.5. Northwestern University. Retrieved from https://CRAN.R-project.org/package=psych
[57] Robey, A. (2019). The benefits of testing: Individual differences based on student factors. Journal of Memory and Language, 108. https://doi.org/10.1016/j.jml.2019.104029
[58] Robitzsch, A. (2020). Why ordinal variables can (almost) always be treated as continuous variables: Clarifying assumptions of robust continuous and ordinal factor analysis estimation methods. Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.589965
[59] Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02
[60] Schwieren, J., Barenberg, J., & Dutke, S. (2017). The testing effect in the psychology classroom: A meta-analytic perspective. Psychology Learning & Teaching, 16(2), 179-196. https://doi.org/10.1177/1475725717695149
[61] Shin, D., & Shim, J. (2021). A systematic review on data mining for mathematics and science education. International Journal of Science and Mathematics Education, 19, 639-659. https://doi.org/10.1007/s10763-020-10085-7
[62] Soderstrom, N. C., & Bjork, R. A. (2014). Testing facilitates the regulation of subsequent study time. Journal of Memory and Language, 73, 99-115. https://doi.org/10.1016/j.jml.2014.03.003
[63] Sun, Z., & Xie, K. (2020). How do students prepare in the pre-class setting of a flipped undergraduate math course? A latent profile analysis of learning behavior and the impact of achievement goals. The Internet and Higher Education, 46. https://doi.org/10.1016/j.iheduc.2020.100731
[64] Szpunar, K. K., McDermott, K. B., & Roediger, H. L. III. (2008). Testing during study insulates against the buildup of proactive interference. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(6), 1392-1399. https://doi.org/10.1037/a0013082
[65] van Alten, D. C. D., Phielix, C., Janssen, J., & Kester, L. (2021). Secondary students' online self-regulated learning during flipped learning: A latent profile analysis. Computers in Human Behavior, 118. https://doi.org/10.1016/j.chb.2020.106676
[66] van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1-67. https://doi.org/10.18637/jss.v045.i03
[67] Vanslambrouck, S., Zhu, C., Pynoo, B., Lombaerts, K., Tondeur, J., & Scherer, R. (2019). A latent profile analysis of adult students' online self-regulation in blended learning environments. Computers in Human Behavior, 99, 126-136. https://doi.org/10.1016/j.chb.2019.05.021
[68] Waspada, I., Bahtiar, N., & Wibowo, A. (2019). Clustering student behavior based on quiz activities on moodle LMS to discover the relation with a final exam score. Journal of Physics: Conference Series, 1217. https://doi.org/10.1088/1742-6596/1217/1/012118
[69] Wissman, K. T., Rawson, K. A., & Pyc, M. A. (2011). The interim test effect: Testing prior material can facilitate the learning of new material. Psychonomic Bulletin & Review, 18, 1140-1147. https://doi.org/10.3758/s13423-011-0140-7
[70] Yang, C., Luo, L., Vadillo, M. A., Yu, R., & Shanks, D. R. (2021). Testing (quizzing) boosts classroom learning: A systematic and meta-analytic review. Psychological Bulletin, 147(4), 399-435. https://doi.org/10.1037/bul0000309
[71] Yang, C., Potts, R., & Shanks, D. R. (2017). The forward testing effect on self-regulated study time allocation and metamemory monitoring. Journal of Experimental Psychology: Applied, 23(3), 263-277. https://doi.org/10.1037/xap0000122
[72] Yerkes, R. M., & Dodson, J. D. (1908). The Relation of Strength of Stimulus to Rapidity of Habit Formation. Journal of Comparative Neurology & Psychology, 18, 459-482. https://doi.org/10.1002/cne.920180503
[73] Zheng, J., Xing, W., Zhu, G., Chen, G., Zhao, H., & Xie, C. (2020). Profiling self-regulation behaviors in STEM learning of engineering design. Computers & Education, 143. https://doi.org/10.1016/j.compedu.2019.103669
[2] Avhustiuk, M. M., Pasichnyk, I. D., & Kalamazh, R. V. (2018). The illusion of knowing in metacognitive monitoring: Effects of the type of information and of personal, cognitive, metacognitive, and individual psychological characteristics. Europe's Journal of Psychology, 14(2), 317-341. https://doi.org/10.5964/ejop.v14i2.1418
[3] Baker, R., Xu, D., Park, J., Yu, R., Li, Q., Cung, B., Fischer, Ch., Rodriguez, F., Warschauer, M., & Smyth, P. (2020). The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: Opening the black box of learning processes. International Journal of Educational Technology in Higher Education, 17(3), 1-24. https://doi.org/10.1186/s41239-020-00187-1
[4] Batsell Jr., W. R., Perry, J. L., Hanley, E., & Hostetter, A. B. (2017). Ecological validity of the testing effect: The use of daily quizzes in introductory psychology. Teaching of Psychology, 44(1), 18-23. https://doi.org/10.1177/0098628316677492
[5] Bentler, P. M., & Satorra, A. (2010). Testing model nesting and equivalence. Psychological Methods, 15(2), 111-123. https://doi.org/10.1037/a0019625
[6] Bertilsson, F., Stenlund, T., Wiklund-Hörnqvist, C., & Jonsson, B. (2021). Retrieval practice: Beneficial for all students or moderated by individual differences?. Psychology Learning & Teaching, 20(1), 21-39. https://doi.org/10.1177/1475725720973494
[7] Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417-444. https://doi.org/10.1146/annurev-psych-113011-143823
[8] Borter, N. (2016). Aufgabenkomplexität und Intelligenz [Dissertation, Universität Bern]. https://boris.unibe.ch/101386/
[9] Boston, C. (2002). The concept of formative assessment. Practical Assessment, Research, and Evaluation, 8(9). https://doi.org/10.7275/kmcq-dj31
[10] Carpenter, S. K. (2011). Semantic information activated during retrieval contributes to later retention: Support for the mediator effectiveness hypothesis of the testing effect. Journal of Experimental Psychology. Learning, Memory, and Cognition, 37(6), 1547-1552. https://doi.org/10.1037/a0024140
[11] Carpenter, S. K., Pashler, H., & Cepeda, N. J. (2009). Using tests to enhance 8th grade students' retention of U.S. history facts. Applied Cognitive Psychology, 23(6), 760-771. https://doi.org/10.1002/acp.1507
[12] Charrad, M., Ghazzali, N., Boiteau, V., & Niknafs, A. (2014). NbClust: An R package for determining the relevant number of clusters in a data set. Journal of Statistical Software, 61(6), 1-36. https://doi.org/10.18637/jss.v061.i06
[13] Chatti, M. A., Dyckhoff, A. L., Schroeder, U., & Thüs, H. (2013). A reference model for learning analytics. International Journal of Technology Enhanced Learning, 4(5-6), 318-331. https://doi.org/10.1504/IJTEL.2012.051815
[14] Clariana, R. B., & Park, E. (2021). Item-level monitoring, response style stability, and the hard-easy effect. Educational Technology Research and Development, 69, 693-710. https://doi.org/10.1007/s11423-021-09981-8
[15] Cogliano, M., Kardash, C. M., & Bernacki, M. L. (2019). The effects of retrieval practice and prior topic knowledge on test performance and confidence judgments. Contemporary Educational Psychology, 56, 117-129. https://doi.org/10.1016/j.cedpsych.2018.12.001
[16] Dalmaijer, E. S., Nord, C. L., & Astle, D. E. (2021). Statistical power for cluster analysis. arXiv. https://doi.org/10.48550/arXiv.2003.00381
[17] DiStefano, C., Zhu, M., & Mîndrilã, D. (2009). Understanding and using factor scores: Considerations for the applied researcher. Practical Assessment, Research, and Evaluation 14(20). https://doi.org/10.7275/DA8T-4G52
[18] Dodonov, Y. S., & Dodonova, Y. A. (2012). Response time analysis in cognitive tasks with increasing difficulty. Intelligence, 40(5), 379-394. https://doi.org/10.1016/j.intell.2012.07.002
[19] Dolan, C. V. (1994). Factor analysis of variables with 2, 3, 5 and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47(2), 309-326. https://doi.org/10.1111/j.2044-8317.1994.tb01039.x
[20] Dunst, B., Benedek, M., Jauk, E., Bergner, S., Koschutnig, K., Sommer, M., Ischebeck, A., Spinath, B., Arendasy, M., Bühner, M., Freudenthaler, H., & Neubauer, A. C. (2014). Neural efficiency as a function of task demands. Intelligence, 42, 22-30. https://doi.org/10.1016/j.intell.2013.09.005
[21] Eriksson, J., Kalpouzos, G., & Nyberg, L. (2011). Rewiring the brain with repeated retrieval: A parametric fMRI study of the testing effect. Neuroscience Letters, 505(1), 36-40. https://doi.org/10.1016/j.neulet.2011.08.061
[22] Fernandez, J., & Jamet, E. (2017). Extending the testing effect to self-regulated learning. Metacognition and Learning, 12, 131-156. https://doi.org/10.1007/s11409-016-9163-9
[23] Foss, D. J., & Pirozzolo, J. W. (2017). Four semesters investigating frequency of testing, the testing effect, and transfer of training. Journal of Educational Psychology, 109(8), 1067-1083. https://doi.org/10.1037/edu0000197
[24] Francis, A. P., Wieth, M. B., Zabel, K. L., & Carr, T. H. (2020). A classroom study on the role of prior knowledge and retrieval tool in the testing effect. Psychology Learning & Teaching, 19(3), 258-274. https://doi.org/10.1177/1475725720924872
[25] Goldhammer, F. (2015). Measuring ability, speed, or both? Challenges, psychometric solutions, and what can be gained from experimental control. Measurement: Interdisciplinary Research and Perspectives, 13(3-4), 133-164. https://doi.org/10.1080/15366367.2015.1100020
[26] Goldhammer, F., Hahnel, C., Kroehne, U., & Zehner, F. (2021). From byproduct to design factor. On validating the interpretation of process indicators based on log data. Large-Scale Assessments in Education, 9, 1-25. http://nbn-resolving.de/urn:nbn:de:0111-pedocs-250050
[27] Greene, R. L. (2008). Repetition and spacing effects. In H. L. Roediger (Ed.), Learning and memory: A comprehensive reference. Vol. 2: Cognitive psychology of memory (pp. 65-78). Elsevier.
[28] Greving, S., Lenhard, W., & Richter, T. (2020). Adaptive retrieval practice with multiple-choice questions in the university classroom. Journal of Computer Assisted Learning, 36(6), 799-809. https://doi.org/10.1111/jcal.12445
[29] Heitmann, S., Grund, A., Fries, S., Berthold, K., & Roelle, J. (2022). The quizzing effect depends on hope of success and can be optimized by cognitive load-based adaptation. Learning and Instruction, 77. https://doi.org/10.1016/j.learninstruc.2021.101526
[30] Ifenthaler, D. (2015). Learning Analytics. In J. M. Spector (Ed.), The SAGE encyclopedia of educational technology (Vol. 2, pp. 447-451). SAGE publication. https://madoc.bib.uni-mannheim.de/38809/
[31] Jensen, J. L., McDaniel, M. A., Kummer, T. A., Godoy, P. D. D. M., & St. Clair, B. (2020). Testing effect on high-level cognitive skills. CBE—Life Sciences Education, 19(3). 2020. https://doi.org/10.1187/cbe.19-10-0193
[32] Jost, N. S., Jossen, S. L., Rothen, N., & Martarelli, C. S. (2021). The advantage of distributed practice in a blended learning setting. Education and Information Technologies, 26, 3097-3113. https://doi.org/10.1007/s10639-020-10424-9
[33] Jovanović, J., Gašević, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet and Higher Education, 33, 74-85. https://doi.org/10.1016/j.iheduc.2017.02.001
[34] Karpicke, J., & Bauernschmidt, A. (2011). Spaced retrieval: Absolute spacing enhances learning regardless of relative spacing. Journal of Experimental Psychology. Learning, Memory, and Cognition, 37(5), 1250-1257. https://doi.org/10.1037/a0023436
[35] Karpicke, J., & Roediger, H. (2007). Expanding retrieval practice promotes short-term retention, but equally spaced retrieval enhances long-term retention. Journal of Experimental Psychology. Learning, Memory, and Cognition, 33(4), 704-719. 2007. https://doi.org/10.1037/0278-7393.33.4.704
[36] Khayi, N. A., & Rus, V. (2019). Clustering students based on their prior knowledge. In F. Collin, A. Merceron, & M. Desmarais (Eds.), Proceedings of the 12th International Conference on Educational Data Mining. Université du Québec à Montréal, Polytechnique Montréal (pp. 246-251). https://educationaldatamining.org/edm2019/proceedings/
[37] Kim, D., Yoon, M., Jo, I.-H., & Branch, R. M. (2018). Learning analytics to support self-regulated learning in asynchronous online courses: A case study at a women's university in South Korea. Computers & Education, 127, 233-251. https://doi.org/10.1016/j.compedu.2018.08.023
[38] Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). Guilford Press.
[39] Kossen, C., & Ooi, C.-Y. (2021). Trialling micro-learning design to increase engagement in online courses. Asian Association of Open Universities Journal, 16(3), 299-310. https://doi.org/10.1108/AAOUJ-09-2021-0107
[40] Kovanovic, V., Gasevic, D., Joksimovic, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74-89. https://doi.org/10.1016/j.iheduc.2015.06.002
[41] Lamotte, M., Izaute, M., & Darnon, C. (2021). Can tests improve learning in real university classrooms? Journal of Cognitive Psychology, 33(8), 974-992. https://doi.org/10.1080/20445911.2021.1956939
[42] Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education-A Literature Review. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends. Studies in systems, decision and control (pp. 1-23). Springer. https://doi.org/10.1007/978-3-319-52977-6_1
[43] Li, S., Chen, G., Xing, W., Zheng, J., & Xie, C. (2020). Longitudinal clustering of students' self-regulated learning behaviors in engineering design. Computers & Education, 153, Article 103899. https://doi.org/10.1016/j.compedu.2020.103899
[44] Li, H., Flanagan, B., Konomi, S., & Ogata, H. (2018). Measuring behaviors and identifying indicators of self-regulation in computer-assisted language learning courses. Research and Practice in Technology Enhanced Learning, 13(19), 1-12. https://doi.org/10.1186/s41039-018-0087-7
[45] Mavroudi, A., Giannakos, M., & Krogstie, J. (2018). Supporting adaptive learning pathways through the use of learning analytics: Developments, challenges and future opportunities. Interactive Learning Environments, 26(2), 206-220. https://doi.org/10.1080/10494820.2017.1292531
[46] McDaniel, M. A, Thomas, R. C., Agarwal, P. K., McDermott, K. B., & Roediger, H. L. (2013). Quizzing in middle-school science: Successful transfer performance on classroom exams. Applied Cognitive Psychology, 27(3), 360-372. https://doi.org/10.1002/acp.2914
[47] Mega, C., Ronconi, L., & De Beni, R. (2014). What makes a good student? How emotions, self-regulated learning, and motivation contribute to academic achievement. Journal of Educational Psychology, 106(1), 121-131. https://doi.org/10.1037/a0033546
[48] Minear, M., Coane, J. H., Boland, S. C., Cooney, L. H., & Albat, M. (2018). The benefits of retrieval practice depend on item difficulty and intelligence. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(9), 1474-1486. https://doi.org/10.1037/xlm0000486
[49] Moosbrugger, H., & Kelava, A. (2012). Testtheorie und Fragebogenkonstruktion. Springer. https://doi.org/10.1007/978-3-642-20072-4
[50] Ning, H. K., & Downing, K. (2015). A latent profile analysis of university students' self-regulated learning strategies. Studies in Higher Education, 40(7), 1328-1346. https://doi.org/10.1080/03075079.2014.880832
[51] Norman, G. (2010). Likert scales, levels of measurement and the "laws" of statistics. Advances in Health Sciences Education, 15(5), 625-632. https://doi.org/10.1007/s10459-010-9222-y
[52] Parpala, A., Mattsson, M., Herrmann, K. J., Bager-Elsborg, A., & Hailikari, T. (2021). Detecting the variability in student learning in different disciplines - A person-oriented approach. Scandinavian Journal of Educational Research, 66(6), 1020-1037. https://doi.org/10.1080/00313831.2021.1958256
[53] Perry, N. E., & Winne, P. H. (2006). Learning from Learning Kits: gStudy traces of students' self-regulated engagements with computerized content. Educational Psychology Review, 18, 211-228. https://doi.org/10.1007/s10648-006-9014-3
[54] Pishgar, F., Greifer, N., Leyrat, C., & Stuart, E. (2021). MatchThem: Matching and weighting after multiple imputation. The R Journal, 13(2), 294-305. https://doi.org/10.32614/RJ-2021-073
[55] R Core Team (2021). R: A language and environment for statistical computing. R foundation for statistical computing. Retrieved from https://www.R-project.org/
[56] Revelle, W. R. (2022). psych: Procedures for psychological, psychometric, and personality research. R package version 2.2.5. Northwestern University. Retrieved from https://CRAN.R-project.org/package=psych
[57] Robey, A. (2019). The benefits of testing: Individual differences based on student factors. Journal of Memory and Language, 108. https://doi.org/10.1016/j.jml.2019.104029
[58] Robitzsch, A. (2020). Why ordinal variables can (almost) always be treated as continuous variables: Clarifying assumptions of robust continuous and ordinal factor analysis estimation methods. Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.589965
[59] Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. https://doi.org/10.18637/jss.v048.i02
[60] Schwieren, J., Barenberg, J., & Dutke, S. (2017). The testing effect in the psychology classroom: A meta-analytic perspective. Psychology Learning & Teaching, 16(2), 179-196. https://doi.org/10.1177/1475725717695149
[61] Shin, D., & Shim, J. (2021). A systematic review on data mining for mathematics and science education. International Journal of Science and Mathematics Education, 19, 639-659. https://doi.org/10.1007/s10763-020-10085-7
[62] Soderstrom, N. C., & Bjork, R. A. (2014). Testing facilitates the regulation of subsequent study time. Journal of Memory and Language, 73, 99-115. https://doi.org/10.1016/j.jml.2014.03.003
[63] Sun, Z., & Xie, K. (2020). How do students prepare in the pre-class setting of a flipped undergraduate math course? A latent profile analysis of learning behavior and the impact of achievement goals. The Internet and Higher Education, 46. https://doi.org/10.1016/j.iheduc.2020.100731
[64] Szpunar, K. K., McDermott, K. B., & Roediger, H. L. III. (2008). Testing during study insulates against the buildup of proactive interference. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(6), 1392-1399. https://doi.org/10.1037/a0013082
[65] van Alten, D. C. D., Phielix, C., Janssen, J., & Kester, L. (2021). Secondary students' online self-regulated learning during flipped learning: A latent profile analysis. Computers in Human Behavior, 118. https://doi.org/10.1016/j.chb.2020.106676
[66] van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45(3), 1-67. https://doi.org/10.18637/jss.v045.i03
[67] Vanslambrouck, S., Zhu, C., Pynoo, B., Lombaerts, K., Tondeur, J., & Scherer, R. (2019). A latent profile analysis of adult students' online self-regulation in blended learning environments. Computers in Human Behavior, 99, 126-136. https://doi.org/10.1016/j.chb.2019.05.021
[68] Waspada, I., Bahtiar, N., & Wibowo, A. (2019). Clustering student behavior based on quiz activities on moodle LMS to discover the relation with a final exam score. Journal of Physics: Conference Series, 1217. https://doi.org/10.1088/1742-6596/1217/1/012118
[69] Wissman, K. T., Rawson, K. A., & Pyc, M. A. (2011). The interim test effect: Testing prior material can facilitate the learning of new material. Psychonomic Bulletin & Review, 18, 1140-1147. https://doi.org/10.3758/s13423-011-0140-7
[70] Yang, C., Luo, L., Vadillo, M. A., Yu, R., & Shanks, D. R. (2021). Testing (quizzing) boosts classroom learning: A systematic and meta-analytic review. Psychological Bulletin, 147(4), 399-435. https://doi.org/10.1037/bul0000309
[71] Yang, C., Potts, R., & Shanks, D. R. (2017). The forward testing effect on self-regulated study time allocation and metamemory monitoring. Journal of Experimental Psychology: Applied, 23(3), 263-277. https://doi.org/10.1037/xap0000122
[72] Yerkes, R. M., & Dodson, J. D. (1908). The Relation of Strength of Stimulus to Rapidity of Habit Formation. Journal of Comparative Neurology & Psychology, 18, 459-482. https://doi.org/10.1002/cne.920180503
[73] Zheng, J., Xing, W., Zhu, G., Chen, G., Zhao, H., & Xie, C. (2020). Profiling self-regulation behaviors in STEM learning of engineering design. Computers & Education, 143. https://doi.org/10.1016/j.compedu.2019.103669