Title: Decoding student success in higher education : a comparative study on learning strategies of undergraduate and graduate students
Source document: Studia paedagogica. 2023, vol. 28, iss. 3, pp. [59]-87
Extent
[59]-87
-
ISSN1803-7437 (print)2336-4521 (online)
Persistent identifier (DOI): https://doi.org/10.5817/SP2023-3-3
Stable URL (handle): https://hdl.handle.net/11222.digilib/digilib.79679
Type: Article
Language
License: Not specified license
Rights access
open access
Notice: These citations are automatically created and might not follow citation rules properly.
Abstract(s)
Learning management systems (LMS) provide a rich source of data about the engagement of students with courses and their materials that tends to be underutilized in practice. In this paper, we use data collected from the LMS to uncover learning strategies adopted by students and compare their effectiveness. Starting from a sample of over 11,000 enrollments at a Portuguese information management school, we extracted features indicative of self-regulated learning (SRL) behavior from the associated interactions. Then, we employed an unsupervised machine learning algorithm (k-means) to group students according to the similarity of their patterns of interaction. This process was conducted separately for undergraduate and graduate students. Our analysis uncovered five distinct learning strategy profiles at both the undergraduate and graduate levels: 1) active, prolonged and frequent engagement; 2) mildly frequent and task-focused engagement; 3) mildly frequent, mild activity in short sessions engagement; 4) likely procrastinators; and 5) inactive. Mapping strategies with the students' final grades, we found that students at both levels who accessed the LMS early and frequently had better outcomes. Conversely, students who exhibited procrastinating behavior had worse end-of-course grades. Interestingly, the relative effectiveness of the various learning strategies was consistent across instruction levels. Despite the LMS offering an incomplete and partial view of the learning processes students employ, these findings suggest potentially generalizable relationships between online student behaviors and learning outcomes. While further validation with new data is necessary, these connections between online behaviors and performance could guide the development of personalized, adaptive learning experiences.
References
[1] Aljohani, N. R., Fayoumi, A., & Hassan, S.-U. (2019). Predicting at-risk students using clickstream data in the virtual learning environment. Sustainability, 11(24), Article 7238. https://doi.org/10.3390/su11247238
[2] Baker, R., Xu, D., Park, J., Yu, R., Li, Q., Cung, B., Fischer, C., Rodriguez, F., Warschauer, M., & Smyth, P. (2020). The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: Opening the black box of learning processes. International Journal of Educational Technology in Higher Education, 17(1), Article 13. https://doi.org/10.1186/s41239-020-00187-1
[3] Bellur, S., Nowak, K. L., & Hull, K. S. (2015). Make it our time: In class multitaskers have lower academic performance. Computers in Human Behavior, 53, 63-70. https://doi.org/10.1016/j.chb.2015.06.027
[4] Bernacki, M. L., Chavez, M. M., & Uesbeck, P. M. (2020). Predicting achievement and providing support before STEM majors begin to fail. Computers & Education, 158, Article 103999. https://doi.org/10.1016/j.compedu.2020.103999
[5] Biggs, J. B. (1987). Student approaches to learning and studying. Study process questionnaire manual. Australian Council for Educational Research.
[6] Boekaerts, M. (1997). Self-regulated learning: A new concept embraced by researchers, policy makers, educators, teachers, and students. Learning and Instruction, 7(2), 161-186. https://doi.org/10.1016/S0959-4752(96)00015-1
[7] Broadbent, J. (2017). Comparing online and blended learner's self-regulated learning strategies and academic performance. The Internet and Higher Education, 33, 24-32. https://doi.org/10.1016/j.iheduc.2017.01.004
[8] Broadbent, J., & Poon, W. L. (2015). Self-regulated learning strategies & academic achievement in online higher education learning environments: A Systematic Review. The Internet and Higher Education, 27, 1-13. http://dx.doi.org/10.1016/j.iheduc.2015.04.007
[9] Çebi, A., & Güyer, T. (2020). Students' interaction patterns in different online learning activities and their relationship with motivation, self-regulated learning strategy and learning performance. Education and Information Technologies, 25(5), 3975-3993. https://doi.org/10.1007/s10639-020-10151-1
[10] Cerezo, R., Sánchez-Santillán, M., Paule-Ruiz, M. P., & Núñez, J. C. (2016). Students' LMS interaction patterns and their relationship with achievement: A case study in higher education. Computers & Education, 96, 42-54. https://doi.org/10.1016/j.compedu.2016.02.006
[11] Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary Education and Management, 11(1), 19-36. https://doi.org/10.1007/s11233-004-3567-9
[12] Conijn, R., Snijders, C., Kleingeld, A., & Matzat, U. (2017). Predicting student performance from LMS data: A comparison of 17 blended courses using Moodle LMS. IEEE Transactions on Learning Technologies, 10(1), 17-29. https://doi.org/10.1109/TLT.2016.2616312
[13] Gašević, D., Jovanović, J., Pardo, A., & Dawson, S. (2017). Detecting learning strategies with analytics: Links with self-reported measures and academic performance. Journal of Learning Analytics, 4(2), 113-128. http://dx.doi.org/10.18608/jla.2017.42.10
[14] Holmes, W., & Tuomi, I. (2022). State of the art and practice in AI in education. European Journal of Education, 57(4), 542-570. https://doi.org/10.1111/ejed.12533
[15] Hung, J.-L., & Zhang, K. (2008). Revealing online learning behaviors and activity patterns and making predictions with data mining techniques in online teaching. Journal of Online Learning and Teaching, 4(4), 426-437.
[16] Jarque, C. M., & Bera, A. K. (1980). Efficient tests for normality, homoscedasticity and serial independence of regression residuals. Economics Letters, 6(3), 255-259.
[17] Li, L.-Y., & Tsai, C.-C. (2017). Accessing online learning material: Quantitative behavior patterns and their effects on motivation and learning performance. Computers & Education, 114, 286-297. https://doi.org/10.1016/j.compedu.2017.07.007
[18] Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an "early warning system" for educators: A proof of concept. Computers & Education, 54(2), 588-599. https://doi.org/10.1016/j.compedu.2009.09.008
[19] Macqueen, J. (1967). Some methods for classification and analysis of multivariate observations. In L. M. Le Cam, & J. Neyman (Eds.), Berkeley Symposium on Mathematical Statistics and Probability (s. 281-297). University of California Press. https://www.cs.cmu.edu/~bhiksha/courses/mlsp.fall2010/class14/macqueen.pdf
[20] Matcha, W., Gašević, D., Jovanović, J., Uzir, N. A., Oliver, C. W., Murray, A., & Gasevic, D. (2020). Analytics of learning strategies: The association with the personality traits. LAK ‘20: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 151-160. https://doi.org/10.1145/3375462.3375534
[21] McKinney, W. (2017). Python for data analysis: Data wrangling with pandas, NumPy, and IPython (2nd Ed.). O'Reilly Media.
[22] Moubayed, A., Injadat, M., Shami, A., & Lutfiyya, H. (2020). Student engagement level in an e-learning environment: Clustering using k-means. American Journal of Distance Education, 34(2), 137-156. https://doi.org/10.1080/08923647.2020.1696140
[23] Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, Article 422. https://doi.org/10.3389/fpsyg.2017.00422
[24] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., & Cournapeau, D. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(85), 2825-2830.
[25] Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the motivated strategies for learning questionnaire (MSLQ). The Regents of The University of Michigan.
[26] Puzziferro, M. (2008). Online technologies self-efficacy and self-regulated learning as predictors of final grade and satisfaction in college-level online courses. American Journal of Distance Education, 22(2), 72-89. https://doi.org/10.1080/08923640802039024
[27] Riestra-González, M., Paule-Ruíz, M. del P., & Ortin, F. (2021). Massive LMS log data analysis for the early prediction of course-agnostic student performance. Computers & Education, 163, Article 104108. https://doi.org/10.1016/j.compedu.2020.104108
[28] Romero, C., Espejo, P. G., Zafra, A., Romero, J. R., & Ventura, S. (2013). Web usage mining for predicting final marks of students that use Moodle courses. Computer Applications in Engineering Education, 21(1), 135-146. https://doi.org/10.1002/cae.20456
[29] Santos, R. M., & Henriques, R. (2023). Accurate, timely, and portable: Course-agnostic early prediction of student performance from LMS logs. Computers and Education: Artificial Intelligence, 5, Article 100175. https://doi.org/10.1016/j.caeai.2023.100175
[30] Susac, A., Bubic, A., Kaponja, J., Planinic, M., & Palmovic, M. (2014). Eye movements reveal students' strategies in simple equation solving. International Journal of Science and Mathematics Education, 12(3), 555-577. https://doi.org/10.1007/s10763-014-9514-4
[31] Trilling, B., & Fadel, C. (2009). 21st century skills: Learning for life in our times. John Wiley & Sons.
[32] Winne, P. H., & Jamieson-Noel, D. (2002). Exploring students' calibration of self reports about study tactics and achievement. Contemporary Educational Psychology, 27(4), 551-572.
[33] Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of Self-Regulation (pp. 531-566). Academic Press. https://doi.org/10.1016/B978-012109890-2/50045-7
[34] Wu, X., Kumar, V., Quinlan, J. R., Ghosh, J., Yang, Q., Motoda, H., McLachlan, G. J., Ng, A., Liu, B., Yu, P. S., Zhou, Z.-H., Steinbach, M., Hand, D. J., & Steinberg, D. (2008). Top 10 algorithms in data mining. Knowledge and Information Systems, 14(1), 1-37. https://doi.org/10.1007/s10115-007-0114-2
[35] Yang, Y., Hooshyar, D., Pedaste, M., Wang, M., Huang, Y.-M., & Lim, H. (2020). Predicting course achievement of university students based on their procrastination behaviour on Moodle. Soft Computing, 24(24), 18777-18793. https://doi.org/10.1007/s00500-020-05110-4
[36] Yeo, I.-K., & Johnson, R. A. (2000). A new family of power transformations to improve normality or symmetry. Biometrika, 87(4), 954-959. https://doi.org/10.1093/biomet/87.4.954
[37] Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of Self-Regulation (pp. 13-39). Academic Press. https://doi.org/10.1016/B978-012109890-2/50031-7
[38] Zimmerman, B. J. (2002). Becoming a self-regulated learner: An Overview. Theory Into Practice, 41(2), 64-70. https://doi.org/10.1207/s15430421tip4102_2
[39] Zimmerman, B. J., & Moylan, A. R. (2009). Self-regulation: Where metacognition and motivation intersect. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Handbook of Metacognition in Education (pp. 299-315). Routledge.
[2] Baker, R., Xu, D., Park, J., Yu, R., Li, Q., Cung, B., Fischer, C., Rodriguez, F., Warschauer, M., & Smyth, P. (2020). The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: Opening the black box of learning processes. International Journal of Educational Technology in Higher Education, 17(1), Article 13. https://doi.org/10.1186/s41239-020-00187-1
[3] Bellur, S., Nowak, K. L., & Hull, K. S. (2015). Make it our time: In class multitaskers have lower academic performance. Computers in Human Behavior, 53, 63-70. https://doi.org/10.1016/j.chb.2015.06.027
[4] Bernacki, M. L., Chavez, M. M., & Uesbeck, P. M. (2020). Predicting achievement and providing support before STEM majors begin to fail. Computers & Education, 158, Article 103999. https://doi.org/10.1016/j.compedu.2020.103999
[5] Biggs, J. B. (1987). Student approaches to learning and studying. Study process questionnaire manual. Australian Council for Educational Research.
[6] Boekaerts, M. (1997). Self-regulated learning: A new concept embraced by researchers, policy makers, educators, teachers, and students. Learning and Instruction, 7(2), 161-186. https://doi.org/10.1016/S0959-4752(96)00015-1
[7] Broadbent, J. (2017). Comparing online and blended learner's self-regulated learning strategies and academic performance. The Internet and Higher Education, 33, 24-32. https://doi.org/10.1016/j.iheduc.2017.01.004
[8] Broadbent, J., & Poon, W. L. (2015). Self-regulated learning strategies & academic achievement in online higher education learning environments: A Systematic Review. The Internet and Higher Education, 27, 1-13. http://dx.doi.org/10.1016/j.iheduc.2015.04.007
[9] Çebi, A., & Güyer, T. (2020). Students' interaction patterns in different online learning activities and their relationship with motivation, self-regulated learning strategy and learning performance. Education and Information Technologies, 25(5), 3975-3993. https://doi.org/10.1007/s10639-020-10151-1
[10] Cerezo, R., Sánchez-Santillán, M., Paule-Ruiz, M. P., & Núñez, J. C. (2016). Students' LMS interaction patterns and their relationship with achievement: A case study in higher education. Computers & Education, 96, 42-54. https://doi.org/10.1016/j.compedu.2016.02.006
[11] Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary Education and Management, 11(1), 19-36. https://doi.org/10.1007/s11233-004-3567-9
[12] Conijn, R., Snijders, C., Kleingeld, A., & Matzat, U. (2017). Predicting student performance from LMS data: A comparison of 17 blended courses using Moodle LMS. IEEE Transactions on Learning Technologies, 10(1), 17-29. https://doi.org/10.1109/TLT.2016.2616312
[13] Gašević, D., Jovanović, J., Pardo, A., & Dawson, S. (2017). Detecting learning strategies with analytics: Links with self-reported measures and academic performance. Journal of Learning Analytics, 4(2), 113-128. http://dx.doi.org/10.18608/jla.2017.42.10
[14] Holmes, W., & Tuomi, I. (2022). State of the art and practice in AI in education. European Journal of Education, 57(4), 542-570. https://doi.org/10.1111/ejed.12533
[15] Hung, J.-L., & Zhang, K. (2008). Revealing online learning behaviors and activity patterns and making predictions with data mining techniques in online teaching. Journal of Online Learning and Teaching, 4(4), 426-437.
[16] Jarque, C. M., & Bera, A. K. (1980). Efficient tests for normality, homoscedasticity and serial independence of regression residuals. Economics Letters, 6(3), 255-259.
[17] Li, L.-Y., & Tsai, C.-C. (2017). Accessing online learning material: Quantitative behavior patterns and their effects on motivation and learning performance. Computers & Education, 114, 286-297. https://doi.org/10.1016/j.compedu.2017.07.007
[18] Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an "early warning system" for educators: A proof of concept. Computers & Education, 54(2), 588-599. https://doi.org/10.1016/j.compedu.2009.09.008
[19] Macqueen, J. (1967). Some methods for classification and analysis of multivariate observations. In L. M. Le Cam, & J. Neyman (Eds.), Berkeley Symposium on Mathematical Statistics and Probability (s. 281-297). University of California Press. https://www.cs.cmu.edu/~bhiksha/courses/mlsp.fall2010/class14/macqueen.pdf
[20] Matcha, W., Gašević, D., Jovanović, J., Uzir, N. A., Oliver, C. W., Murray, A., & Gasevic, D. (2020). Analytics of learning strategies: The association with the personality traits. LAK ‘20: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 151-160. https://doi.org/10.1145/3375462.3375534
[21] McKinney, W. (2017). Python for data analysis: Data wrangling with pandas, NumPy, and IPython (2nd Ed.). O'Reilly Media.
[22] Moubayed, A., Injadat, M., Shami, A., & Lutfiyya, H. (2020). Student engagement level in an e-learning environment: Clustering using k-means. American Journal of Distance Education, 34(2), 137-156. https://doi.org/10.1080/08923647.2020.1696140
[23] Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, Article 422. https://doi.org/10.3389/fpsyg.2017.00422
[24] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., & Cournapeau, D. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(85), 2825-2830.
[25] Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the motivated strategies for learning questionnaire (MSLQ). The Regents of The University of Michigan.
[26] Puzziferro, M. (2008). Online technologies self-efficacy and self-regulated learning as predictors of final grade and satisfaction in college-level online courses. American Journal of Distance Education, 22(2), 72-89. https://doi.org/10.1080/08923640802039024
[27] Riestra-González, M., Paule-Ruíz, M. del P., & Ortin, F. (2021). Massive LMS log data analysis for the early prediction of course-agnostic student performance. Computers & Education, 163, Article 104108. https://doi.org/10.1016/j.compedu.2020.104108
[28] Romero, C., Espejo, P. G., Zafra, A., Romero, J. R., & Ventura, S. (2013). Web usage mining for predicting final marks of students that use Moodle courses. Computer Applications in Engineering Education, 21(1), 135-146. https://doi.org/10.1002/cae.20456
[29] Santos, R. M., & Henriques, R. (2023). Accurate, timely, and portable: Course-agnostic early prediction of student performance from LMS logs. Computers and Education: Artificial Intelligence, 5, Article 100175. https://doi.org/10.1016/j.caeai.2023.100175
[30] Susac, A., Bubic, A., Kaponja, J., Planinic, M., & Palmovic, M. (2014). Eye movements reveal students' strategies in simple equation solving. International Journal of Science and Mathematics Education, 12(3), 555-577. https://doi.org/10.1007/s10763-014-9514-4
[31] Trilling, B., & Fadel, C. (2009). 21st century skills: Learning for life in our times. John Wiley & Sons.
[32] Winne, P. H., & Jamieson-Noel, D. (2002). Exploring students' calibration of self reports about study tactics and achievement. Contemporary Educational Psychology, 27(4), 551-572.
[33] Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of Self-Regulation (pp. 531-566). Academic Press. https://doi.org/10.1016/B978-012109890-2/50045-7
[34] Wu, X., Kumar, V., Quinlan, J. R., Ghosh, J., Yang, Q., Motoda, H., McLachlan, G. J., Ng, A., Liu, B., Yu, P. S., Zhou, Z.-H., Steinbach, M., Hand, D. J., & Steinberg, D. (2008). Top 10 algorithms in data mining. Knowledge and Information Systems, 14(1), 1-37. https://doi.org/10.1007/s10115-007-0114-2
[35] Yang, Y., Hooshyar, D., Pedaste, M., Wang, M., Huang, Y.-M., & Lim, H. (2020). Predicting course achievement of university students based on their procrastination behaviour on Moodle. Soft Computing, 24(24), 18777-18793. https://doi.org/10.1007/s00500-020-05110-4
[36] Yeo, I.-K., & Johnson, R. A. (2000). A new family of power transformations to improve normality or symmetry. Biometrika, 87(4), 954-959. https://doi.org/10.1093/biomet/87.4.954
[37] Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of Self-Regulation (pp. 13-39). Academic Press. https://doi.org/10.1016/B978-012109890-2/50031-7
[38] Zimmerman, B. J. (2002). Becoming a self-regulated learner: An Overview. Theory Into Practice, 41(2), 64-70. https://doi.org/10.1207/s15430421tip4102_2
[39] Zimmerman, B. J., & Moylan, A. R. (2009). Self-regulation: Where metacognition and motivation intersect. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Handbook of Metacognition in Education (pp. 299-315). Routledge.