DEVELOPMENT AND VALIDATION OF AN INSTRUMENT FOR ASSESSING STUDENTS' PERCEIVED EFFECTIVENESS OF AI-ENHANCED ADVANCED MATHEMATICS COURSES
DOI:
https://doi.org/10.35631/IJEPC.1164075Keywords:
AI-enhanced Advanced Mathematics, Content Validity, Instrument Development, Pilot Study, Reliability, Students’ Perceived EffectivenessAbstract
This study developed a measurement tool to assess students’ perceived effectiveness of an AI-enhanced advanced mathematics course and verified its validity and reliability. As artificial intelligence (AI) technologies become more prevalent in higher education, AI-supported learning environments have emerged as a promising approach to enhancing educational quality and enriching students’ learning experiences. Nevertheless, at present, existing research has yet to provide adequately developed quantitative measurement tools capable of assessing students’ perceptions of the effectiveness of AI-enhanced advanced mathematics courses and their learning outcomes. Therefore, based on prior research and educational theory, this study developed a measurement tool to evaluate students’ perceived learning outcomes in an AI-enhanced advanced mathematics learning environment. This instrument comprises four dimensions: ‘improvement in mathematical understanding’, ‘development of mathematical thinking skills’, ‘changes in motivation to learn’, and ‘satisfaction and willingness to recommend’. A convenience sampling method was used for the preliminary study, which involved 98 undergraduate students enrolled on an advanced mathematics course utilising AI. Five experts in educational technology and mathematics education were invited to assess the instrument’s content validity through the Content Validity Index (CVI). The results showed that the average Content Validity Index for the entire scale (S-CVI/Ave) was 0.85, confirming sufficient content validity. Internal consistency reliability was then checked using Cronbach’s α coefficients. The reliability analysis revealed that all four dimensions showed acceptable to strong consistency, with α values ranging from 0.754 to 0.866. The results offer initial empirical support for the reliability and validity of the proposed tool for assessing students’ perceived effectiveness of the AI-enhanced advanced mathematics course. This instrument can assist educators and higher education institutions in evaluating students’ learning experiences, identifying areas for improvement, and optimising AI-enhanced mathematics teaching practices. In future research, it will be necessary to conduct exploratory and confirmatory factor analyses using a larger and more diverse sample to further validate the measurement instrument.
Downloads
References
Ahmad, K. A., & Idris, I. (2025). A CEFR‑Informed English Test: How Content Validity Index and Content Validity Ratio are used for Content Validation. EDUCATUM Journal of Social Sciences, 11(1). https://doi.org/10.37134/ejoss.vol11.1.7.2025
Akpur, U., Alci, B., & Karatas, H. (2016). Evaluation of the Curriculum of English Preparatory Classes at Yildiz Technical University Using CIPP Model. Educational research and Reviews, 11(7), 466-473.
Becker, K., & Park, K. (2011). Effects of integrative approaches among science, technology, engineering, and mathematics (STEM) subjects on students’ learning: A preliminary meta‑analysis. Journal of STEM Education, 12(5/6), 23–37.
Bond, M., Khosravi, H., De Laat, M., et al. (2024). A meta systematic review of artificial intelligence in higher education: A call for increased ethics, collaboration, and rigour. International Journal of Educational Technology in Higher Education, 21, 4. https://doi.org/10.1186/s41239-023-00436-z
Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications.
Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: the state of the field. International journal of educational technology in higher education, 20(1), 1-22.
DeVellis, R. F., & Thorpe, C. T. (2021). Scale development: Theory and applications. Sage publications.
Eti, N. E., Mosia, M. M., & Egara, F. O. (2026). The role of AI‑driven personalised learning in enhancing mathematics problem‑solving skills: A systematic review. Frontiers in Computer Science. https://doi.org/10.3389/fcomp.2026.1813431
Etikan, I., Musa, S. A., & Alkassim, R. S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1–4. https://doi.org/10.11648/j.ajtas.20160501.11
Haynes, S. N., Richard, D. C. S., & Kubany, E. S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7(3), 238–247. https://doi.org/10.1037/1040‑3590.7.3.238
Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial intelligence in education: Promises and implications for teaching and learning. Center for Curriculum Redesign. https://discovery.ucl.ac.uk/id/eprint/10139722/
Ifenthaler, D., & Schweinbenz, V. (2013). The acceptance of tablet PCs in classroom instruction: The teachers’ perspectives. Computers in Human Behavior, 29(3), 525–534. https://doi.org/10.1016/j.chb.2012.11.004
Ingkavara, T., Wongkia, W., & Panjaburee, P. (2023). Trends of adaptive/personalized learning and intelligent tutoring systems in mathematics: A review of academic publications from 2010 to 2022. Engineering Proceedings, 55(1), 34. https://doi.org/10.3390/engproc2023055034
Ji, L., Meng, ., Dai, Y., & Wang, B. (2024). Exploring the promotion of university informatization teaching reform through modern educational technology. Educational Theory and Research, 2(4), 58–60. https://doi.org/10.61369/etr.6423
Joshi, A., Kale, S., Chandel, S., & Pal, D. K. (2015). Likert scale: Explored and explained. British Journal of Applied Science & Technology, 7(4), 396–403. https://doi.org/10.9734/BJAST/2015/14975
Li, K. (2025). Factors influencing learners’ learning interest when using AI chatbots‑assisted math learning in higher education. Frontiers in Psychology, 16, 1716913. https://doi.org/10.3389/fpsyg.2025.1716913
Li, M. Y. (2023). Research on the application of artificial intelligence technology in ideological and political education in colleges and universities (Master’s thesis, China University of Mining and Technology). https://doi.org/10.27623/d.cnki.gzkyu.2023.002624
Li, Z., & Li, M. (2025). Research on talent cultivation in advanced mathematics education through artificial intelligence. Journal of Contemporary Educational Research, 9(5), 133–140. https://doi.org/10.26689/jcer.v9i4.9890
Liu, B., Zhang, W., & Wang, F. (2026). Can generative artificial intelligence effectively enhance students’ mathematics learning outcomes?—A meta‑analysis of empirical studies from 2023 to 2025. Education Sciences, 16(1), 140. https://doi.org/10.3390/educsci16010140
Merino-Campos, C. (2025). The impact of artificial intelligence on personalized learning in higher education: A systematic review. Trends in Higher Education, 4(2), 17. https://doi.org/10.3390/higheredu4020017
Ministry of Education of the People’s Republic of China. (2018). Education Informatization 2.0 Action Plan .
Ministry of Education of the People’s Republic of China. (2020). Artificial Intelligence Empowerment Education Action Plan (2020–2025)
Othman, N., Ewan, M. E. M., & Wan Yunus, F. (2024). Artificial Intelligence in Educational Measurement: A Bibliometric Review (1997 to 2024). Journal of Applied Science, Engineering, Technology and Education. https://ascijournal.eu/index.php/asci/article/view/4126
Owan, V. J., Abang, K. B., Idika, D. O., Etta, E. O., & Bassey, B. A. (2023). Exploring the potential of artificial intelligence tools in educational measurement and assessment. Eurasia Journal of Mathematics, Science and Technology Education, 19(8), em2307. https://www.ejmste.com/article/exploring-the-potential-of-artificial-intelligence-tools-in-educational-measurement-and-assessment-13428
Peng, J., & Li, Y. (2025). Frontiers of artificial intelligence for personalized learning in higher education: A systematic review of leading articles. Applied Sciences, 15(18), 10096. https://doi.org/10.3390/app151810096
Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497. https://doi.org/10.1002/nur.20147
Rahmat, R. A., Saidi, S. B., & Mohd Nasir, N. S. (2024). Content validity of Digital Knowledge using CVI method. Environment‑Behaviour Proceedings Journal, 9(SI20), 21–28. https://doi.org/10.21834/e-bpj.v9iSI20.6092
Revilla, M., Saris, W. E., & Krosnick, J. A. (2014). Choosing the number of categories in agree–disagree scales. Sociological Methods & Research, 43(1), 73–97. https://doi.org/10.1177/0049124113509605
Romero Jeldres, M., Díaz Costa, E., & Nadim, T. F. (2023). A review of Lawshe’s method for calculating content validity in the social sciences. Frontiers in Education, 8, Article 1271335. https://doi.org/10.3389/feduc.2023.1271335
Son, T. (2024). Intelligent tutoring systems in mathematics education: A systematic literature review using the substitution, augmentation, modification, redefinition model. Computers, 13(10), 270. https://doi.org/10.3390/computers13100270
Song, X. L. (2024). Construction and empirical study of a quality evaluation indicator system for after-school physical education services in secondary schools based on the CIPP model (Master’s thesis, Guangxi Normal University, China). CNKI. https://doi.org/10.27036/d.cnki.ggxsu.2024.000020
Wang, F., & Sahid, S. (2024). Content validation and content validity index calculation for entrepreneurial behavior instruments among vocational college students in China. Multidisciplinary Reviews, 7(9), 2024187. https://doi.org/10.31893/multirev.2024187
Yusoff, M. S. B. (2019). ABC of content validation and content validity index calculation. Education in Medicine Journal, 11(2), 49‑54. https://doi.org/10.21315/eimj2019.11.2.6
Zhang, L., Wang, H., & Chen, Y. (2024). A thematic analysis of current research on AI-empowered mathematics teaching in Chinese universities from the TPACK-theoretical perspective. Education Reform and Development, 6(2), 88–96. https://doi.org/10.26689/erd.v7i3.9676
Zhang, X. (2023). An Innovative Model of Advanced Mathematics Curriculum Education Incorporating Artificial Intelligence Technology. Applied Mathematics and Nonlinear Sciences, 9(1), 2023. https://doi.org/10.2478/amns.2023.2.01524
Zhao, X. (2025). Exploration of the path for constructing smart classroom teaching models in colleges and universities in the era of artificial intelligence. Reform and Opening, (10), 72–76 + 83. https://doi.org/10.16653/j.cnki.32-1034/f.2025.010.009
