RELIABILITY AND VALIDITY OF THE MALAYSIAN POLYTECHNIC LECTURERS’ ASSESSMENT LITERACY TEST INSTRUMENT (UL3PM) USING THE RASCH MODEL: A PILOT STUDY
DOI:
https://doi.org/10.35631/IJEPC.1164057Keywords:
Assessment Literacy, Rasch Model, Reliability, UL3PM, ValidityAbstract
This study investigated the reliability and validity of the Malaysian Polytechnic Lecturer Assessment Literacy Test (UL3PM) instrument. The pilot study involved 33 lecturers from Polytechnic Tun Syed Nasir Syed Ismail (PTSN). The instrument contains two sections of questions. Section A includes demographic information, and Part B has 60 multiple-choice questions related to assessment literacy that encompass six dimensions: Assessment Basics; Assessment Planning; Assessment Development; Assessment Implementation, Assessment and Assessment Improvement; and Equity and Ethics in Assessment implemented in Malaysian polytechnics. The data obtained were analysed using the Rasch Model to assess the reliability and validity of the Malaysian Polytechnic Lecturer Assessment Literacy Test (UL3PM) instrument. Overall, it was found that the UL3PM instrument has high item and individual reliability with values of 0.87 and 0.91, respectively. The validity of the UL3PM instrument was investigated through item fit, individual fit, variable maps, and unidimensionality. The study found that the UL3PM instrument is reliable and valid for measuring assessment literacy among Malaysian polytechnic lecturers with minor adjustments to misfitting and redundant items.
Downloads
References
Afshar, H. S., & Ranjbar, N. (2021). EAP teachers’ assessment literacy: From theory to practice. Studies in Educational Evaluation, 70, 101042.
Ahn, I., & Bong, M. (2020). Self-efficacy and self-concept as predictors of teacher-student relationships and academic achievement: A longitudinal study. Journal of Educational Psychology, 112(4), 715–729. https://doi.org/10.1037/edu0000385
Anggara, D. S., & Abdillah, C. (2023). Content validity analysis of literacy assessment instruments. Cakrawala Pendidikan: Jurnal Ilmiah Pendidikan, 42(2), 447-459.
Aria, D., Sukyadi, D., & Kurniawan, E. (2021). Teacher assessment literacy: Indonesian EFL secondary teachers’ self-perceived on classroom-based assessment practice. English Review: Journal of English Education, 10(1), 15-26.
Arjana, I. M., Parmiti, D. P., Candiasa, I. M., & Widiartini, N. K. (2025). Unlocking the predictive power of the Rasch model: A systematic literature review on educational instrument calibration and assessment accuracy. Edelweiss Applied Science and Technology, 9(6), 2003–2022.
Asres, A. M., & Abebe, A. S. (2024). Teachers’ Competence in The Educational Assessment of Students in A Competence Based Education in Amhara Region, Ethiopia: The Case of TVET Teachers and Assessors.
Atjonen, P., Pöntinen, S., Kontkanen, S., and Ruotsalainen, P. (2022). In enhancing preservice teachers' assessment literacy: focus on knowledge base, conceptions of assessment, and teacher learning. Front. Educ. 7, 891391. doi: 10.3389/feduc.2022.891391
Bøhn, H., & Tsagari, D. (2021). Teacher educators’ conceptions of language assessment literacy in Norway.
Bond, T.G. and C.M. Fox, 2007. Applying the Rasch Model. Fundamental Measurement in the Human Sciences. 2nd Edn., Lawrence Erlbaum Associates Publishers, New Jersey, ISBN: 0805854622.
Bond, T.G. and C.M. Fox, 2007. Applying the Rasch Model. Fundamental Measurement in the Human Sciences. 2nd Edn., Lawrence Erlbaum Associates Publishers, New Jersey, ISBN: 0805854622.
Boone, W. J. (2021). Rasch analysis for instrument development: Why, when, and how? CBE—Life Sciences Education, 20(1), rm1. https://doi.org/10.1187/cbe.20-04-0076
Brown, H. D., & Abeywickrama, P. (2019). Language assessment: Principles and classroom practices(3rded.).PearsonEducation. https://www.academia.edu/39088154/Language_Assessment_Principles_and_Classroom_Practices_H_Douglas_Brown_H_Douglas_Brown_Priyanvada_Abeywickrama
Chong, J., Mokshein, S. E., & Mustapha, R. (2021). A Content Validity Study for Vocational Teachers’ Assessment Literacy Instrument (VOTAL). International Journal of Academic Research in Business and Social Sciences, 11(4). https://doi.org/10.6007/ijarbss/v11-i4/9547
Christoforidou, M., and Kyriakides, L. (2021). Developing teacher assessment skills: the impact of the dynamic approach to teacher professional development. Stud. Educ. Eval. 70, 101051. doi: 10.1016/j.stueduc.2021.101051
Diao, J., & Hu, K. (2022). Preparing TVET teachers for sustainable development in the information age: Development and application of the TVET teachers’ teaching competency scale. Sustainability, 14(18), 11361.
Diao, J., Han, X., Zhou, Q., & Wang, Y. (2023). Professional Competencies in TVET: Framework, Indicators and Assessment Instrument. In Handbook of Technical and Vocational Teacher Professional Development in the Digital Age (pp. 41-73). Singapore: Springer Nature Singapore.
Fitriyah, I., Masitoh, F., & Widiati, U. (2022). Classroom-based language assessment literacy and professional development need between novice and experienced EFL teachers. Indonesian Journal of Applied Linguistics, 12(1), 124-134.
Gan, L., & Lam, R. (2022). A review on language assessment literacy: Trends, foci and contributions. Language Assessment Quarterly, 19(5), 503-525.
Giraldo, F. (2021). A reflection on initiatives for teachers’ professional development through language assessment literacy. Profile Issues in TeachersProfessional Development, 23(1), 197-213.
Giraldo, F. (2021a). Language assessment literacy and teachers’ professional development: A review of the literature. Profile Issues in TeachersProfessional Development, 23(2), 265-279.
Hamzah, F. M., Rashid, M. N. A., Rahman, M. N. A., & Rasul, M. S. (2022). Evaluating the validity and reliability of authentic learning instruments using Rasch Model: A study in Malaysian polytechnics. International Journal of Global Optimization and Its Application, 1(3), 182–189. https://myjms.mohe.gov.my/index.php/ijgoia/article/view/20606
Hannigan, C., Alonzo, D., & Oo, C. Z. (2022). Student assessment literacy: Indicators and domains from the literature. Assessment in Education: Principles, Policy & Practice, 29(4), 482-504.
Hashim, S., Zakariah, S. H., Taufek, F. A., Zulkifli, N. N., Lah, N. H. C., & Murniati, D. E. (2021). An observation on implementation of classroom assessment in technical and vocational education and training (TVET) subject area. Journal of Technical Education and Training, 13(3), 190-200.
Herppich, S., Praetorius, A.-K., Forster, N., Glogger-Frey, I., Karst, K., Leutner, D., et al. (2018). Teachers' assessment competence: integrating knowledge-, process-, and product-oriented approaches into a competence-oriented conceptual model. Teach. Teach. Educ. 76, 181–193. doi: 10.1016/j.tate.2017.12.001
Ismail, N., Abdullah, S., & Othman, N. (2022). Assessment literacy among technical and vocational education and training (TVET) lecturers in Malaysia: Challenges and prospects. Journal of Technical Education and Training, 14(3), 12–25. https://doi.org/10.30880/jtet.2022.14.03.002
Ismail, N., Abdullah, S., & Othman, N. (2022). Assessment literacy among technical and vocational education and training (TVET) lecturers in Malaysia: Challenges and prospects. Journal of Technical Education and Training, 14(3), 12–25. https://doi.org/10.30880/jtet.2022.14.03.002
Istiara, F., & Hastomo, T. (2023). Exploring lecturers and administrative staffs’ strategies to hone EFL students’ digital literacy. JOALL (Journal of Applied Linguistics and Literature), 8(1), 151-172.
Fauzan, F., Arifin, F., Lubis, M. A., & Firdaus, F. M. (2022). Lecturer's Digital Literacy Ability in the Pandemic. Cypriot Journal of Educational Sciences, 17(4), 1130-1142.
Jaya, I., & Nasir, M. (2025). Holistic Assessment in TVET: Enhancing Writing Evaluation Through a Pilot Study. Journal of Language Testing and Studies, 1(1), 55-68.
Jean de Dieu, H., & Zhou, K. (2021). Physical literacy assessment tools: a systematic literature review for why, what, who, and how. International Journal of Environmental Research and Public Health, 18(15), 7954.
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. https://doi.org/10.1016/j.edurev.2007.05.002
Juanjuan, G., & Mohd Yusoff, N. (2022). A review of teachers’ assessment literacy proficiency measures. International Journal of Evaluation and Research in Education (IJERE), 11(3), 1143. DOI: 10.11591/ijere.v11i3.22300
Kalajahi, S. a. R., & Abdullah, A. N. (2016). Assessing Assessment Literacy and Practices among Lecturers. Pedagogika, 124(4), 232–248. https://doi.org/10.15823/p.2016.65
Kao, Y. T. (2023). Empowering preservice English teachers with language assessment literacy concepts and practices: Application of Vygotskian concept‐based language instruction. The Modern Language Journal, 107(S1), 68-94.
Khalid, N. H. M., Latif, A. A., & Yusof, I. J. (2021). Research Focusing on Assessment Literacy: A Systematic Literature review. Studies of Applied Economics, 39(10). https://doi.org/10.25115/eea.v39i10.5816
Khalid, N. H. M., Yusof, I. J., Latif, A. A., & Jani, M. D. M. (2023). Psychometric properties of teacher classroom assessment literacy instrument using Rasch model analysis. Int J Eval & Res Educ ISSN, 2252(8822), 8822.
Khalid, N. H. M., Yusof, I. J., Latif, A. A., & Jani, M. D. M. (2023). Psychometric properties of teacher classroom assessment literacy instrument using Rasch model analysis. International Journal of Evaluation and Research in Education (IJERE), 12(2), 638. https://doi.org/10.11591/ijere.v12i2.23607
Khine, M. S. (2020). Rasch measurement. In Springer eBooks. https://doi.org/10.1007/978-981-15-1800-3
Linacre, J. M. (1994). Sample size and item calibration (or person measure) stability. Rasch Measurement Transactions, 7(4), 328.
Linacre, J. M. (2002). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions, 16(2), 878.
Linacre, J. M. (2003). Winsteps computer program version 3.48. Chicago. http://www.winsteps.com/a/ winsteps-tutorial
Linacre, J. M. (2005). Winstep Rasch measurement computer program. MESA Press.
Linacre, J. M. (2007). A user’s guide to WINSTEPS Rasch-model computer program. MESA Press.
Linacre, J. M. (2010). When to stop removing items and persons in Rasch misfit analysis? Rasch Measurement Transactions, 23(4), 1241.
Linacre, J. M. (2012). A user’s guide to WINSTEPS: Rasch model computer programs. MESA Press.
Linacre, J. M. (2017). Winsteps Rasch measurement computer program user’s guide. Winsteps.com.
Linacre, J.M., 2011. Winsteps Rasch Measurement Computer Program Users Guide. Beaverton Publisher, Beaverton, Oregon,.
Lutovac, S., & Flores, M. A. (2022). Conceptions of assessment in pre-service teachers’ narratives of students’ failure. Cambridge Journal of Education, 52(1), 55-71.
Mardiana, H. (2024). Perceived impact of lecturers’ digital literacy skills in higher education institutions. Sage Open, 14(3), 21582440241256937.
Modarresi, G., Jalilzadeh, K., Coombe, C., & Nooshab, A. (2021). Validating a test to measure translation teachers' assessment literacy. Journal of Asia TEFL, 18(4), 1503.
Mokshein, S. E., Ahmad, H., Lebar, O., Dollah, M. U., Yunus, J., Rahmat, A., & Ahmed, H. H. (2022). Validation of the Malaysian-Based Assessment Practice Inventory for Teacher Educators (MAPITE) using Rasch Model. Journal of Engineering and Applied Sciences, 17(9), 2783–2798. doi:10.36478/jeasci.2019.2783.2798
Mudau, P. K. (2022). Lecturers’ views on the functionality of e-portfolio as alternative assessment in an open distance e-learning. International Journal of Educational Methodology, 8(1), 81-90.
Muhammad Hanafi, N. B., Mat Ali, N. B., Zamani, S. B., Yamin, N. A. B., & Ismail, N. B. (2020). Examining assessment literacy: A study of technical teachers. European Journal of Molecular & Clinical Medicine, 7(8), 705–715.
Mutinda, F. M. (2023). Classroom Assessment Practices by Mathematics Teachers in Technical Vocational Education Training (TVET) Institutions in Kenya (Doctoral dissertation, University of Nairobi).
Njuki, J. W., & Mukundi, R. M. (2023). Enhancing integration of Information Literacy in TVET curriculum in response to COVID-19 pandemic effects. American Journal of Education and Practice,[Lewes, DE], 7(4), 42-53.
Norhasim, N., & Mohamad, M. (2020). Teachers’ challenges on the implementation of classroom-based assessment for oral proficiency: A literature review. Creative Education, 11(11), 1922–1929. https://doi.org/10.4236/ce.2020.1110140
Nur, L., Nurhasan, N., Al Ardha, M. A., Chaeroni, A., Friskawati, G. F., Muslihin, H. Y., ... & Malik, A. A. (2025). Measurement of physical literacy of elementary school students in Indonesia: an analysis using the Rasch model. Retos: nuevas tendencias en educación física, deporte y recreación, (65), 909-920.
Oo, C. Z., Alonzo, D., & Asih, R. (2022). Acquisition of teacher assessment literacy by pre-service teachers: A review of practices and program designs. Issues in Educational Research, 32(1), 352-373.
Osman, K., Sovey, S., & Matore, M. E. E. M. (2022). RasCh Analysis for Disposition Levels of Computational Thinking Instrument among Secondary school students. Eurasia Journal of Mathematics Science and Technology Education, 18(3), em2088. https://doi.org/10.29333/ejmste/11794
Padmadewi, N. N., Artini, L. P., Ratminingsih, N. M., & Marsakawati, N. P. E. (2022). Needs Analysis of Literacy Assessment Using Blended Learning for Beginner EFL Learners. Journal of Language Teaching & Research, 13(2).
Pastore, S. (2023). Teacher assessment literacy: a systematic review. Frontiers in Education, 8. https://doi.org/10.3389/feduc.2023.1217167
Purnami, D. E., Hartinah, S., & Susongko, P. (2024). Model Assesment Literasi Membaca pada Siswa Sekolah Dasar Berbasis Pemodelan Rasch. Journal of Education Research, 5(3), 3217-3223.
Ramaligela, S. M. (2022). Student's Perceptions of assessment legitimacy in TVET Sector in South Africa. In Research Anthology on Vocational Education and Preparing Future Workers (pp. 452-467). IGI Global Scientific Publishing.
Rasch, G. (1980). Probabilistic Models for Some Intelligence and Attainment Tests. Chicago, IL: University of Chicago Press.
Rasul, M. S., Ismail, N., Rajuddin, M. R., & Rauf, R. A. A. (2009). Aspects of employability skills in technical and vocational education and training (TVET) curriculum. Journal of Technical Education and Training, 1(1), 1–8. https://publisher.uthm.edu.my/ojs/index.php/jtet/article/view/529
Rohaya Talib. (2009). Pembinaan dan Pengesahan Instrumen Ujian Literasi Pentaksiran. Tesis PhD, Universiti Teknologi Malaysia.
Rozali, M. Z., Puteh, S., Yunus, F. A. N., Hamdan, N. H., & Latif, H. F. M. (2022). Reliability and validity of instrument on Academic Enhancement Support for Malaysian student-athletes using Rasch Measurement Model. Asian Journal of University Education, 18(1), 1–15. https://doi.org/10.24191/ajue.v18i1.17199
Singh, H. K. J., Singh, C. K. S., Ong, E. T., Wong, W. L., & Gopal, R. et al. (2023). Developing items to measure the assessment literacy of ESL teachers. Journal of Higher Education Theory and Practice, 23(16), 225–245.
Sun, H., & Zhang, J. (2022). Assessment literacy of college EFL teachers in China: Status quo and mediating factors. Studies in Educational Evaluation, 74, 101157.
Tajeddin, Z., Khatib, M., & Mahdavi, M. (2022). Critical language assessment literacy of EFL teachers: Scale construction and validation. Language Testing, 39(4), 649-678.
Tsagari, D. (2021). Gauging The Assessment Literacy Levels Of English Language Teachers In Norway. The European Journal of Applied Linguistics and TEFL, 10(1), 161-192.
Vinodhen, V. (2025). Impact of online resources on assessment literacy of polytechnic educators. Forum for Education Studies., 3(3), 2678. https://doi.org/10.59400/fes2678
Wright, B. D., & Master, G. N. (1982). Rating scale analysis. Cesa Press.
Wright, B. D., & Mok, M. M. (2004). An overview of the family of Rasch measurement models. In E. V. Smith, & R. M. Smith (Eds.), Introduction to Rasch measurement: Theory, models, and applications. JAM Press.
Wright, B. D., & Mok, M. M. (2004). An overview of the family of Rasch measurement models. In E. V. Smith, & R. M. Smith (Eds.), Introduction to Rasch measurement: Theory, models, and applications. JAM Press.
Yan, Z., & Pastore, S. (2022). Are teachers literate in formative assessment? The development and validation of the Teacher Formative Assessment Literacy Scale. Studies in Educational Evaluation, 74, 101183.
Yusop, S. R. M., Rasul, M. S., Mohamad Yasin, R., Hashim, H. U., & Jalaludin, N. A. (2022). An assessment approaches and learning outcomes in technical and vocational education: A systematic review using PRISMA. Sustainability, 14(9), 5225. https://doi.org/10.3390/su14095225
Yusop, S. R. M., Rasul, M. S., Yasin, R. M., & Hashim, H. U. (2023). Identifying and validating vocational skills domains and indicators in classroom assessment practices in TVET. Sustainability, 15(6), 5195. https://doi.org/10.3390/su15065195
