MIXED REALITY (MR) MODEL BASED ON VISUAL INTERACTION (VI) DESIGN PRINCIPLES FOR DEAF LEARNERS: A SYSTEMATIC LITERATURE REVIEW (SLR)
DOI:
https://doi.org/10.35631/JISTM.1144017Keywords:
Human-Computer Interaction, Assistive Technology, Immersive Technology, Mixed Reality, Visual Interaction, Deaf LearnersAbstract
Although Mixed Reality (MR) technologies have increasingly been explored to support Deaf education, existing studies primarily emphasize technological capabilities rather than the visual interaction requirements and cognitive needs of Deaf learners. This study presents a systematic literature review (SLR), following the PRISMA guidelines, to examine MR applications developed for Deaf individuals between 2015 and 2025. Five academic databases were searched using predefined inclusion and exclusion criteria. Eleven MR applications for Deaf individuals were identified, of which six focused specifically on educational settings and were further analysed to examine their features, interaction approaches, and design limitations. The findings indicate that while existing applications incorporate advanced features such as 3D avatars, gesture recognition, and gamified interaction, none systematically integrates Visual Interaction (VI) design principles to support clarity, consistency, accessibility, and cognitive usability. Furthermore, no MR application was specifically developed for Malaysian Deaf learners. Based on these findings, this paper proposes the Mixed Reality–Visual Interaction Design for Deaf Learners (MRVI-DL), a design framework that integrates MR capabilities with VI principles to support the development of accessible and cognitively appropriate learning environments for Deaf learners.
Downloads
References
Adamo-Villani, N., & Anasingaraju, S. (2017). Holographic signing avatars for deaf education. In E-learning, e-education, and online training (pp. 54–61). Springer. https://doi.org/10.1007/978-3-319-52836-6_8
Alan Cooper, A., Reimann, R., & Cronin, D. (2014). About Face: The essentials of interaction design. Wiley.
Atanan, Y., Sombunsukho, S., & Boonlue, S. (2017). E-future classroom: A study mixed reality learning environment for deaf learners in Thailand. International Journal of Environmental and Science Education, 12(10), 2291–2315.
Bala, A. (2020). Sign language and its linguistic structure: A global perspective. Routledge.
Ben Shneiderman, B., et al. (2016). Designing the user interface. Pearson.
Booth, A. (2016). Searching for qualitative research for inclusion in systematic reviews: A structured methodological review. Systematic Reviews, 5(1), 74. https://doi.org/10.1186/s13643-016-0249-x
Chen, S., Cheng, H., Su, S., Patterson, S., Raja Kushalnagar, R., Huang, Y., & Wang, Q. (2025). Customizing generated signs and voices of AI avatars: Deaf-centric mixed-reality design for Deaf-hearing communication. Proceedings of the ACM on Human-Computer Interaction, 9(2), 1–31.
Chong, C. L. (2018). Sign language education in Malaysia: BIM and KTBM perspectives. Malaysian Journal of Communication, 34(2), 45–60.
Claes Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (pp. 1–10). https://doi.org/10.1145/2601248.2601268
Correa, D. P. A., Mejía, J. P., Lenis, A. M., Camargo, C. A., & Navarro-Newball, A. A. (2018). Building alternative methods for aiding language skills learning for the hearing impaired. In Colombian conference on computing (pp. 192–200). Springer.
Davis, J. (2022). PokerFace mask: Exploring augmenting masks with captions through an interactive, mixed-reality prototype (Master’s thesis). Dartmouth College.
Diane Brentari, D. (2010). Sign languages: A linguistic introduction. Cambridge University Press.
Don Norman, D. A. (2013). The design of everyday things (Rev. ed.). Basic Books.
Edmondson, S., & Howe, J. (2019). Exploring the social inclusion of deaf young people in mainstream schools using their lived experience. Educational Psychology in Practice, 35(2), 216–228.
Garrett, J. J. (2010). The elements of user experience. New Riders.
Glasser, A., Mason Riley, E., Weeks, K., & Raja Kushalnagar, R. (2019). Mixed reality speaker identification as an accessibility tool for Deaf and hard of hearing users. In Proceedings of the 25th ACM symposium on virtual reality software and technology (pp. 1–3). ACM.
Gonzalez, C. (1995). Visual design of interaction, dialog, or interface? ACM SIGCHI Bulletin, 27(1), 12–13.
Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta-analyses? Research Synthesis Methods, 11(2), 181–217. https://doi.org/10.1002/jrsm.1378
Helen Sharp, H., Rogers, Y., & Jennifer Preece, J. (2019). Interaction design: Beyond human–computer interaction (5th ed.). Wiley.
Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2019). Cochrane handbook for systematic reviews of interventions. Wiley.
Hirabayashi, N., Fujikawa, N., Yoshimura, R., & Fujisawa, Y. (2019). Development of learning support equipment for sign language and fingerspelling by mixed reality. In Proceedings of the 7th ACIS International Conference on Applied Computing and Information Technology (pp. 1–6).
Imashev, A., Oralbayeva, N., Kimmelman, V., & Sandygulova, A. (2022). A user-centered evaluation of the data-driven sign language avatar system: A pilot study. In Proceedings of the 10th International Conference on Human-Agent Interaction (pp. 194–202).
Johnson, J. (2014). Designing with the mind in mind. Morgan Kaufmann.
Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering.
Kydyrbekova, A., Kenzhekhan, A., Omirbayev, S., Oralbayeva, N., Imashev, A., & Sandygulova, A. (2023). Interaction design of the mixed reality application for Deaf children. In International Conference on Human-Computer Interaction.
Lidwell, W., Holden, K., & Butler, J. (2010). Universal principles of design. Rockport Publishers.
Mokhtar, E. S., Omar, A. C., & Aziz, N. (2022). Proposed design principles of Malay sign language mobile application for hearing-impaired alpha generation. In 2022 IEEE 12th International Symposium on Computer Applications and Industrial Electronics (ISCAIE) (pp. 233–240). IEEE.
Mullet, K., & Sano, D. (1996). Designing visual interfaces. ACM SIGCHI Bulletin, 28(2), 82–83.
Okoli, C., & Schabram, K. (2015). A guide to conducting a systematic literature review of information systems research.
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., & Moher, D. (2021). The PRISMA 2020 statement. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
Papadopoulos, T., Evangelidis, K., Kaskalis, T. H., Evangelidis, G., & Sylaiou, S. (2021). Interactions in augmented and mixed reality: An overview. Applied Sciences, 11(18), 8752.
Paul Milgram, P., & Fumio Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems, 77(12), 1321–1329.
Radianti, J., Majchrzak, T. A., Fromm, J., & Wohlgenannt, I. (2020). A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers & Education, 147, 103778.
Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: A key to evidence-based decisions. ACP Journal Club, 123(3), A12–A13.
Rokhsaritalemi, S., Sadeghi-Niaraki, A., & Choi, S. M. (2020). A review on mixed reality: Current trends, challenges and prospects. Applied Sciences, 10(2), 636.
Samala, A. D., Rawas, S., Criollo-C, S., Bojic, L., Prasetya, F., Ranuharja, F., & Marta, R. (2024). Emerging technologies for global education. SN Computer Science, 5(8), 1175.
Waldow, K., & Fuhrmann, A. (2020). Addressing deaf or hard-of-hearing people in avatar-based mixed reality collaboration systems. In IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (pp. 594–595). IEEE.
Wen, H., Xu, Y., Li, L., Ru, X., Wu, Z., Fu, Y., & Wang, X. (2024). Enhancing sign language teaching: A mixed reality approach. In 2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 740–745). IEEE.
World Health Organization. (2021). Deafness and hearing loss. WHO.
Yang, F. C., Mousas, C., & Adamo-Villani, N. (2022). Holographic sign language avatar interpreter. Computer Animation and Virtual Worlds, 33(3–4), e2082. https://doi.org/10.1002/cav.20
