Hybrid CNN-GRU Architecture for Efficient Emotion Recognition in Immersive Augmented Reality Systems
DOI:
https://doi.org/10.68337/cpsm.v1.i1.2026-015Keywords:
Emotion recognition, augmented reality, eye tracking, CNN, GRU, deep learning, affective computingAbstract
This paper presents a hybrid deep learning architecture that takes gaze data as input and recognizes emotions efficiently and in real time in resource-constrained augmented reality (AR) environments. Unlike conventional systems, which typically rely on facial expressions or speech signals, the proposed system uses eye tracking and gaze dynamics as the main indicators of emotional state, which offers better privacy, stability, and usability in practical AR applications. The model combines a convolutional neural network (CNN), which extracts spatial representations from eye-region images and gaze heatmaps, with a gated recurrent unit (GRU), which models time-varying gaze behavior. The model was evaluated on a self-curated dataset of 3,500 temporally sequenced samples annotated with six emotion classes. The proposed model achieved an accuracy of 91.2%, outperforming the baseline long short-term memory (LSTM) and standalone CNN models while remaining efficient. The GRU-based architecture was also more sensitive to temporal patterns and had a lower computational cost, with an average inference latency of 14.5 ms per sample, which meets real-time processing requirements.
References
[1] Philipp V. Rouast, Marc T. P. Adam, and Raymond Chiong, "Deep learning for human affect recognition: Insights and new developments," IEEE Trans. Affect. Comput., vol. 12, no. 2, pp. 524-543, 2021, doi: 10.1109/TAFFC.2018.2890471.
[2] Caitlin Mills, Julie Gregg, Robert Bixler, and Sidney K. D'Mello, "Eye-Mind reader: An intelligent reading interface that promotes long-term comprehension by detecting and responding to mind wandering," Hum.-Comput. Interact., vol. 36, no. 4, pp. 306-332, 2021, doi: 10.1080/07370024.2020.1716762.
[3] Premananda R., Sunil S. Harakannanavar, Sapna Kumari C., and Veena I. Puranikmath, "Development of real time emotion detection in faces using deep learning approach," Int. J. Intell. Syst. Appl. Eng., vol. 12, no. 4, pp. 2621-2633, 2024. [Online]. Available: https://www.ijisae.org/index.php/IJISAE/article/view/6689
[4] Byoung Chul Ko, "A brief review of facial emotion recognition based on visual information," Sensors, vol. 18, no. 2, Art. no. 401, 2018, doi: 10.3390/s18020401.
[5] Moataz El Ayadi, Mohamed S. Kamel, and Fakhri Karray, "Survey on speech emotion recognition: Features, classification schemes, and databases," Pattern Recognit., vol. 44, no. 3, pp. 572-587, 2011, doi: 10.1016/j.patcog.2010.09.020.
[6] Usman Alhaji Abdurrahman, Lirong Zheng, Abdulrauf Garba Sharifai, and Ishola Dada Muraina, "Heart rate and pupil dilation as reliable measures of neuro-cognitive load classification," in Proc. 2022 Int. Conf. Advancements in Smart, Secure and Intelligent Computing (ASSIC), 2022, pp. 1-7, doi: 10.1109/ASSIC55218.2022.10088296.
[7] Mohammad Soleymani, Jeroen Lichtenauer, Thierry Pun, and Maja Pantic, "A multimodal database for affect recognition and implicit tagging," IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 42-55, 2012, doi: 10.1109/T-AFFC.2011.25.
[8] Khafidurrohman Agustianto, Hendra Yufit Riskiawan, Dwi Putro Sarwo Setyohadi, I. Gede Wiryawan, Andi Besse Firdausiah Mansur, and Ahmad Hoirul Basori, "Eye gaze based model for anxiety detection of engineering students," in Emerging Technology Trends in Internet of Things and Computing, ser. Commun. Comput. Inf. Sci., vol. 1548, 2022, pp. 195-205, doi: 10.1007/978-3-030-97255-4_14.
[9] Pradeep Raj Krishnappa Babu and Uttama Lahiri, "Classification approach for understanding implications of emotions using eye-gaze," J. Ambient Intell. Humanized Comput., vol. 11, no. 7, pp. 2701-2713, 2020, doi: 10.1007/s12652-019-01329-8.
[10] Jia Zheng Lim, James Mountstephens, and Jason Teo, "Emotion recognition using eye-tracking: Taxonomy, review and current challenges," Sensors, vol. 20, no. 8, Art. no. 2384, 2020, doi: 10.3390/s20082384.
[11] Yihua Cheng, Haofei Wang, Yiwei Bao, and Feng Lu, "Appearance-based gaze estimation with deep learning: A review and benchmark," IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 12, pp. 7509-7528, 2024, doi: 10.1109/TPAMI.2024.3393571.
[12] Shreya Ghosh, Abhinav Dhall, Munawar Hayat, Jarrod Knibbe, and Qiang Ji, "Automatic gaze analysis: A survey of deep learning based approaches," IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 1, pp. 61-84, 2024, doi: 10.1109/TPAMI.2023.3321337.
[13] Jindi Bao, Xiaomei Tao, and Yinghui Zhou, "An emotion recognition method based on eye movement and audiovisual features in MOOC learning environment," IEEE Trans. Comput. Social Syst., vol. 11, no. 1, pp. 171-183, 2024, doi: 10.1109/TCSS.2022.3221128.
[14] Magdiel Jiménez-Guarneros and Gibran Fuentes-Pineda, "CFDA-CSF: A multi-modal domain adaptation method for cross-subject emotion recognition," IEEE Trans. Affect. Comput., vol. 15, no. 3, pp. 1502-1513, 2024, doi: 10.1109/TAFFC.2024.3357656.
[15] Yifei Lu, Wei-Long Zheng, Binbin Li, and Bao-Liang Lu, "Combining eye movements and EEG to enhance emotion recognition," in Proc. 24th Int. Joint Conf. Artif. Intell. (IJCAI), 2015, pp. 1170-1176. [Online]. Available: https://www.ijcai.org/Proceedings/15/Papers/169.pdf
[16] Jing Zhang, Sung Park, Ayoung Cho, and Mincheol Whang, "Recognition of emotion by brain connectivity and eye movement," Sensors, vol. 22, no. 18, Art. no. 6736, 2022, doi: 10.3390/s22186736.
[17] Luma Tabbaa, Ryan Searle, Saber Mirzaee Bafti, Md Moinul Hossain, Jittrapol Intarasirisawat, Maxine Glancy, and Chee Siang Ang, "VREED: Virtual reality emotion recognition dataset using eye tracking & physiological measures," Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., vol. 5, no. 4, pp. 1-20, 2021, doi: 10.1145/3495002.
[18] Jain Joseph, R. P. Aneesh, and Joseph Zacharias, "Deep learning based emotion recognition in human-robot interaction with multi-modal data," AIP Conf. Proc., vol. 3122, Art. no. 030007, 2024, doi: 10.1063/5.0216000.
[19] Vijya Tulsani, Prashant Sahatiya, Jignasha Parmar, and Sohil Parmar, "Real-time facial emotion recognition for AI-enhanced personalized learning," J. Inf. Syst. Eng. Manage., vol. 10, no. 40s, pp. 427-436, 2025, doi: 10.52783/jisem.v10i40s.7313.
Downloads
Published
Versions
- 2026-10-01 (2)
- 2026-09-30 (1)
Data Availability Statement
The data supporting the findings of this study are available from the corresponding author upon reasonable request.Conference Proceedings Volume
Section
Categories
License
Copyright (c) 2026 The Authors
This work is licensed under a Creative Commons Attribution 4.0 International License.
Articles published in Conference Proceedings in Science and Management are made available under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Authors retain copyright in their work and grant the journal first publication rights. Readers are free to share and adapt the published material for any purpose, including commercial use, provided appropriate credit is given to the author, a link to the license is provided, and any changes are indicated.
