Multimodal Emotion Detection for Education and Work Environment by using Improved Artificial Intelligence Machine Vision System

Abstract

The use of Artificial Intelligence (AI) has significantly advanced emotion recognition within Human-Computer Interaction (HCI). This paper aims to develop a multimodal emotion detection system for educational and work environments using an enhanced AI machine vision system. The primary focus is on training and testing a multimodal AI model in Python, utilizing Convolutional Neural Networks (CNN). The results from the trained facial emotion AI model demonstrated substantial improvements. Training accuracy increased from 30.49% to 72.21%, while validation accuracy improved from 37.6% to 60.58%. Simultaneously, training loss decreased from 180.69% to 73.65%, and validation loss reduced from 172.97% to 107.53%. This CNN-based model can detect seven emotions: happy, sad, neutral, angry, fear, disgust, and surprise, using OpenCV. Furthermore, the ECG emotion AI model, also trained with CNN, successfully recognized patterns for the same seven emotions. When these two models are combined into a multimodal AI system, they can detect facial and ECG-based emotions simultaneously. This comprehensive approach allows for the detection of both visible and hidden emotions, such as stress or anxiety, which may not be easily discernible through facial expressions alone. The integration of these models into a multimodal AI system provides a more accurate and holistic understanding of human emotions, enhancing applications in educational and work settings. The improved detection capabilities can lead to better user experiences and more effective responses to emotional states, ultimately contributing to advancements in HCI.