Real-Time Emotion Recognition From Eye-Tracking Videos

Smart glasses such as Google Glass,HoloLens,and some head-mounted smart devices such as enhanced implementation devices(AR),virtual reality devices(VR),are changing our lives.VR devices are giving users an immersive experience in a non-physical world,while smart glasses and AR devices allow users to interact with fictional objects in the real world.We envision that this type of smart device will become the central hub of personal devices and will be used widely by users in the future.On these devices,it is critical to provide intelligent interaction and improve the user's intelligent interactive experience.A key factor affecting the intelligence level of these wearable devices is the ability to recognize emotions.The smart devices can sense the emotional changes of the owner and perform appropriate operations at appropriate time.For example,for smart glasses,AR,and the like which can interact with the real world,by perceiving the user s emotions and the scenes that the user sees at this time,the advertisement design and delivery can be guided.For VR and other immersive devices,user's emotions can also guide the design of game scenes,game plots,and so on.However,on these smart devices,currently there are no effective emotion recog-nition methods.The traditional recognition technology relies on the expression of the entire face.However,after wearing such a device,it is difficult to capture a complete facial expression due to the occlusion of the device itself.Some methods try to in-troduce special hardware to sense the user's emotions,however these methods require additional hardware,also add extra costs,and even such devices cause inconvenience to the user.Therefore,without affecting the user's experience,how to quickly identify the user's emotions and ensure a good accuracy on the existing equipment becomes a problem.In order to solve this problem,this paper proposes a novel recognition algorithm and builds a corresponding prototype system.The work of this paper mainly includes the following aspects:1.A system named EMO is proposed and designed to recognize seven basic emotions in real time on smart devices,the recognition accuracy reached 72.2%.To achieve this goal,EMO uses the eye tracking camera on personal smart devices to capture the user's eye tracking videos,through these videos,EMO can identify the user's expressions.2.Based on deep learning,a feature extraction is designed to extract features from these videos;and also a personalized classifier is designed to adapt to the personal-ized expressions of different users on different kinds of expressions.3.A fast forwarder and a frame sampler are designed to evaluate and avoid unnec-essary calculations with little loss of accuracy,thus significantly reduce the time cost.On the Open-Q820 development board,equipped with our fast forwarder,the recognition speed reached 12.8fps;when further equipped with our frame sampler,the recognition speed can reach 97fps.4.A prototype platform is built,respectively equipped with Open-Q820 and Hikey development board.Based on this,the whole EMO system is implemented and a comprehensive experimental evaluation is conducted.The experimental results show that EMO can identify user's expressions in real time,and is significantly better than the most advanced methods in terms of recognition accuracy,recognition speed and resource consumption.