An Augmented Reality System For Education Based On Gesture Recognition Using Kinect

Computer Vision (CV) is a research field which deals with how to use computers to solve a series of vision related problems. Gesture recognition is one of the most significant subjects in CV. The purpose of it is to use computers to understand human gestures and then create experiences of natural human computer interaction (HCI). The"Standard Natural Classroom"(SNC) of E-learning lab, Shanghai Jiao Tong University uses a lot of technologies including Artificial Intelligence, Computer Vision, multimedia etc. to provide teacher and students more interactive learning environment on the basis of traditional classroom. This article proposes and implementes a prototype of education aid system which combines gesture recognition and augmented reality based on the exploration of applications using latest technology product Kinect under the framework of the innovated intelligent classroom SNC.The major work in this article includes several parts. First one is the research of using the latest device Kinect in SNC. We propose that Kinect can be used to provide more natural way of human computer interactive in SNC. Second one is that we pick DTW algorithm to dynamically recognize gestures using the human body model of Kinect after review of gesture recognition methods. At the same time, we propose a method that combines dynamic and static gesture recognition. On the basis of the recognition result of movements of arms, we apply skin model to detect and extract the contour of hands. We choose Hu moments to extract features of hand gestures for recognition in our condition. Through adding the static recognition of hands to the dynamic recognition process, the false accept possibility can be decreased, and the recognizable gestures can be increased. Finally after the application review on augmented reality, we use 3D modeling technology to construct an augmented reality system for education which appears to be a virtual globe and is based on the human body detection and tracking of Kinect with gesture recognition.It is the first time that such work being proposed for researches in SNC; it has significant application innovation value. The research on related algorithms has academic value, too. The work in this article is also a good base of research and development works afterwards.