Image-based gesture recognition with support vector machines

Recent advances in various display and virtual technologies, coupled with an explosion in available computing power, have given rise to a number of novel human-computer interaction (HCI) modalities, among which gesture recognition is undoubtedly the most grammatically structured and complex. However, despite the abundance of novel interaction devices, the naturalness and efficiency of HCI has remained low. This is due in particular to the lack of robust sensory data interpretation techniques. To address the task of gesture recognition, this dissertation establishes novel probabilistic approaches based on support vector machines (SVM). Of special concern in this dissertation are the shapes of contact images on a multi-touch input device for both 2D and 3D. Five main topics are covered in this work.;The first topic deals with the hand pose recognition problem. To perform classification of different gestures, a recognition system must attempt to leverage between class variations (semantically varying gestures), while accommodating potentially large within-class variations (different hand poses to perform certain gestures). For recognition of gestures, a sequence of hand shapes should be recognized. We present a novel shape recognition approach using Active Shape Model (ASM) based matching and SVM based classification. Firstly, a set of correspondences between the reference shape and query image are identified through ASM. Next, a dissimilarity measure is created to measure how well any correspondence in the set aligns the reference shape and candidate shape in the query image. Finally, SVM classification is employed to search through the set to find the best match from the kernel defined by the dissimilarity measure above. Results presented show better recognition results than conventional segmentation and template matching methods.;In the second topic, dynamic time alignment (DTA) based SVM gesture recognition is addressed. In particular, the proposed method combines DTA and SVM by establishing a new kernel. The gesture data is first projected into a common eigenspace formed by principal component analysis (PCA) and a distance measure is derived from the DTA. By incorporating DTA in the kernel function, general classification problems with variable-sized sequential data can be handled.;In the third topic, a C++ based gesture recognition application for the multi-touchpad is implemented. It uses the proposed gesture classification method along with a recursive neural networks approach to recognize definable gestures in real time, then runs an associated command. This application can further enable users with different disabilities or preferences to custom define gestures and enhance the functionality of the multi-touchpad.;Fourthly, an SVM-based classification method that uses the DTW to measure the similarity score is presented. The key contribution of this approach is the extension of trajectory based approaches to handle shape information, thereby enabling the expansion of the system's gesture vocabulary. It consists of two steps: converting a given set of frames into fixed-length vectors and training an SVM from the vectorized manifolds. Using shape information not only yields discrimination among various gestures, but also enables gestures that cannot be characterized solely based on their motion information to be classified, thus boosting overall recognition scores.;Finally, a computer vision based gesture command and communication system is developed. This system performs two major tasks: the first is to utilize the 3D traces of laser pointing devices as input to perform common keyboard and mouse control; the second is supplement free continuous gesture recognition, i.e., data gloves or other assistive devices are not necessary for 3D gestures recognition. As a result, the gesture can be used as a text entry system in wearable computers or mobile communication devices, though the recognition rate is lower than the approaches with the assistive tools. The purpose of this system is to develop new perceptual interfaces for human computer interaction based on visual input captured by computer vision systems, and to investigate how such interfaces can complement or replace traditional interfaces.;Original contributions of this work span the areas of SVMs and interpretation of computer sensory inputs, such as gestures for advanced HCI. In particular, we have addressed the following important issues: (1) ASM base kernels for shape recognition. (2) DTA based sequence kernels for gesture classification. (3) Recurrent neural networks (RNN). (4) Exploration of a customizable HCI. (5) Computer vision based 3D gesture recognition algorithms and system.