Hand Detection And Tracking For Human-computer Interaction On Resource-limited Systems

Vision-based human-computer interaction (VbHCI) is a novel and promising technology which will enable more natural, intuitive and effective communication between human and computer. However, developing a VbHCI system is still a challenging task due to varying illumination, complex background, low-quality video data, and limited amount of processing power and storage capacity. In this dissertation, a real-time VbHCI system based on hand detection and tracking is designed and developed for resource-limited platform, and several fundamental computer vision techniques, such as background subtraction, skin color detection and object tracking, are studied to make them more suitable for VbHCI.Several important factors that impact the performance of VbHCI system are discussed. And then, a real-time VbHCI system which based on hand detection and tracking is designed and developed for resource-limited platform. Extensive experiments and two prototype applications have validated the VbHCI system.Conventional background subtraction methods are susceptible to quick illumination changes under HCI environment which often result in a drastic increase of false positives. A hierarchical background subtraction method is proposed. On the first level, scene background is modeled by local shape structure which is represented by shape context histograms. Contours of foreground objects are extracted by comparing the shape context histogram of edge points in the current frame against the shape-based background models. On the second level, pixels within the foreground object contours are modeled by their color values using an improved Gaussian mixture model. Experimental results demonstrate that the proposed method can robustly detect foreground objects under rapidly changing illumination conditions. Furthermore, the proposed method is capable of real-time processing on resource-limited platform.Rapidly changing illumination conditions under HCI application environment make skin color detection a challenging task, as skin colors in an image highly depend on the illumination under which the image was taken. This paper presents a method for skin color detection under rapidly changing illumination conditions. Skin colors are modeled under the Bayesian decision framework. Face detection is employed to online sample skin colors and a dynamic thresholding technique is used to update the skin color model. When there is no face detected, color correction strategy is employed to convert the colors of the current frame to those as they appear under the same illuminant of the last model updated frame. Skin color detection is then applied on the color corrected image. To improve efficiency, a novel method is proposed to detect illumination changes, and the skin color model is updated only if the illumination has changed.A novel tracking framework that explicitly decomposes the long-term tracking task into motion detection, object tracking, object detection and object validation is proposed and applied to face and hand tracking. Mean-shift tracking algorithm is adopted as the basic tracker, and multiple visual cues, including skin color and motion, are integrated for robust tracking. Off-line trained cascade AdaBoost object detector is employed to supports the tracker once it fails. Motion template is used to reduce scanning windows for object detection. Online trained validator decides if the object represents a specific instance selected for tracking.At last, we summarize the presented work. According to the imperfect aspects, we analyze and discuss the future work.