The Research And Application Of Mouse For The Disabled Based On WebRTC

Vision-based gesture recognition is a key and hot point in the research of human-computer interaction. This paper summarizes.the typical vision-based gesture recognition workflow, including image acquisition, image preprocessing, gesture segmentation, gesture modeling, feature extraction and classification. The similarities and differences of gesture recognition algorithm based on static hand-shaped structure and based on the trajectory of dynamic gesture motion have been compared carefully. This paper has researched real-time gesture recognition of persons with disabilities under the monocular vision. The Monocular Vision means using a single computer camera to get gesture information which is to be processed and studied subsequently.The traditional visual gesture recognition has been applied to human-computer interaction of persons with disabilities. Mouse for the disabled has been proposed base on gesture recognition and WebRTC framework, which uses B/S structure to separate image acquisition, command execution and image match. The browser is just responsible for the image acquisition and the simulation of mouse operation, and the server-side completes command feedback and image match which is transparent to the client browser. New features in HTML5such as WebRTC and WebSocket have been introduced to make browsers capture the image of the disabled controllable object and submitted to the server-side. B/S architecture allows the system to be built on the cloud environment and have good scalability, flexibility and customization.For the implementation of server-side gesture recognition module, this paper have proposed two image match algorithms which are based on the static gesture recognition and the other is based on dynamic gesture tracking respectively:The static gesture recognition algorithm use proportion of shape and rotation function for feature extraction and background difference and Euclidean distance for image matching to ensure rapid response. Experiments show this algorithm has achieved initial design purpose and has acceptable delay, but there is a problem that the morphological structure discrimination of the disabled controllable object itself not enough.The dynamic gesture tracking algorithm use CamShift to track Controllable object and record its trajectories. It combines with the controllable morphological characteristics of the object to determine whether record its trajectory information to graph the gesture, and converts the gesture graphics conversion to8-direction vector for feature extraction using Levenshtein distance. This algorithm has solved the problem which may arise in the static gesture recognition algorithm.