Viewpoint Free Sign Language Recognition Based On The Constraint Of Fictitious Stereo Vision

Sign language recognition aims to translate sign language into text or speechby computer, so as to facilitate the communication between the deaf and the hearingpeople and help the deaf or hard-of-hearing better integrate into the society.According to data collection of sign language, sign language recognition are gen-erally divided into two major categories: dataglove-based sign language recognitionand vision-based sign language recognition. Since the vision-based method is moreconvenient to the end-users than dataglove-based method,it attracts more attention ofthe researchers. However, most of the current methods require a specific view of thesigners, generally the frontal view. This constraint means that the signers can onlyperform with specific location and orientation, and limits the freedom of the signer.The thesis aims to achieve viewpoint free sign language recognition within acertain scope with only one camera, so as to remove the restriction of a specified viewand provide convenience for the user. The scope now covers a range from 0? to 45?from the point of view of the angle between the optical axis of the camera and theorientation of the signer. The size of sign language vocabulary is preliminarily set tomedium.The thesis focuses on the following two aspects:(A) Viewpoint free sign language recognition based on viewpoint dependent fea-tures. Based on the FSV (Fictitious Stereo Vision) constraint that all correspondingframe pairs between two sequences of the same sign but from different viewpointscan be explained as captured synchronously in some fictitious stereo vision system,the thesis proposes a novel FSV recognition framework for viewpoint free sign lan-guage recognition. The proposed FSV recognition framework converts the recogni-tion task to a verification task in the framework of epipolar geometry and achievesrecognition by verifying whether two sign sequences can be explained as capturedsynchronously in some fictitious stereo vision system. Because the FSV recognitionframework employs the image coordinate of feature points, it is a way to viewpointfree sign language recognition from viewpoint dependent features. Based on the pro-posed FSV recognition framework, the thesis proposes three methods for viewpoint free sign language recognition, including the method based on the uniqueness of fun-damental matrices, the method of employing evidence theory to verify the uniquenessof fundamental matrices and the method based on the correspondence between thecompounded frame pair. Experiments show the efficiency of the proposed three meth-ods.(B) Viewpoint free sign language recognition under short-duration data defi-ciency. Data deficiency refers to the case that some frames in a sign language sequencecan provide only small number of efficient features or no efficient feature for match-ing. Data deficiency may be caused by many factors such as self-occlusion, the imageblur and the imperfection of the feature extraction algorithm. Besides, in the appli-cation of viewpoint free sign language recognition, the observable feature set of anobservation sample may be different from that of its matched template sample, whichmay also cause data deficiency. Data deficiency may affect the efficiency and feasi-bility of recognition algorithm. The thesis emphasizes the case of short-duration datadeficiency and aims to achieve robust and efficient methods for viewpoint free signlanguage recognition under such cases. In the light of more frames providing morefeatures, the thesis proposes two novel methods for short-duration data deficiency,including the method based on the homography of tiny motions and the fundamental-matrix-centered Sample-Consensus method. Experiments show the efficiency of theproposed two methods.It is worth noting that the proposed FSV recognition framework and all the pro-posed methods for viewpoint free sign language recognition not only fit for sign lan-guage recognition, but also can be applied to more broad fields such as viewpoint freemotion recognition and rigid-motion analysis.