Study On Key Technologies Of The Chinese Sign Language Synthesis Based On The Video Stitching

Sign language,expressing information through the motion of hands,is the main communication tool for the hearing disabled people.Sign language synthesis,translating the natural language into sign language,can bridge the communicating gap between the deaf and normal person.Video-based sign language synthesis produces sign language video by stitching the sign language word video clips from the database.However,there may be obvious difference in the adjacent sing language videos.For accomplishing the more realistic and smooth sign language synthesis,the transition frames describing the motion is introduced.The existing methods about generating transition frames are based on the transition model database,consuming much manpower.Therefore,the study to generate the transition frames is of great importance for research and application.Analyzing the feature of sign language,we design a new sign language synthesis scheme with the color,depth and skeleton information captured by Kinect,and focus on the algorithm of generating transition frames.First,the sign language video database is constructed based on Kinect.Then,the frame number and skeletons of the transition frames are obtained through preprocess.Finally,the sign language video is synthesized through combining the sign language word video with the generated transition frames.The transition frames are created by mesh deformation and image morphing.The main research contents of this dissertation are listed as follows.1.Research the sign language video preprocessing.As the preparations for generating transition frames,preprocessing includes word segmentation,the best joining points judgment and the transition path computation.Then,deliver the results of preprocessing into the transition frames generation module.2.Research the mesh deformation algorithm.The transition frames should comply with the biomechanical principles of human movements.To make the deformation algorithm simulates the sign language movements better,this dissertation provides a local-controlled as-rigid-as-possible deformation algorithm,which is composed of as-rigid-as-possible energy.group lasso penalty,and smoothness penalty.The proposed deformation method satisfies the properties of rigidity,locality,smoothness and shape-awareness simultaneously.The deformation model can be solved with an alternating least squares algorithm,called "local-global" algorithm.We use the proposed deformation algorithm to deform the sign language video frame.Then,the generated forward and backward deformation sequences will be the inputs for the following morphing process.3.Research the image morphing algorithm.For generating the transition frames which have to express the motion variation and information transmission,this dissertation proposes a deformation-assisted image morphing algorithm.Firstly,a series of deformed images is obtained through the skeleton control.Then,the deformed sequences are aligned to obtain the final transition frames by utilizing the patch-based image synthesis,conforming to source similarity and temporal coherence.The similarity distance considering the color and gradient distance can preserve the image details better.With smoothness and realness,the experimental results on the sign language video demonstrate the performance of the proposed sign language synthesis method.