The Research Of Chinese Sign Language Video Synthesis Aided By3D Information

There are a large number of hearing-impaired people in China, and Chinese SignLanguage (CSL) is their natural language to communicate with outside world. Signlanguage expresses semantics using the motion of hands, face, head and body, whichis a kind of visual-spatial language. Comparing with CSL animation based on virtualhuman, the CSL video is more realistic acceptable. It can provide a more vivid visuallanguage expression interface for hearing-impaired people and meet their rights toenjoy the modern civilization as well as other people. What s more, it can improvequality-of-life for the hearing-impaired people. So research on CSL synthesis hasbroad social significance. This thesis synthesized CSL video with computers throughcombining a series of sign language word video clips according to the syntax rules.Since adjacent sign language word video clips have visual discrepancies in motiontrails, gestures and other aspects, transition video frames need to be inserted toenhance the sense of reality. There are two methods to get transition video frames:retrieving predefined video frames database or compositing them through stitchingimage blocks. The former needs to gather all possible transition for any two signlanguage words, which requires huge storage and human labor. Therefore compositionmethod is more available and the synthesis of transition video frames becomesimportant for achieving the synthesized video-based sign language.With the object of smoothing the cascaded CSL video clips, this thesis researchesmethod for synthesizing the transition frames and proposed a kind of synthesismethod assisted by3D model. With help of existing3D CSL motion data, the3Dhand model is driven to generate gestures of transition video frames, and the projectedgesture image is stitched with other image blocks together to generate transition videoframes. Main works in this thesis are as follows:1. Construction of3D hand model which is consistent with CSL video clipswaited for connection: Since the virtual hand model which is compatible with3DCSL motion data has different ratio from hands in the CSL video clips waited to becascade, the existed virtual hand model need to be adjusted to acquire higher visualeffect. This article designs feature selection scheme according to existing3D visualhand model, and calibrates the defined feature points with their two-dimensional, then calculates three-dimensional information of the feature points with help ofreconstruction method of non-rigid objects based on Gaussian distribution. Thecomputed3D information is used for adjusting the structural proportion of3D handmodel to make projected images are proportional to hand images which are institching sign language video.2. Sign language video transition frame synthesis: when the above-mentionedhand model are driven by data to generate gesture projection image for transitionvideo frame, the hand images texture and its fusion with other image blocks shouldbe considered to get a better visual effect for the composited sign language videoframe. This thesis proposes a hand texture synthesis method based texture templateimage which are predefined. For the fusion of various image blocks, blocks andgradient based method is adopted to blend the gesture images, arm images and torsoimages seamlessly which make the transition video frames having higher visualreality.3. Realization of Chinese sign language video synthesis prototype system:Process the original captured sign language video according to data requirement ofhand language video synthesis system, including splitting the videos into signlanguage word video clips according to sign language dictionary, marking videosemantic information for frames, building arm pattern library and so on. Accordingwhat has been discussed above, the system includes several modules:3d hand modeloptimization module, transition frames generation module and texture optimizationmodule, CSL text segmentation module, movement path estimation module,3Danalyzing and hand model rendering module, the previous three modules are designand realized in this thesis.