Motion Matching, Choreographing And Recognition Driven By Live Performance

Perceptual user interface takes advantage of human's perceptual capabilities in or-der to present semantic information in intuitive and natural manners. Performance based human computer interaction technique is one important part of perceptual user interface, which allows users to directly control applications with their natural body movements. Be-cause of the unprecedented and revolutionary way of manipulation, it attracts great research interests of a lot of researchers. Recent years, it becomes a research focus in the fields of computer animation, virtual reality and video games. The main purpose of this thesis is to study and explore the key issues of this technology, including motion index construction, reactive motion generation, motion adaptation, automatic generation of training samples, automatic feature extraction, machine learning, etc. Based on these algorithms, a series of demo applications are developed on both PC and mobile platforms using Xsens motion sensors, Wii Remote,5DT data gloves and Nokia N95 respectively. Furthermore, a series of subjective and objective experiments were done to validate the algorithms.Specifically, the main research topics of this thesis are as follows:1. Intuitive motion retrieval with motion sensors. In the process of 3D animation pro-duction, how to intuitively present the query example is a key issue of content-based motion retrieval. In this thesis, an index for a large motion database is created based on geometric features. During motion retrieval, a user performs the motion in his mind with on-body sensors, the system will then search into the large database for the matched motion clips with a coarse-to-fine strategy. The system not only provides an intuitive and visual way of communication for animators, but improves greatly the motion retrieval efficiency during motion reuse.2. Responsive action generation in accordance with physical constraints. Responsive motion generation of avatars is a key issue in 3D graphics applications, such as vir- tual reality and video games. In this thesis, a performance-driven avatar control in-terface with physical constraints is presented. When the user performs an action, the system will first select a series of candidate motions according to the user's perfor-mance. It will then sort the candidates according to the defined physical constraints. The best motion is finally selected to drive the avatar. In order to make the retrieved motion to meet the kinematics and kinetics constraints perfectly, the retrieved motion will be further adapted in real time.3.3D intuitive gesture interaction via motion sensing. Motion sensing techniques are less limited by space and lighting from the point of view of human computer in-teraction. In this thesis, accelerometers are employed to explore how to effectively build gesture-based user interfaces. For each gesture, users need only to perform it once. Training samples are generated semi-automatically by noise-adding tech-niques, which will be used to train the machine learning model after being prepro-cessed and feature-extracted. A series of experiments were done on a data set of 70 gestures. Results show that it can greatly improve the end user experience in human computer interaction.4. Performance-driven motion choreographing. Live performance is an intuitive way to draft the desired motion in the choreographer's mind. In this thesis, a novel ap-proach is presented to choreograph motions by live performance with accelerometer-s. The process begins by asking the user to place the accelerometers on the his limbs and perform some actions. The computer then recognizes the actions with Hidden Markov Model. At last, the captured actions are further synthesized with motion re-timing and exaggeration based on the acceleration signals from the accelerometers. Experimental results show that it can effectively recognize actions with great spatial-time variance, and is easy-to-use especially for a novice with little experience.5. Sign language recognition based on a hierarchical machine learning model. Signs are essentially time series performed by signers according to some specified rules. In order to aid the communication between deaf and common people, a sign language recognition system is developed in this thesis. A series of training samples with great variance are captured using a motion capture system and data gloves. A hierarchical machine learning model is developed, which can exclude lots of impossible signs in the foregoing layers with very little computation cost and make the strong Hid-den Markov Model focus on the difficult ones. Experiments on a database with a dictionary of 68 signs and 1224 motion clips show the efficiency of the algorithm.