Research And Application Of Combining Multi-Modal Interactions Based On Scenarios

The current focus of human-computer interaction research is to provide a "human-centered"experience under the principle of natural and effective interactions.Multi-modal communications are used in our daily lives,for example,combinations of voice,gesture and eyesight.Multi-modal interactions are introduced to allow users to interact with computers via various perception channels,therefore multi-modal interactive systems are more efficient,potentially more natural and more human-centered than traditional user interfaces.Establishing a multi-modal interactive system is a challenge,because various interactive interfaces and complex contextual environments provide difficulties for interactive designers.Ambiguities tend to be introduced when combining inputs from multiple channels.The way to provide interactions and combine various interactive channels must be carefully considered and designed in order to enhance the robustness of the interaction.Meanwhile,different combinations strategies must be adopted in different contexts.In this dissertation,the problem of combining multi-modal interactions in different scenarios to support developers in the rapid implementation of multi-modal interactive systems is studied.Its main work is summarized as follows:(1)A method of generating interaction paths and texts describing its attributes based on state machines.In a state machine,an interaction path is defined to be a set of states traversed by state transitions from the start state to the end state.All the possible paths traversed from the start state to the end state can be all possible combinations of multi-modal interactions.All the concurrent or synchronized interactions can be clearly represented by adopting finite state machines to generate interaction paths.However,finite state machines can only describe the transitions between states,and they can not describe the attributes of each interaction.In this dissertation,a textual definition of interaction methods is given to describe the attributes of relationships among interactions by transition states,suitable working scenarios and ambiguities.(2)An algorithm for selecting interaction paths is proposed,which can help select interaction paths for suitable scenarios,according to the attributes of different interactive texts.Suitable interaction scenarios can be selected so that this interaction paths can be distinguished from others.This algorithm can help reveal different user situations according to different scenarios in complicated contexts,thus helping to choose best scenarios to ensure the robustness of interaction.(3)An automatic tool is developed to generate combined interaction methods.Firstly,the tool can convert textual descriptions of interaction into adjacent matrix by using the interaction text algorithm.Adjacent matrix has its advantages in identifying and planning paths,which can automatically generate interactive methods for certain scenarios.A multi-modal markup language is used to record particular interactive methods used in a certain scenario,so as to increase multi-modal interaction designers'efficiency.(4)Lastly,a set of Problem Frames tools are taken as the subject for case studies.A more natural way of operation is added to the set of tools-gesture-based operations.Perceptual coupling technique is used to track gesture with an inertial navigation sensor assisted cameras,in order to improve the speed and accuracy of gesture recognitions and tracking.In addition,the proposed method of generating combined multi-modal interactions based on scenarios is empirically validated while ensuring high robustness.