Multimodal Interaction Technology And Application For Elderly

With the increasing of age, the people’s sense perception and cognitive ability will change. The sense perception and cognitive ability decreased significantly at the age of 60 years and above in old people. The efficiency and accuracy of the interaction which is based on the mouse、keyboard and graphic will decrease, and can’t satisfy old people’s demandsAccording to the old people’s sense perception characteristics, the paper uses the touch、speech input and multimedia output technology, and puts forward a multimodal interaction(MMI) model. Through analyzing the existing multimodal fusion technology, the paper designs a complete multimodal information fusion method and algorithm which is based on the model. To Verify the feasibility of the interaction modal and the effectiveness of the fusion strategy, the paper complete the interaction process and the fusion algorithm implementation by software on Android Platform, and applied to the Chinese chess program running on android handheld device. Finally, the paper design an ergonomic assessment method for the old people, and then evaluate the HCI model and fusion strategy on the MMI system.The paper’s achievements obtained are as follows:(1) Presents a multimodal interaction model for the old people with speech and touch input channel. According to the model, design a multimodal hierarchical fusion strategy based on task and describe the model and the input-output data of each layer.(2) Complete the software development by java of the interaction model and fusion algorithm on the android platform, then design a multimodal interaction system based on an Chinese chess application. People can use one channel serial input or multi-channel parallel input mode to generate the various task and then control the application. Moreover, the system choose the multimedia output mode with speech and graphic and supply the cognitive ability for elderly.(3) Presents a HCI ergonomic assessment model, then complete the interactive test and ergonomic evaluation by using the multimodal Chinese chess system. The test result shows that the old people can execute the interactive tasks naturally and efficiently by using the multimodal interaction model, the fusion results achieved expected effect. Moreover, the ergonomic assessment results show that the system has good comprehensive ergonomics performance.The paper has important significance for the multimodal data fusion technology and the practical application of human-computer interaction for the old people.