The Research And Implementation Of The Geometric Learning System Based On Multimodal Interaction Technology

Multimodal interaction technology makes full use of the parallelism of the various perception and control action, expands input and output bandwidth, and improves the naturality and flexibility of interaction. This paper researches and explores the application of multimodal interaction technology in teaching, and combines pen input, speech recognition with mouse and keyboard. Ultimately, through the development of the geometric leaning system for primary and high school, this paper deeply researches the problem that how to apply these technology.During the research, the author found that the whiteboard most used the WIMP(Window, Icon, Menu, Point Device)interface when teachers used computer in the classroom. Under the guidance of relative theory of multimodal interaction technology, this paper makes use of handwritten screen, microphone, Whiteboard, and other tools to develop a software system which more Suitable for multimodal interaction. This system, which Combining voice with pen input, is designed According to PIBG paradigm(Physical, Icon, Button, Gesture), developed geometric learning system based on pen input platform of Chinese Academy of Sciences and Microsoft Speech SDK. It makes speech recognition to be an effective accessorial interaction means of pen input. We Import the scenes design method to the process of multimodal human computer interface design, and do some exploration of availability software development. Moreover, the structure description of the task and the parallel algorithms are studied for the strategy of the information integration in this paper. Another important work in this paper is to integrate geometric shapes recognition into pen input system. This recognition method combines geometric shapes and command gesture recognition with characters recognition. The course of recognition is independent of the amount and order of strokes.This paper was supported by the National High-Tech Research and Development Program of China (863 Program) (No.2006AA01Z328) and the Open Foundation of State Key Laboratory of Computer Science, The Chinese Academy of Sciences (No.SYSKF0704).