Research On One-person Two-machine Control Technology Based On Multi-modal Behavior Characteristics

With the development of human-computer interaction technology,the human-computer cooperation mode of single task can not meet the increasing needs of human-computer interaction.When the user's environment is complex and the interactive tasks are changeable,it is difficult to guarantee the basic human-computer interaction characteristics such as high efficiency and nature.Therefore,multi-task-oriented multi-modal human-computer interaction technology has been widely concerned and deeply studied by more and more scholars.Among the approaches of human-computer interaction,eye movement,gesture and voice are the most commonly used and basic interaction methods,which have the advantages of direct and bi-directional interaction.Among the many choices of interaction objects,unmanned ground vehicles and unmanned aerial vehicles have become the representatives of this field due to their extensive applications in military and civil fields.Therefore,based on the theory and method of behavioral science,kinematics,control science,human-computer interaction technology and so on,considering the behavioral characteristics of eye movement,gesture and voice,combined with the movement character of unmanned ground vehicles and unmanned aerial vehicles,“one-person one-machine” prototype system architecture is constructed and“one-person two-machine” prototype system model is designed.Finally,the systems are implemented and validated through the experiments.Firstly,this paper summarizes the research status of eye movement,gesture,voice and multi-modal control technology at home and abroad in recent years,and introduces the research content and chapter arrangement of this paper.Secondly,according to the behavioral characteristics of eye movement,gesture and voice,the model of each modal and its corresponding interactive algorithm are constructed.Next,considering the characteristics of the two controlled objects,the two-dimensional and three-dimensional motion models of unmanned ground vehicles and unmanned aerial vehicles are analyzed and constructed respectively.Then,based on the behavioral characteristics of the modal and the motion characteristics of unmanned ground vehicles and unmanned aerial vehicles,the realization mechanism of the three-modal control unmanned ground vehicles and unmanned aerial vehicles is designed,and a control modal switching mechanism is proposed,so as to construct“one-person one-machine” prototype system architecture and apply it to the flight control of the unmanned aerial vehicles.Finally,based on the “one-person one-machine” prototype system architecture,a controlled object switching mechanism is explored,then the “one-person two-machine” prototype system model is constructed,and its application in collaboration control of unmanned ground vehicles and unmanned aerial vehicles is realized.For the “one-person one-machine” and“one-person two-machine” prototype system,the related experiments are designed and their rationality and effectiveness are verified.