Research On Multi-modal Interaction Technology For Human-Multi-Robot Cooperative Control

In recent years,the research of multi-robot systems has attracted more and more attention from scientific researchers.Compared with single robots with complex structures but limited task execution capabilities,multi-robot systems composed of multiple robots with relatively simple structures and functions can complete tasks that are difficult for a single robot to complete through inter-member complementary capabilities and action coordination.Multi-robot systems has good robustness,scalability and other advantages,so it has good application prospects.Although research in recent years has greatly improved the autonomous intelligence of multi-robot systems,due to the current level of technology,when the environment or tasks performed by multi-robot systems are more complex,it still cannot achieve complete autonomous intelligence.Therefore,the intervention of the operator is beneficial and necessary for the task execution of the multi-robot system.An important direction of multi-robot system research is human-multi-robot system interaction.However,the existing human-multi-robot system interaction research is not sufficient.On the one hand,it lacks a natural and efficient command input method,and on the other hand,it lacks an intuitive feedback method for multi-robot motion state.Therefore,in order to provide users with an easy-to-operate and easy-to-understand human-multi-robot system interactive interface,the research on multi-modal interaction technology for human-multi-robot cooperative control has been carried out.Firstly,the simulation framework of multi-robot system is introduced,which realizes the formation and evasion control of multi-robots.At the same time,the hand controller is used to issue speed control instructions to command the movement of multi-robot formations,which lays the foundation for verifying the effectiveness of the interactive interface.Then based on the simulation framework,the following three tasks were carried out:1.A multi-robot system formation command input technology based on a hybrid gaze-brain-computer interface is proposed.The user can issue formation control commands to the multi-robot system by looking at the menu options on the interactive interface,and at the same time propose the brain-computer interface and gaze tracking for command selection in order to improve the accuracy of command selection,this method uses the unused EEG interface and gaze tracking as the command input channel,which is expected to provide users with a natural,intuitive and efficient way of inputting formation commands.2.A wearable haptic feedback device has been designed and a multi-robot system formation command execution state feedback technology based on the device has been proposed.The device can provide users with squeeze tactile feedback and vibration tactile feedback,which are used to feed back the execution status of multi-robot size transformation and formation transformation instructions.This method is expected to improve the user's situational awareness of the execution state of formation commands by multiple robots.3.A human-multi-robot interaction platform based on a multi-modal interactive interface is built,and the effectiveness of the proposed interactive interface in human-multi-robot interaction is verified through experiments.The first experiment is a multi-robot control experiment using a hybrid gaze-brain-computer interface and a hand controller as an input modality.The experimental results show that the proposed formation command input method based on the hybrid gaze-brain-computer interface is natural,intuitive and efficient,and can reduce the user's workload and improve the efficiency of task execution;the second experiment is a multi-robot formation command execution state feedback experiment based on a wearable haptic feedback device.The experimental results show that the use of haptic feedback overcomes the problem of insufficient feedback information only through visual observation,thereby enhancing the user's situational awareness of the multi-robot system and reducing the control differences between individuals.