Research On Human-Swarm Natural Interaction Technique And System Based On Multi-Modality

In recent years,research on Swarm Robotic System has received increasing attention.The large number of robots of the swarm and the partial intelligence of robot has put forward the new requirements for the interaction between human and robot.How to realize the effective interaction between human and Swarm in a natural way is one of the issue that Human-Swarm Interaction research needs to solved.In order to solve the problem,this paper developed a new type of human-swarm natural interaction technique and system based on multi-modality,with the cooperation of three-dimensional gesture modality and natural language command modality and the complementation of the interaction information,achieving an natural and efficient interaction between human and swarm.The main work includes:(1)Three-dimensional gesture interaction: The system employs Leap Motion as the gesture sensor to capture the track of hand;The three-dimensional lasso technology is proposed,using cubic spline to describe the lasso curve,calculating the relationship between the lasso and swarm robots after transformed to screen space,enables the operators to select a batch of the swarm robots;Using AABB and OBB as the collision bounding box of digital map and the index finger of virtual hand respectively,detecting the collision with separate axis algorithm,realize the interaction of coordinates control.The speed-based adaptive filtering algorithm to process the gesture data,realizing the interaction of motion trajectory control.(2)Natural language command understanding and multi-modality integration: The speech signal is recognized by the Speech Platform SDK;The control command library and command labels are defined,and the corpus of corresponding natural language command is collected.The corpus of corresponding natural language command and corresponding command labels are used as the dataset and classification labels respectively to train the text classifier based on the maximum entropy model to understand the natural language command;After the process of command understanding,the command label is integrated with gesture information from threedimensional gesture in the manner of command label filling,realizing the multi-modality cooperation of gesture modality and speech modality,which reduces the cognitive load of the operator.(3)Designed and implemented a human-swam natural interaction system based on multimodality.The system employs a head-mounted augmented reality display device HoloLens as the visual feedback modality,with the advantages of portable and intuitive.Experiment of swarm robots selection and experiment of motion trajectory control are carried out,verifying the feasibility and availability of the system.