Study On Human-Social Robot Interaction Based On Multimodal Sensors Fusion And Perception

With the advent of an aging society and the continuous increase of the empty-nest elderly and children,social robots have great application potential in these special communities such as autistic children,older singletons,and disabled people.Social robots can help autistic children improve their social skills,help elderly people with life difficulties cope with daily life independently and safely,and improve their quality of life.However,most of the existing human-social robot interaction modes rely on a single sensor system.As a result,problems such as single interaction mode,poor interaction performance,and weak learning ability restrict its applications.Therefore,researching new human-social robot interaction methods to solve the above problems is the key to the development of robots with important theoretical and application value.This thesis elaborates on the development status of human-social robot interaction based on multimodal information.The main research works are formed by analyzing,refining,and summarizing the issues existing in the current human-social robot interaction process.It studies the methods of human-social robot interaction and social robot learning from demonstration,multimodal information fusion,and perception,etc.,to achieve efficient and intelligent interactions.The main contributions and innovations of this thesis can be concluded as follows.(1)In order to enable social robots to effectively learn human skills,this paper proposes a non-contact skill learning framework based on learning from demonstration with multimodal sensor fusion.The proposed framework can avoid repeated teaching from teachers or parents when teaching children learning skills.Firstly,multimodal sensors are used to collect demonstration data synchronously to provide a complete and complementary information source for social robots to learn human teaching tasks.To improve the accuracy of the demonstration data,the Kalman filter algorithm is used to perform a data-level fusion of the multimodal sensor data.The dynamic time warping algorithm is applied to address the time mismatch of multiple demonstration task trajectories.Then,the teaching task is modeled through the broad learning network to obtain a continuous and smooth task model.The social robot can acquire human skills through learning the task model.Finally,the feasibility of the multimodal skill learning framework is verified by the comparative experiments.(2)To enhance the performance of human-like skills learning,a multimodal unsupervised emotional intention recognition framework is proposed based on human sensorimotor behaviors.The proposed framework aims at enabling social robots to accurately and effectively recognize human emotions to help them make reasonable inferences,decision-making,and appropriate responses during the interaction process.To this end,a deep vision sensor and a wearable device are used to collect human sensorimotor behaviors with two different modalities.Then,the two modal information is preprocessed and fused with feature-level.After that,the fused emotion data are fed into an unsupervised neural network for classification.Finally,several experiments demonstrate the effectiveness of the multimodal emotional intention recognition framework.(3)A multimodal sign language interaction framework applied in social robots is proposed in this thesis.It can help deaf-mute people to better integrate into social life and improve the quality and experience of interaction with normal people.The proposed framework combines the human hand 3D vector and arm surface electromyography signals.The gesture features are extracted from two modal sensor raw data through preprocessing.Then,feature-level fusion is performed for the two types of features to obtain relatively complete sign language information.After that,a deep learning method is used to recognize sign language gestures.The social robot completes the interactions through voice and action feedback according to sign language recognition results.Finally,experimental results verify the effectiveness of the multimodal sign language interaction framework.Compared with other sign language interaction systems,the proposed multimodal framework can effectively solve the problems of limited interaction communities and gesture categories in human-robot gesture interaction.