| With the development of artificial intelligence,there are more and more scenarios for the application of intelligent service robots,including hospital care robots,bank front desk service robots,and home service robots.Real-time detection of human targets and continuous attention to service objects is an important function of the above-mentioned service robots.In the real environment,issues such as illumination variation,object occlusion,scale transformation,and target loss will affect the positioning and tracking of the target,reduce the experience of human-computer interaction,and restrict the further development of intelligent server people.How to improve the performance of person detection,location,and tracking is the focus of this article.The research work includes:(1)Extract and construct a person-related dataset,including various data samples of different lighting,different scenes,rich person scales,target occlusion and so on.YOLO v5 model is adopted to train a person detector,and then train four different network structures at the same time.Select the best network model through a comparative test to achieve the function of real-time detection of people.(2)Propose an object detection method based on fusing vision and person’s sound source information.It’s important to improve the accuracy of target object detection and the robustness of tracking,this paper makes full use of the character’s sound.Firstly,a sound source localization model is trained by fusing the two modalities of image and audio.According to the visual information and audio characteristics in the video,the sound source location in the video is judged to assist the robot in positioning and tracking.The Res Ne Xt-101 module is used to extract the image features of the video,the Sinc Net module is used to extract the audio features,and then the active sound source localization is performed based on the fusion features of the audio and video images.On the AVA-Active Speaker dataset,AUC achieves 92.4%.In addition,this paper also uses the microphone array of the Pepper robot to determine the direction of the person’s sound source based on the Time Difference of Arrival(TDOA)algorithm.(3)Construct an object tracking module based on the Siamese framework for feature extraction.The feature extractor uses Res Net50 to extract the features of the template image and the search image.After each branch,use the Transformer’s encoder to encode the template features and search features.And input the features after the encoder into the decoder for the fusion of the two branch features,make full use of the global information of the features through the Cross-attention mechanism,and then predict the category and corresponding coordinates based on the fused features.Extensive experiments are performed on GOT-10 K,La SOT,Tracking Net,and other datasets,the results prove the effectiveness of the target tracking module in this article. |
