Research On Application Of Multimodal Information Fusion In Robot Target Positioning And Grasping

Computer vision-based object recognition and location technology has been widely used in industrial production.This paper proposes an pose estimation algorithm based on the needs of industrial production,by extracting the multi-modal information of two-dimensional images and three-dimensional images,the identification and location of the objects are achieved,and the reliability and robustness of the algorithm are verified by the grasping experiments.In order to increase the reliability of the recognition,the deep learning was used to classify and train different objects,and a model of classification recognition was obtained.It was verified that the model had a good recognition effect through experiments.The content of this paper is as follows:(1)Perform calibration experiments on Kinect2 depth cameras,USB cameras and robots,convert the coordinate systems of the three to the world coordinate system,and provide a theoretical basis for later gripping.(2)The target 2D image information is obtained by USB camera.The contour is recognized through the contour detection and matching process.Then the image SIFT feature is extracted for location tracking and the position of object is obtained.(3)Obtaining a point cloud image by Kinect2 camera and the best model can be sorted through pre-processing,Euclidean cluster segmentation,computing VFH feature and KD-tree searching,identifying the point cloud image.Then the orientation is obtained by registering the point clouds.(4)An pose estimation algorithm is proposed,which combines the above two methods to complete the identification and positioning of objects.The effect of the method is verified by the robotic gripping experiment.The result show that the multi-modal information of two-dimensional image and three-dimensional point cloud image can be used to identify and locate different target objects.Compared with the processing method using only two-dimensional or point cloud single-mode image information,the positioning error can be reduced by 50%,the robustness and accuracy are better.(5)CNN convolutional neural network is used to classify objects.Firstly,collect images of only objects,and then use the “DCGAN” neural network to amplify the number of objects.After that,CNN convolutional neural network was established to complete the classification training of different objects and obtain a classification recognition model.Through experiments,the model can accurately identify different objects.