Research On The Robotic Grasp Technology Based On Multimodal Deep Learning

Robotic grasp problem is a key issue in robot community.In recent decades,researchers have explored various aspects in depth ranging from structure design of the gripper and planning and control for grasping to multi-sensor fusion.It can be seen from the research changes from analytic methods to data-driven methods that new technologies and new methodologies are applied to robotic grasp problem continuously.The age of big data and deep learning has come.The combination of the new data scale and the new research method can product inestimable value.Considering the inspiration given by deep learning in other research areas,this paper applies deep learning to robotic grasp,and builds grasp classifiers to achieve the building of the robotic grasp system.A multimodal grasp classifier based on deep learning is established.Taking the distinction between data modalities into account and abandoning the simple fusion approach in data input layer,this paper builds grasp classification model using a late fusion approach based on multimodal convolutional neural network(CNN).First,two single modality CNN models are built on color image and depth image respectively.Next,these two classifiers used as feature extractors extract the color and depth grasping features,and the fusion features are used to train a top-layer classifier.Integrating the feature extractors with the top-layer network,the complete multimodal grasp classifier is built to achieve the grasp's precise classification.A robotic grasp system is constructed.This paper constructs the robotic grasp system by embedding the multi-modal grasp classifier in the grasp system.The system consists of several components.First,interpolating the missing depth pixels by the median filter,calibrating the Kinect to get the aligned color image and the scene point cloud in the robot coordinate frame belong to the data acquisition section.Second,segmenting the object from background by Random Sampling Consistency algorithm,detecting the major orientations of the grasped object by Sobel operator and discretizing the grasp search space belong to the candidate gras p sampling section.Third,the multimodal grasp classifier ranks candidate grasps and gets the best one.Fourth,removing outliers,employing mean filtering operation to obtain smooth point cloud for the normal vector estimation and planning a grasp config uration according to the object point cloud corresponding to the best rectangle belong to the rectangle-configuration mapping process.Finally,the Baxter Research Robot reaches the destination joint position obtained by the solution service of the Baxter inverse kinematics and executes the grasp.Integrating all of the above components,this paper builds the robotic grasp system.Three grasp classifiers are evaluated and the experimental study of the robotic grasp system is carried out.This paper assesses three classifiers on the Cornell Grasping Dataset,and detection results show that the multimodal classifier produces significant performance improvements over the other two single modality classifiers.Lastly,experimental system is built including acquiring scene data by Kinect,infering the best grasp configuration by the workstation and grasping the object by the Baxter robot.Grasping experiment results demonstrate that the grasp system mentioned in this paper is feasible and robust.