The Study On Pose And Position Estimation For Robotic Grasping Based On Convolutional Neural Networks

The localization and pose estimation for targets plays an important role in robotic grasping,since the positions of targets,scene understanding,decision of grasping pose and generation of trajectory strongly depend on the accuracy of this estimation.The 3D sensors make this task more convenient,but it is still challenging to estimate the position and pose of targets only given the 2D image and methods in case have many limitations.Among them,how to understand the targets becomes a more important issue,for which Convolutional Neural Networks are indeed a powerful choice.To this end,two types of methods are proposed to do 3D estimation of position and pose based on CNNs in terms of different experiment environments.These two types of methods both apply the idea of object detection network in 2D world to 3D world and achieve the estimation in grasping task by changing the inner architecture,in-out information and training dataset.In short,the main work of this paper follows below.1)Modeling and calibration.The hand-eye coordination system was established and calibrated,including the intrinsic parameters and distortion factor of the camera as well as the transformation matrix between the camera and base coordinate system of robot arm.Calibration method proposed by Zhang Zhengyou was used in the former one,and regarding the latter one,the solution to the typical problem of AX =XB was found to get the unknown transformation matrix.Finally,the calibration result was demonstrated accordingly and five groups of standard positions and poses were obtained by calibration of board to verify and evaluate the following proposed methods.2)2D trajectory tracking based on CNNs.Only 2D information is needed for 3D estimation when targets move within a certain plane,so two CNN-based networks combined with particle filter and clustering were proposed to do object tracking.Region proposal network could output lots of information of 2D bounding boxes that could be used in particle filter and clustering algorithms,reducing the computation of networks and increasing the speed and accuracy of tracking.Afterwards,two types of datasets proved that the proposed methods improved both the speed and accuracy of tracking.Finally,this method was evaluated to estimate the 3D position and trajectory of one target in the case of this target moving within a certain plane.3)Estimation network of position and pose based on CNNs.Given the feature extracted by CNNs,two branches of networks were designed for 2D information and 3D viewpoint(in-plane rotation)estimation respectively.These two branches shared the CNNs feature,and the first network output the 2D bounding boxes and classification of targets,which could be used to select the branch network for the classifications of viewpoint and in-plane rotation.With the IDs of viewpoint and in-plane rotation,the detected area was compared with the template in each viewpoint to the final 3D estimation of targets.To do evaluation,this estimation network was trained on the synthetic datasets and tested on the 3D printed targets.The results demonstrated that the model can fit in the real environment with good generality although trained on synthetic data.5)Training of models.As for the first type of method,only the first part of object detection network was trained with the second part not updated.In the second type of network,the feature extraction network was frozen and only the two branches of prediction heads were trained separately to avoid overfitting and the influence between two branches.ADAM optimization algorithm was used in the training phase to do the back-propagation and the update of parameters.6)Experiment of robotic grasping.An experiment platform of robotic grasping based on color camera was established to test the performance of position and pose estimation network in the task of grasping.Basically,kinematics and inverse kinematics of manipulator were solved and 3D position and pose of target was obtained by estimation method,which was used in the following trajectory planning of robotic grasping.The experiment result showed that this proposed 3D position and pose estimation method could meet the requirements in the task of robotic grasping.Finally,conclusions of this paper were drawn based on above experiments and pros and cons of proposed methods were analyzed with some suggestions of potential improvements.