| China has a large area of tomato planting,and the annual yield of tomato accounts for about 7.5% of the total yield of fruit and vegetables in China.At present,the harvesting of processed tomatoes has been mechanized,and the picking of fresh tomatoes still relies on manual work,which not only takes time and effort,but also has low economic benefits.The research and application of fruit and vegetable picking robots is a hot topic in today’s research,but there are only a few picking robots that can be truly applied in the actual agricultural environments.One of the limiting factors is that the picking end-effector lacks the ability to perceive the target fruit information during the process of grasping fruits and vegetables,resulting in different degrees of grasping and crushing damage,grasping slippage,fruit stalk falling off and other phenomena.In this paper,the visual tactile sensing device for picking end-effector is taken as the research object,and the mechanism of visual tactile sensing based on visual detection contact deformation is studied,and a visual tactile sensor based on visual detection contact deformation is designed and prepared.Based on this,the analysis method of tomato size and posture based on spatial multi-visual tactile sensing is studied,and the tomato picking end-effector with visual tactile sensing is designed and developed.Finally,the experimental verification of tomato size and posture information acquisition is completed.The main contents of this paper are as follows:(1)The mechanism of visual tactile sensing based on visual detection of contact deformation is studied,and two methods for obtaining the normal displacement of the embedded marker points in the elastomer of the visual tactile sensor are proposed.The first method is based on the improved DFD method to obtain the normal displacement of the embedded marker points in the elastomer.Based on the traditional DFD method,considering the scaling of the marker points caused by the contact deformation of the elastomer,the relationship between the normal displacement of the marker points and the imaging size of the marker points is re-derived and improved,so as to obtain a more accurate normal displacement of the marker points.The second method is based on the BP neural network method to obtain the normal displacement of the embedded marker points in the elastomer.By modeling simulating the process of grasping tomatoes with different size and postures by the tomato picking end-effector with visual tactile sensing,the elastomer node displacement dataset is constructed.The shear displacement of the node is used as input and the normal displacement is used as output to establish a BP neural network prediction model.(2)The parametric structure design of the visual tactile sensor was carried out,and the visual tactile sensor based on visual detection of contact deformation was prepared by using 196 silicone gel,jelly glue,micro camera,LED lamp,transparent acrylic board and other materials.The physical characteristic parameters of the elastomer were obtained by tensile experiment,and the three-dimensional displacement measurement experiment of the embedded marker points in the elastomer was carried out.The experimental results show that the prepared visual tactile sensor can well obtain the three-dimensional displacement of the embedded marker points in the elastomer.The calculation error of the shear displacement of the marker points is between0.0266-0.1323 mm,and the RMSE of shear displacement measurement is 0.0916 mm and 0.0918 mm,respectively.The RMSE of the normal displacement of the marker points obtained based on the improved DFD method is 0.175 mm,while the RMSE of the normal displacement of the marker points obtained by the BP neural network prediction is 0.29 mm.(3)The research on the analysis method of tomato size and poseture based on spatial multi-visual tactile sensing was carried out.Taking the commonly used three-finger and two-knuckle picking end-effector as the object,according to the layout of the visual tactile sensor,the coordinates of embedded marker points in the elastomer under the coordinate system of each visual tactile sensor are obtained respectively.By calibrating the micro-camera and solving the forward kinematics model of the single finger mechanism,the corresponding rigid transformation matrix is obtained.The coordinates of the embedded marker points in the elastomer under the each visual tactile sensor coordinate system are converted to the same base coordinate system,so as to obtain the point cloud of the marker points(equivalent to the appearance local point cloud of the grabbed tomato).A tomato with average size and standard shape was selected as a template,and the tomato template was scanned by a 3D laser scanner to obtain the tomato template point cloud.Finally,based on the point cloud registration algorithm,the tomato template point cloud and the point cloud of marker points are initially matched and precisely matched,and the size size and posture information of the captured tomato can be obtained.(4)Taking the three-finger and double-knuckle picking end-effector as the design scheme,the structural design of the picking end-effector was completed based on the statistical size of the tomatoes.By analyzing the fruit grasping motion of the picking end-effector,the size parameter and layout of the visual tactile sensors on each knuckle of the picking end-effector were determined.The 3D model of the tomato picking end-effector with visual tactile sensing was established and the prototyped was made,and the tomato size and posture information acquisition experiments were carried out.The experimental results show that the average absolute error of the transverse and longitudinal sizes of tomatoes are 11.37% and 11.05%,respectively,and the average absolute error of the horizontal angle and vertical deflection angle between the tomato fruit axis and the projection plane of the field of view are 9.48% and 10.97%,respectively. |
