Research On Visual-tactile Fusion Of Robotic Grasping

With the rapid development of robotics,robots have gradually replaced humans and entered a wider range of applications.Due to the complexity of the work environment and the diversity of work tasks,robots need different types of sensors to capture working condition information to improve their perception ability.The limited visual information and insufficient tactile perception ability during robot grasping operations have become the bottlenecks in robot technology that need to be addressed urgently.In this thesis,a Visual-tactile fusion method in robot grasping is studied.The main research work completed is as follows:(1)A method of Collecting and preprocessing tactile data.Firstly,in order to solve the problem of difficulty in obtaining tactile information during traditional robot grasping operations,a tactile sensing unit based on XELA tactile array sensors was constructed,which collected tactile data for analysis,and imaged the tactile data.Then,aiming at the interference and noise generated during the process of the mechanical gripper approaching the target object,a haptic data filtering algorithm based on the Hale wavelet transform was studied,which improved the tactile perception accuracy of the captured object,a visual-tactile dataset(VTDA)was built.(2)The construction of a visual-tactile fusion sensing network model.Combining the current typical architecture of visual and tactile perception,a C3D-CBTR visual and tactile fusion sensing network model is constructed to achieve a deep fusion of visual and tactile features.Aiming at the difficulties of spatiotemporal feature modeling and temporal feature fusion in visual tactile fusion algorithms,a visual tactile fusion model based on the n global attention mechanism was constructed.This model introduces different attention mechanisms to construct a global attention mechanism from three aspects of image and tactile space,channel,and timing,respectively,to achieve the combination of 2D and 3D dimensions,spatiotemporal feature weighted fusion of purpose-frame frame association,and prediction of multiple types of grasping states.Through comparative experiments,it is found that the prediction accuracy of the proposed model on the self-built dataset VTDA reaches 83.84%,and the prediction accuracy on the public dataset GSA and dataset VTD reaches 99.05% and 98.11%,respectively,verifying the effectiveness of the algorithm.(3)A grasping method of visual-tactile fusion.Aiming at the problems of object sliding and serious deformation caused by insufficient process perception information during robot grasping,a robot grasping strategy based on visual-tactile fusion was studied.Specifically,a real-time grasping adjustment strategy for robotic grippers based on a visual-tactile fusion grasping state classifier is designed to adjust the grasping width and force at the next moment according to the grasping state at the current moment,achieving fine adjustment of robot grasping posture.Based on this theory,a robot visual tactile hand-eye system platform was built with a fixed wrist camera and XELA tactile sensor above the handle,and comprehensive verification experiments were conducted.The Visual-tactile fusion method in robot grasping proposed in this thesis is based on the built robot Visual-tactile fusion grasping system platform,and physical object grasping experiments are conducted to verify the algorithm performance.Through experiments,the average success rate of grasping 10 objects has reached 93%,verifying the practical feasibility of the method in this thesis.After testing,the average success rate of grasping 10 types of objects reached 93%.The comprehensive validation experimental results show that the robot grasping visual-tactile fusion algorithm proposed in this thesis has strong practicality.The research related to this thesis also has certain academic significance for enriching the theory of robot visual tactile perception.