Research On Intelligent 2D And 3D Visual Servo Control Of Industrial Robots

Robot perception has been an important research area for many years.In this regard,computer vision plays an important role in enabling the robot to visually perceive its environment.The control of robot solely based on its visual feedback is known as visual servo control.There have been various attempts to improve the state of the art in visual servo control and enable it for robust application in industrial scenarios,however,few works have been presented so far which perform rigorous study of visual servo in challenging industrial environments.In this dissertation,various experiments related to the application of computer vision and visual servo algorithms in highly demanding industrial scenarios are presented.First of all,the idea of position based visual servo was investigated for an auto-part spray painting robot.In order to enable it with perceiving the various types of geometries in real time,point clouds were employed.With the idea of dimensionality reduction,3D point clouds were projected onto a 2D planar subspace,and ideas from 2D image processing,like contour searching were used.A modified feature extraction pipeline was developed which used fan shape model to extract useful features from the binary images of the car parts.Statistical matching was used to recognize the auto part from a finite set,which was compared with other state of the art object recognition methods in point cloud processing.As per the computational cost comparison,the proposed algorithm provided real time performance as compared to other famous feature extraction techniques in the widely popular point clouds library.For the pose estimation part,the famous iterative closest point algorithm was modified,its problem with the convergence to local optimum was addressed by combining it with genetic algorithm.In comparison to the vanilla configuration of iterative closest point algorithm,the modified approach performed better and reduced the pose estimation error which was vital for optimum paint coat thickness and vision based control.Secondly,another challenging industrial scenario was chosen as the quality inspection of remote radio units in telecommunication industry.Modern industrial standards have strict requirements on the robustness and real time operation of automation systems.In order to meet these requirements,image based visual servo was studied in detail.As computer vision forms a vital part of the visual servo control,therefore an algorithm wasdesigned to visually recognize the power port of the remote radio unit from a stream of images.Image based visual servo depends on the continuous tracking of the region of interest for the generation of control command,therefore a reliable 6 degrees of freedom object tracker was developed based on the ideas of contour searching in computer vision.It was shown that in comparison to state of the art camshift tracker,the proposed algorithm provided superior performance while both of the algorithms matched the computational cost almost equally.Subsequently,for the selection of image features,a unique set was chosen which related the degrees of freedom of the robot with each of the unique member of this set.In addition,a depth free image Jacobian matrix was derived which related the image feature acceleration with that of the moving camera's acceleration,the resultant control design was carried out by using the theory of variable structure control.A PD-type control law was constructed but the design parameters were reduced to one,and the stability was proven using the Lyapunov stability analysis.Finally,before realizing the designed image based visual servo pipeline on a physical system,numerical simulations were performed to check the stability,consequently the algorithm was implemented on an industrial arm-type manipulator with the moving camera setup.The experimental validation supported the research findings and proved the robustness and applicability of designed algorithm in an industrial scenario.In addition to the experimental study of classical visual servo approaches,advancements in modern artificial intelligence based based visual servo techniques were also studied.Deep learning has revolutionized the area of computer vision and as it forms an integral part of visual servo,therefore it is important to study the advances in this field.For this purpose,deep neural networks were used for the application of position based visual servo control with monocular camera setup in joint space of the robot.Position based servo generally requires the control in Cartesian coordinates of the robot,which in turn requires the forward and inverse kinematic analysis.In this dissertation,the idea of deep neural network as a regression function approximator has been presented which not only estimates the inverse kinematics of the robot,but also the intrinsic and extrinsic camera parameters which have a strong influence on control performance.In practice,estimating these parameters in real time is either computationally costly or non-trivial,using data based approaches in this regard come handy and provide a robust performance.For the generation of training data set,an automated data generation recipe was constructed which generated thousands of training samples virtually.The network training was carried outby using domain adaptation and transfer learning,pre-trained network models were finetuned for the application of quality inspection in remote radio units.This required less data as compared to the training of network from scratch.State of the art Pose Net was modified for this purpose and its optimal hyper-parameters were sought with grid search.Finally,the performance of Pose Net was compared with Alex Net and the resulting algorithm was physically realized on an arm-type industrial manipulator with moving camera setup.Finally,the idea of reinforcement learning and its implementation for a direct visual servo task was explored in the end.For modeling sparse rewards reliably,Siamese deep neural networks were used.The ideas of domain adaptation and transfer learning were explored,while the data set generation algorithm from previous work was slightly modified.In this work,the pre-trained weights of VGG16 deep neural network were used as image feature extractors.The subsequent layers of the networks were designed on the line of Alex Net.Rigorous experimentation was performed to obtain the optimal structure of the resultant network,while the training was carried out with extensive hyper-parameter tuning.It was shown that proposed solution performed well,not only in the application of quality inspection of remote radio unit,but also on arbitrary scenes.Moreover,the performance was also checked on benchmark data sets.Finally,the performance comparison was carried out upon the generated data set,classical keypoint matching was used for this purpose,and it was shown that the proposed setting provided a better solution to the problem of scene identification in deep reinforcement learning of visual servoing agents.This work serves as the basis of future research for understanding and implementing the deep reinforcement learning in direct visual servo,with this work already available,the design of scene identification techniques in visual servo becomes less challenging.