Research On Manipulator Control System Based On Gesture Recognition

With the development of science and technology,mechanical arm is more and more used in industrial manufacturing,medical treatment,aerospace,deep sea exploration and other fields.With the diversification of application scenarios and the increasing complexity of work tasks,people have put forward higher requirements for the interaction mode of manipulator.At present,the most widely used manipulator control mode is still using mouse and keyboard or manipulator manipulator control panel to control the manipulator arm.For people in three-dimensional space in life,this method cannot be simply controlled,so a more convenient and direct human-computer interaction mode will be needed.In daily life,hands do most of the work,so gesture interaction is also one of the simplest and most direct ways.At the same time,the mechanical arm also realizes mechanical movement by imitating the joint structure of human hand,and gesture interaction will be the simplest and most direct way.This topic combines gesture interaction with manipulator control,aiming at using the intuitive and convenient gesture interaction to develop a simple and convenient manipulator arm control system.The specific research contents are as follows:(1)The hand centroid tracking was realized by using Convolutional Posing Machines(CPM)algorithm.The average values of key points 0,15 and 7 were approximated to the centroid points of the hand,and the Kalman filtering method was used to track the centroid of the hand to improve the stability of the hand centroid tracking.In order to obtain the 3D coordinates of hand tracking points,the 3D tracking of hand centroid was realized by combining the method of depth camera.(2)Static gesture recognition and dynamic gesture recognition are realized.The Mobile Net?v2 network is used to recognition the static gesture.This network is a kind of lightweight image classification network,which can use less computing power and memory resources to achieve high accuracy image recognition tasks.Dynamic gesture recognition adopts TSN?LSTM model,LSTM network is used to extract video timing information,which greatly improves the recognition accuracy of dynamic gesture.(3)The kinematics of the manipulator is analyzed and deduced.The forward kinematics is derived and realized by the traditional coordinate system transformation method.By establishing D-H matrix for all adjacent bars of the manipulator,the position and pose of the middle bar are used to realize the motion representation of the end points of the manipulator.The inverse kinematics adopts the analytical method.By building a model for the manipulator arm,the position and pose of each bar of the manipulator arm are pushed backward from the end point of the manipulator arm by a recursive method opposite to the forward kinematics.(4)User interface design,using Py Qt tool to design the control interface,providing a simple and convenient visual window.(5)The realization of the whole system is described,including the acquisition of depth camera video,correction of depth camera and RGB camera,gesture recognition engineering realization,etc.(6)The system verification platform was built,and the three important modules of the system design(hand tracking,static gesture recognition,and dynamic gesture recognition)were experimented to verify the feasibility of the system...