Research On The Application Of Manipulators In Automated Testing And Gesture Imitation Technology

With the development of the times and the improvement of productivity,many people are freed from tedious work,all of which are attributed to the development of industrial robots.At the same time,industrial robot technology has also been greatly improved,human-computer interaction technology has also been rapidly developed,and its application scope has become wider.The human-computer interaction method is more in line with the interaction between humans,such as human faces.Smart speakers that can recognize and talk,etc.These interaction methods greatly reduce the operation of the human body to operate the machine-related instructions,so that the human body can interact with the machine more naturally,and greatly improve the comfort of the human when using the machine.Human-computer interaction technology will be the key research object of scientific researchers in the foreseeable future.With the advent of the mobile Internet era,the ways of human-computer interaction have become richer.The human body can establish a rich interaction with the machine through the application program running on the smart terminal,and at the same time it also creates a blowout development of the application program.An application program is essentially a computer program,and it needs to be constantly repaired and updated in order to better interact with the user.In addition to the way the application interacts with the human body,the human body can also control the robot through its own limb movements to achieve the purpose of human-computer interaction.Therefore,the work of this thesis is mainly in the following two aspects:On the one hand,in view of the problem that third-party frameworks are needed for automated testing in the current market,this thesis proposes a brand-new automated testing system.Firstly,use wireless projection technology to replace the traditional camera.This not only saves testing costs but also reduces the difficulty of identifying user interface elements.Secondly,advanced optical character recognition technology is used to accurately locate the text element to be clicked from the complex background image.When image elements need to be located,the normalized correlation matching algorithm is used for positioning.Finally,the positioned coordinates are transferred to the high-precision UR3 robot to perform the click operation.The system can perform automated tests on Chinese and English mobile applications respectively.On the other hand,aiming at the problem of real-time capture of human body joint information to realize the robotic arm imitating the human body to grasp objects,this paper proposes a real-time grasping system.First,the Kinect sensor is used to capture the joint motion information of the human arm and the Vicon optical motion capture system is used to construct a rigid body for the human arm and fingers,and then the joint motion information is captured in real time.Second,the "50-frame detection algorithm" is used to avoid erroneous jittering of joint information caused by human arm vibration.Then,the method of circular queue is used to solve the problem of robot arm instruction coverage.Final,the collected movement information of the arm and finger is transferred to the manipulator and the manipulator to realize the movement and grasping action.The system can grab hard objects and soft objects.