Research On The Theory And Technology Of Simulating Mouse And Keyboard Behavior Based On Kinect

Traditional mouse and keyboard control methods can not fully meet the needs of the disabled People,field work and other personnel,so it is proposed to use Kinect to identify the different action types to achieve the simulation of cursor and keyboard behaviors,to let the computer observe human motion and understand human action commands,so that it can get better interactive experience.In this is,the main purpose is to apply the color image captured by the Kinect sensor to the action recognition,to achieve the mouse and keyboard behavior simulation.The main works of this paper are:1.The mouse and keyboard behavior simulation involves the coordinate mapping and coordinate conversion of skeletal point,the calculation formula is given.Coordinate mapping realizes the cursor and HandRight skeletal points have the same moving trial,at the same time by adjusting the formula factor lx,ly to find the best mapping effect.Coordinate conversion realizes to match human joints and color images.When the coordinate conversion is completed,the purpose of reducing action types is achieved by extracting the pixel color of the skeletal point.2.In the key press and bounce behavior simulation involved to action recognition of making a fist and hands outstretched,this paper proposes to recognize the fist and the hands outstretched by joint angles computing method,and find the critical angle of fist recognition.The state is fist When the joint angle is less than a,and the state is hands outstretched When the joint angle is more than a.3.Through the upper or lower rotation of the right hand to simulate the mouse single-click or double-click,due to excessive force,there will be mutual interference between the two movements,which suspension frame technology is introduced to avoid interference.In this paper,the theory of suspension frame is described,at the same time through the contrast experiment to suspension frame technology can effectively avoid the mutual interference between the actions.4.Cursor position is out of screen range when simulating cursor movement.In this paper gives actual position formula of cursor when the cursor crosses the border.5.In the simulation of key behavior,due to the inevitable jitter of the human body results to jitter of cursor,this paper is through the contrast experiment to verify jitter elimination technique can avoid the jitter of the cursor,to improve the accuracy of keyboard behavior simulation.6.When the skeletal points are far away from the Kinect sensor or at the edge of the Kinect sensor field of view,it will cause the sensor to capture the wrong skeletal information,and when the Kinect sensor is rotated,so that skeletal point is located at the center of the field of view,it is the best way to capture the information of skeletal point.Therefore,this paper presents a new method to calibrate skeletal points for red,when Kinect detects that the pixel color are red in the image position which is converted skeletal point,it can be automatic calculation angle of rotation,and through sensor rotation,so that marked the red skeletal point is located in the center of the sensor field,so as to catching the wrong correct point information,and to achieve the purpose of improving the recognition accuracy and precision.