Research On Finger Interaction Technology Based On Depth Information

With the advancement of computer technology,human-computer interaction technology is also developing rapidly.Part of human-computer interaction has gradually changed from real keyboard control to virtual key touch.The emergence of virtual keyboard is the best application case for virtual key touch.At present,the virtual keyboard mostly uses a hardware solution with a color camera and a laser,but this solution has problems such as limited touch control,low fingertip detection rate,and single interaction mode.The depth information can solve the problem of background interference,and the ability of deep learning in image detection is also very outstanding,which can meet the needs of the system.This article combines in-depth information and deep learning to innovate and implement a virtual button fingertip interaction technology solution with simple equipment,strong anti-interference ability,and rich touch operation.The main contents of this paper are as follows:(1)Aiming at the problem that the external environment has a greater impact on the virtual keyboard,this paper uses a deep camera to build a virtual keyboard touch system,which reduces the complexity of the device compared with the traditional virtual keyboard solution.Aiming at the problem of inaccurate extraction of the fingertip position caused by the influence of light in computer vision,this article adds depth information to the virtual button fingertip touch,and uses the Time Of Flight(TOF)method to obtain the distance between the fingertip and the desktop background.Using the calculated depth difference,a new virtual keyboard touch solution was finally proposed,which solved the limitations of the traditional judgment method.In this paper,we have added a space-free keyboard input to the innovative touch solution,which simplifies the complexity of the visual keyboard and improves the stability of the visual keyboard to a certain extent.(2)Aiming at the problem that traditional gestures are not easy to detect,this paper uses the YOLO v3(You Only Look Once v3)deep learning network in the field of virtual keyboards to detect and track the position of the fingertip.Created a hand image dataset based on the experimental environment of this system,and enhanced the dataset before sending it to the network.Multi-scale changes have also been made in the network structure to enhance the detection ability of small objects,which has made the ability to detect small targets improved to a certain extent.Testing on the data set in this article,the comprehensive detection rate has increased by 2.94%.While detecting the fingertips,this article also recognizes and tracks the virtual keys.When the virtual keyboard is moved,the position of the keyboard after the movement can be recognized in time.(3)In view of the limitations of keyboard control,this article also incorporates the operation mode of air gesture control in the system.The air gesture sets four basic functions,which can be used to cancel,delete,clear,and close the camera.This method simplifies the operation mode and the number of keyboard keys at the same time,and human-computer interaction can be performed through gestures in the air.Compared with the traditional method of fixing the button position,the system in this article is more flexible.Finally,a key click touch experiment and a continuous click touch experiment were performed on the virtual keyboard,and the recognition accuracy rates were 97.13% and 93.33%,respectively.The same test was performed on the keyboard without physical objects,and the recognition accuracy rates were99.00% and 96.25%,respectively.The average number of frames per second(Frames Per Second,FPS)of the system in this paper has reached 33,which satisfies the fluency of use.Finally,the existing methods are compared,and operating experiments are carried out in different environments to verify the effectiveness of the system in this paper.