The Research Of Embedded Virtual Keyboard Based On Machine Vision

With the development of virtual reality technology, the way of human-computer interaction has been changed from traditional keyboard and mouse into three-dimensional interaction. Virtual keyboard can meet the requirements of virtual reality’s application scenarios because it determines the user’s input through using cameras to detect the fingers’ movement in the three dimensional space. The implementation of current virtual keyboard is using infrared camera and infrared laser to assist location. Although the principle of this plan is simple, it is susceptible to the interference of environmental lights. To solve this problem, this paper designs and implements an embedded virtual keyboard system based on machine vision, using binocular camera detection instead of locating by infrared camera and infrared laser. The main research work of this paper includes three parts:1) The framework design of virtual keyboard system. In hardware design, the system uses ARM as core processor. ARM processes the images that captured by binocular camera, then send the results to computer. In software design, the system’s program uses MVC structure model, which separates the program’s data access, interface, control logic and encapsulates them in different modules respectively. The coupling of program can be reduced in this way.2) The research and improvement of the virtual keyboard system’s algorithm. Firstly, this paper proposes an improved foreground segmentation algorithm and implements the extraction of hand foreground area. Secondly, this paper improves the instability of the single fingertip detection algorithm based on K-vector and proposes an improved multiple fingertips detection algorithm based on convex hull and K-vector. The improved multiple fingertips detection algorithm locates the coordinate of fingertips by calculates the K-value of the hand contour’s convex hull vertex. Finally, this paper calculates the depth of fingertip that is used to determine whether the fingertip pressed, by using the binocular distance measurement principle.3) The implementation of virtual keyboard system. Under the Linux and QT development environment, this paper uses C++ and OpenCV to program and implements the synchronous acquisition of binocular camera through multi-thread technology and V4L2 framework. Then, this paper completes the location of keys and the recognition of the fingertips’ state through the information of fingers’ position and depth, and implements the embedded virtual keyboard system based on machine vision.Experimental results shows that the virtual keyboard designed by this paper can detect the movement of user’s fingertip in real time and reduce the interference of environmental lights. And it demonstrates that the virtual keyboard system has achieved high percentage of accuracy.