Deep Learning Based Hand Key Points Detection And Its Application On Mobile Devices

With the rapid development of HCI and wearable smart device,hand gesture based interaction has become the research hotspots of computer vision and AI field.However,there are two problems with hand key points detection.One problem is that there is lack of egocentric RGB dataset for hand gesture based interaction,which is hindering the research on the data-driven approaches(such as deep learning).Another problem is that the traditional hand key points method is easily interfered by the quality of dataset,background color and illumination conditions,the algorithm performance has encountered a bottleneck period.What's worse,the large models and high computation cost lead to poor real-time performance so these algorithms are not widely applicable in mobile device.Therefore,how to obtain the positions of hand key points in real time for subsequent interaction becomes an urgent problem to be solved.To solve these problems,we focus on deep learning based hand key point detection and its applications on mobile device,including egocentric fingertip detection,third perspective hand pose estimation,model compression and hand key point detection on mobile device.Specifically,the contributions of this thesis include:Since there is lack of egocentric RGB dataset for hand gesture based interaction,we build a new dataset named EgoGesture and propose a heatmap-based solution for egocentric fingertip detection,which is named YOLSE(You Only Look what You Should See).Compared with state-of-art algorithms,our egocentric fingertip detection algorithm not only meets the requirements of accuracy and real-time performance,but also reduces the dependence of the fingertip detector on the hand detector.For third perspective hand pose estimation,we creatively decompose the hand pose estimation task into two sub-tasks,2D hand key point detection and depth value regression.We propose a multi-task attention-based convolution network to solved these two tasks.Experiment shows that our method can accurately obtain the position of the two-dimensional hand key points and use the depth value so that we can get the three-dimensional coordinates of the hand key points.Last but not least,we propose a mobile fingertip detection model named MobileYOLSE and design a real-time fingertip detection system on IOS.MobileYOLSE greatly reduces the parameters and size of the model by using the inverse residual module.Using the L1 channel prune and the combination of the parameters of batch normalization layer and convolution layer,the size and computation cost of our model is reduced greatly without losing any accuracy.And we achieve real-time fingertip detection on CPU of mobile device.