Research On Human Eye Image Synthesis Method With Style Transfer

Gaze estimation has great potential whether in human-computer interaction or autonomous driving.At present,the effect of the human gaze estimation is affected by the quality of training data.The training data for gaze estimation mainly contains two types:real images and synthetic images.The real image is more in line with the requirements of the actual scene,but there are many interference factors in the image,and it requires manually annotated.The synthetic image is subject to external interference factors and can be automatically annotated,but it lacks authenticity and the data distribution is not as rich as the real image.Therefore,the gaze estimator trained using these two human eye images is not ideal when tested in actual scenarios.Aiming at the problems of poor generalization of the gaze estimator trained from synthetic images,high acquisition and annotation cost of real images as well as the interference factors,we introduce the idea of style transfer into gaze estimation and propose a human eye image synthesis method with style transfer.We focus on purifying real images,by combining with the advantages of synthetic images whose distribution is more uniform and easier to learn.Specifically,the annotation information of the real image as the content,and the distribution of the synthetic image as the style,we aim to obtain a gaze estimator with better robustness in real scenes,which is trained by the images generated by our proposed method.The overall structure of our proposed method consists of three parts,semantic segmentation network,feature extraction network and loss network.Previous style transfer methods only considered the global features,which may change the shape of pupil and iris and then make the error of gaze estimation larger.Therefore,we take the first step to train a semantic segmentation network to obtain pupil and iris regions,and then learn global and local features of the human eye through a feature extraction network.We calculate style loss and content loss through the novel loss function to reduce the distribution gap between the synthetic image and real image,and then obtain the output image.The output image retains the important annotation information of the real image,such as pupil and iris,reduces interference from illumination and other factors,and learns the distribution of the synthetic image as much as possible.Finally,in order to more fully and comprehensively prove the effectiveness of our proposed method,experiments were performed using a mixed research(qualitative and quantitative)method on style transfer and appearance-based gaze estimation tasks respectively.In the style transfer task,compared with the baseline methods,we can not only better retain the color and texture information of the reference style image,but also the speed can meet the real-time requirements.To verify our proposed method can improve the accuracy of gaze estimation,we evaluate our generated images and original images several baseline methods in the gaze estimation task,and the experimental results show that we can achieve the state-of-the-art gaze estimation results on several public datasets.