A Research Of Face Tamper Detection Based On Semantic Segmentation

Nowadays,the image generation technology has developed rapidly,and fake face generation technology has emerged at the same time.There are some representative technologies,such as Deepfake,Face2Face^[1],NeuralTextures^[2],and so on.People can use these models to exchange the faces in images or videos,or even generate a non-existent face.These technologies mainly include three steps:face localization,face conversion and face stitching.At present,face localization technology has developed very maturely.The general face localization algorithm mainly finds feature points on the face,such as nose,mouth,chin,and eyebrows.By extracting these features,you can know the organ distribution of the face and complete the positioning of the face.For example,we can use the dlib and OpenCV packages for face extraction,or we can use deep neural networks such as CNN models to achieve the localization.Face conversion mainly includes generation models such as adversarial generative model?GAN?and variational autoencoder?VAE?.We can train the encoder to exchange the faces between two people.For example,VAE compresses a face image into a vector in an unsupervised manner,and then restores this vector to former face image.If we encode the face image of person A,and then use the decoder of person B to decode it,the facial expression in B's face image can be changed to A's.The GAN extracts the feature points of the face,and the generator uses these features to generate the target face image.The discriminator is trained so that the discriminator cannot distinguish the differences between the generated face and the real face,and completes the face generation.While deep learning technologies such as Deepfake are becoming more and more popular,they also bring huge hidden risk.We have to admit that face-swapping is one of the consequences of abusing deep learning.Before that,we could post photos without any concern on social media,such as friends circles,Weibo,Facebook,etc.,and we could also believe the videos and images in online news.And since these face-swapping technologies became open source,we can find that some people use the face-swapping technology to process and publish some fake videos,and even worse,use the collected face images to generate pornographic videos of other people's faces.Therefore,defending against the Deepfake technology is equally important.At present,many articles have focused on the field of defending against Deepfake technology.The mainstream detection method is to train a binary classification model on the real image and the fake image dataset to determine whether the test image is generated.The disadvantage of this is that the result of experiment depends on the dataset,and the classifier cannot distinguish the dataset which is generated by different generation models.Convolutional neural network?CNN?usually ignores the traditional features in the image,such as the noise,so the result performance is not very good.In view of the above problems,this paper mainly improved the mainstream method to conduct face tamper detection experiment.It was noted that there was noise in the contour and other edge information after face change.We could improve the results by adding traditional features to the deep neural network.Based on the above ideas,the main experimental work in this paper includes the following:1.A CNN classification model was trained based on Xception^[3],which was used to distinguish the fake face from the real face,and test the global image and the face ROI region.2.Test the Deepfake Detection Challenge data set using the model in Face-Forensics++^[4].3.Based on the traditional feature method,JPEG compression rate feature,BAG feature and noise feature were used to conduct multi-group comparison and analyze the influence of different features on the experimental results.According to the above experiments,the following conclusions can be drawn:1.The face tamper detection model has a great dependence on the data set now.If the data set is replaced,the method performs poorly,results are closed to random prediction,which do not has reference value.2.After fine-tuning and retraining the model,it is obviously improved to a certain extent compared with the previous experiment.The experimental results show that the median prediction method is the best.The specific prediction accuracy is about 0.900,the recall rate is about 0.85,and the overall AUC is 0.748.3.On the basis of 2,traditional characteristics were introduced for prediction,among which the combination of BAG characteristics and ELA characteristics showed the best performance.The two-way prediction method was used to improve the experimental accuracy by 3%and the AUC by 5%on the basis of the previous experiment,indicating that the traditional characteristics are effective for such problems to a certain extent.