Face Manipulation Detection And Application Based On Multi-source Fusion And Mixed Attention

Detecting manipulation based on neural network to solve media forensics problems has a huge advantage,but we can see much less frequent use of the detection model to detect the manipulation.This paper takes face fake detection as the target scene,proposes a detection model,verifies its effect in the scene,and analyses the performance.The main work is as follows:(1)Aiming at the low performance of a single recognition network in recognizing fake images,a face manipulation detection model fused with multi-source visual cues is proposed based on the YOLOv4 model.Firstly,a variety of filters are used to extract information in the frequency domain and noise domain,and we fuse the extracted information.Secondly,MobileNeXt is used as the backbone to improve the ability of feature extraction.Finally,the detection ability of multi-scale feature fusion network improved by DO-CONV is adopted.Our network is tested on the Face-Forensics++ dataset and the Celeb-DFv2 dataset,the ability of detection has significantly improved.In the Celeb-DFV2 dataset,the AUC is increased by 1.3% and the accuracy is increased by 4.3%.(2)Considering the problem of the detection model in the special features and channel features,a subspace attention mechanism has been proposed.First of all,the network divides the channels of input features into different groups,and then introduces a spatial attention mechanism into each group to extract spatial attention features,and uses channel attention to extract channel attention features.Then we fuse the channel attention feature and spatial attention feature of each group to get the non-linear relationship information between cross-channels.Finally,the features generated by each group are fused and outputted.The test results under different datasets show that the subspace mixed attention mechanism can improve the detection accuracy of face manipulation when about 0.4% of the number of parameters is introduced into the network.In the Celeb-DFv2 dataset,the AUC of frame-level detection task increased by 1.2% and the accuracy increased by1.8%,while the AUC of video-level detection task increased by 0.3% and the accuracy increased by1.4%.(3)Considering the limitation of network model size and calculation in mobile terminals,face manipulation detection network structure suitable for mobile terminal devices is proposed by modifying the model structure.Firstly,a multi-source visual fusion module is used to extract multi-source visual features,and a lightweight mixed attention network structure is designed to further extract features.Secondly,according to the dataset label analysis,the input of feature fusion module in the original network was reduced from three size features to two size features.The convolution layer used for feature fusion was replaced with a single deep over-parameterized convolution,and the convolution layer used for the compression channel was replaced with a single1×1 convolution layer.Finally,the network was trained,and then the detection network is deployed to the mobile terminal for testing with ncnn framework.After testing,it can be found that the accuracy of detection is reduced,but the frame-level face fake detection can be completed on the mobile devices.In the Celeb-DFv2 dataset,the AUC for frame-level tasks decreased by only 0.2%.But the number of model parameters is reduced to 1/3 of the original network.