Research And Implementation Of Facial Avatar Control Svstem Based On Hand-Over-Face

With the rise of short video applications,3D facial expression avatar has received widespread attention,but the industry's implementation does not consider the scene where the hand covers the face(Hand-Over-Face),and the special effect map covers the hand,which affects the expression of information.This paper is optimized based on this scene.In the specific implementation,it is found that the Hand-Over-Face scene brings two major challenges:occlusion affects the reconstruction of the 3D face reconstruction model;the similarity of the skin color of the hand and face affects the hand segmentation model.For this reason,this paper proposes the following innovations:First,a 3D face reconstruction model based on triple network and differential replay is proposed.The model realizes a robust 3D face reconstruction model without 3D data supervision,and uses 2D image features for model training.Similar studies did not pay attention to the non-linear correlation between face shape coefficients and facial expression coefficients,which resulted in stiff predictions of facial expressions and a slower convergence process.To this end,the model introduces a triple network and uses positive and negative samples to decouple the model's learning process of face shape parameters and expression parameters,thereby improving the prediction effect of the model.At the same time,to accelerate the model convergence speed,the shape vector storage module is designed to select training samples according to the Euclidean distance of the face shape vector between positive and negative.Not only that,but also increase the Hand-Over-Face data during the training process to improve the model's robustness to the scene.Second,propose a Hand-Over-Face segmentation model that joint hand key point and Hand-Over-Face segment network with multi-branch and multi-stage training.Due to the similarity of the skin color characteristics of the hand and face in the Hand-Over-Face scene,the current mainstream segmentation algorithm can't handle this scene,may cause problems such as broken fingers and unclear edges.To solve these problems,the model introduces the key point information of the hand for multi-task and multi-stage training to optimize the problem of finger incompleteness;at the same time,the different weight is introduced to focus on the hand edge information to optimize finger adhesion,edge burrs.Third,design and implement a facial avatar control system which concentrate on Hand-Over-Face occlusion.This research finally achieves a PC client that can run on multiple platforms.The client provides a simple and intuitive operation interface that allows users to experience facial avatar control during Hand-Over-Face occlusion,supports real-time display of special effects by uploading pictures or camera capture.