Deep Learning Based Methods For Facial Attractiveness Assessment And Analysis

The attractiveness of human face has been studied in different fields of social science and natural science throughout the ages.Psychological research shows that the representation and evaluation of facial attractiveness have a certain data-driven and objective property,which provides the possibility for machines to realize intelligent evaluation of facial attractiveness.In the past few years,with the development of machine learning,especially deep learning methods,facial attractiveness has allured the attention of many researchers,and an automatic system of facial attractiveness assessment can be used in many in scenarios such as auxiliary plastic surgery,personalized recommendation in social network,and face beautification systems.The topic of this paper is to investigate the feature learning and prediction model of facial attractiveness assessment from different views,and then build an intelligent facial beauty prediction system.Specifically,considering the weakness of feature learning and classifier optimization in independent stages in traditional machine learning methods,the single-paradigm assessment of current facial attractiveness benchmark databases,and the existing facial attractiveness assessment methods ignoring relative aesthetic mechanism and face attribute information,and the formulation of personalized aesthetic perception of humans,this paper is expanded from deep learning based facial beauty prediction,where the content is mainly about the construction of facial attractiveness database,facial beauty prediction algorithm and personalized facial beauty assessment algorithm,with the purpose of making machine achieve the same aesthetic perception as human.In summarize,this paper mainly includes the following innovations and contributions:(1)Considering the lack of large-scale and diverse facial attractiveness database currently,we construct a multi-paradigm facial attractiveness database(SCUT-FBP5500 database).A larger-size database can help to introduce deep learning techniques into facial attractiveness analysis,thereby promoting the further development of related research about facial beauty The follow-up research work of this paper is mainly built based on SCUT-FBP5500 database.In this database,we collect 5,500 facial images with different attributes,and invite 60 volunteers to mark images with various labels such as rating scores and facial landmark coordinates.Due to diverse attributes and various labelling information,multi-paradigm facial beauty prediction can be conducted on this dataset,such as the prediction of facial attractiveness or attractiveness distribution.Further,we prove that the rating scores are reliable by investigating the statistics such as rating distribution,standard deviation,and correlation of these data,which is also confirmed to be learnable and predictable through plenty of benchmark experiments on SCUT-FBP5500.(2)Considering the relative aesthetic mechanism in human's aesthetic perception,we propose a relative aesthetic guided facial beauty prediction algorithm.Currently,existing literatures usually define facial beauty prediction as a standalone task of classification or regression,while ignoring ranking information in the process of human aesthetic rating,which discounts the performance of prediction tasks.Motivated by the observations,we redefine facial beauty prediction as a ranking guided facial beauty regression task,and construct a ranking guided regression Convolutional Neural Network,namely R3CNN,which can implement ranking and regression tasks simultaneously.Specifically,we develop a series of training schemes for R3CNN,including hard pairs sampling strategy,ensemble loss function and cascade fine-tuning method.Finally,the experimental results show that R3CNN can improve the performance of facial beauty prediction on the basis of common CNN model,and the ablation studies also confirm the necessity and effectiveness of hard pairs sampling strategy,ensemble loss function and cascade fine-tuning,respectively.(3)Considering the effect of attributes on facial beauty,we propose to learn attribute-aware feature representation for facial beauty prediction.It is a kind of dynamic feature learning method,which can effectively integrate attribute information into feature representation,thereby achieving a better facial beauty prediction.Further,our method is significant and can applied to other facial attribute recognition tasks.In this paper,there are two means to learn attribute-aware feature representation,namely,attribute-aware convolution and attribute-aware batch normalization.Further,we also combine them to construct an attribute-aware feature representation framework.The experimental results show that attribute-aware convolution and attribute-aware batch normalization can improve the performance of facial beauty prediction,respectively.And the combination of these two means,namely,the attribute-aware feature representation framework,not only can improve the performance on the basis of independent attribute-aware convolution module,but also can make the network optimization more stable.(4)Aiming at the practical applications of personalized face recommendation,we define and model personalized facial beauty prediction from the perspective of meta-learning for the first time,whose goal is to train a model(as well as called personalized facial beauty prediction model)that can adapt and predict the aesthetic preferences of different users.It is significant to some applications such as social networking recommendation.In this paper,personalized facial beauty prediction is specifically defined as a classification or regression task based on meta-learning.When defined as a classification task,we build a classification model based on prototype network to make classification decisions on different user's data;when defined as a regression task,we build a regression model based on fine-tuning to achieve personalized facial beauty regression.The experiments are conducted on a new-built personalized facial beauty dataset,namely SCUT-PFBP,and the experimental results confirm that personalized facial beauty prediction task is feasible and learnable.