Research On Advertising Click-Through Rate Prediction Algorithm Based On Knowledge Distillation

With the rapid development of mobile Internet,recommender systems have been widely used in industry.Considering the increasing Internet advertising market,the task of Click-Through Rate Prediction(CTR)has become one of the core tasks of recommender systems.The main task of the CTR algorithm lies in classifying a user’s response of an item to be positive or negative.With the wide application of deep learning in recent years,knowledge distillation technology has made remarkable achievements in the fields of computer vision and natural language processing.On the one hand,knowledge distillation can enhance the performance of the model,on the other hand,it also contributes to the miniaturization and lightweight of the model.Therefore,this paper studies the advertising click-through rate prediction algorithm based on knowledge distillation.The main work content is divided into the following aspects:(1)Currently,most CTR models follow the framework of Embedding Layer & Feature Interaction.As the cornerstone of the CTR models,the modeling effect of the embedding layer has a direct impact on the subsequent modules.And the number of parameters in CTR models is heavily concentrated in the embedding layer,which has an important impact on the predictive performance of the model.Considering the increasing number of features used by modern recommender systems,we propose an enhanced feature importance learning for click-through rate prediction based on knowledge distillation(EFI)to enhance the feature importance learning ability of the model.EFI only affects the embedding layer of CTR models,and is compatible with most state-of-the-art CTR models.Through experiments on two authoritative public datasets,we verify the validity of EFI on other state-of-the-art CTR models.(2)Most of the existing CTR models are constituted of two parts: a deep neural network(DNN)and a wide model.A lot of efforts have been devoted to designing new wide models and improving the existing ones,but rare attention has been paid to the enhancement of the predictive power of the DNN component.In general,much fewer parameters are contained in the DNN component than in the wide model,and thus the predictive capability of DNN is inevitably affected by the wide model when they are trained together.In order to resolve this issue,we propose an adaptive deep neural network for click-through rate based on knowledge distillation.We assemble several student DNNs as a stronger teacher model and co-train it with a wide model,so that the feature expression ability of a single DNN can be preserved through this way.In addition,one of the student DNNs is trained independently which can be applied in resources-limited devices.Our method needs not pre-train any high-capability teacher models,which endows our method with higher efficiency compared with existing methods.