The Design And Implementation Of Recommendation Model Quality Analysis System Based On Fuzz Testing

In the era of information explosion,recommendation systems are crucial for Internet companies.For example,You Tube utilizes it to provide videos from hundreds of mil-lions of resources for different users based on their personal interests.Therefore,guar-anteeing the quality of recommendation models becomes a very valuable research prob-lem.Existing recommendation model testing methods use quantitative evaluation met-rics to analyze users' historical behavioral data.On the one hand,historical data induces recommendation results to be overly inclined to existing choices,which reduces rec-ommendation diversity and causes historical data bias problem.On the other hand,the evaluation metrics are not rich enough,resulting in incomplete quality assessment of the recommendation model.These problems make the existing testing methods unable to meet the needs of developers for recommendation model quality evaluation.There-fore,there is a need to develop a high-quality recommendation model quality analysis system to discover the deep quality defects of the recommendation model and improve the testing adequacy.Therefore,this thesis designs and implements a recommendation model quality analysis system based on fuzz testing,which augments test cases by fuzz testing tech-nology and combines quantitative metrics with interpretable analysis to complete rec-ommendation quality evaluation.Considering diverse forms of features in recommen-dation scenarios and the relationship between features,the system utilizes feature struc-ture to create a corpus to record feature correlations,and to specify feature types and mutation methods.The corpus constructed by this system transforms and stores the original testing dataset based on feature types,and completes data scale statistics and feature distribution analysis.Based on the feature correlations recorded in the corpus,the system provides rich built-in mutation methods to adjust the feature values of test cases in different forms,such as text,numeric,dictionary,and list,to generate new test cases and add them to the corpus to augment testing dataset which will be used to evalu-ate the recommendation system.In order to rich evaluation metrics,the system provides characteristic evaluation metrics such as heat value,which can effectively measure rec-ommendation diversity,and analyzes the quality of feature engineering based on t-SNE dimensionality reduction.The system consists of a data flow module,a fuzz testing task module,a task management module and a report generation module.The data flow module supports test file uploading and data downloading? the fuzz testing task module is responsible for feature structure creation,data augmentation and metrics cal-culation? the task management module provides users with task information? the report generation module is responsible for automated test report generation.The system has been piloted within Ant Group and has been used to evaluate the quality of models under eight domains.This system has uncovered 8 models with qual-ity defects through quantitative metrics and 12 models with feature engineering defects through interpretability analysis metric,which shows high practicality of the system.