Design And Implementation Of Content Risk Control Algorithm Management Platform

Internet content risk control has become a key technology for Internet companies and governments.Due to Internet data has the characteristics of fast generation speed and high update frequency,relying solely on manual review has high costs and high error rates.The current review method is mainly based on the combination of machine review and human review based on rules and algorithm models.This requires fast iteration of algorithms and models,and the cycle of data processing needs to be as short as possible,but traditional algorithm management platforms cannot meet these requirements.The demand for a risk control algorithm management platform arises at the historic moment.Traditional algorithm management platforms have shortcomings such as algorithm and model coupling,low data processing performance,and weak model iteration ability.In the risk control scenario,the model needs to be iterated quickly to adapt to the rapid changes and confrontations of illegal content,and there are high requirements for the speed of data processing.This article combines the requirements of the algorithm management platform in the risk control scenario,and takes the risk control system of a large domestic Internet company as the background,designs and implements the risk control algorithm management platform,and solves the following problems:Through componentized design and atomized management of models,the algorithm and model are decoupled,so that the algorithm Researchers and specific model developers can perform their duties to improve the overall iterative efficiency of risk control;data caching,parallelization,data preprocessing and other technologies have greatly improved the data processing performance of the platform.It is currently in the peak period.It can run stably;through technology such as model file splitting,mirroring and data separation,and automated AB testing,the iterative efficiency and deployment speed of the model are greatly improved,so that various risk control models can be iterated quickly and stably.At present,this platform has been put into use in large Internet companies.In terms of model iteration efficiency and ease of use,the average iteration speed of models and algorithms has increased by 1.5 times,which has been well received by the algorithm team.In terms of performance,in a scenario where the average QPS is 5000 and the peak QPS is 20000,the operation is stable and efficient.