Design And Implementation Of Content Processing And Monitoring System Based On Machine Learning

Nowadays,with the development and popularization of mobile products,people's demand for content and information products is increasing with each passing day.In contrast,the enormous increase of content has brought unprecedented pressure to editors and auditors.In order to meet the needs of the big data era,the project team decided to develop a content processing and monitoring system based on machine learning,in order to complete the automatic filtering and distribution of content,so as to provide efficient and intelligent data processing process for various content forms of the company.This paper describes the content processing and monitoring system designed by the project team according to the background of the current information age and the machine learning theory with practical application potential.Firstly,the background of the project and the development status at home and abroad are introduced,and the functional and non-functional requirements of the whole system are analyzed.After that,the system architecture,functional modules and database are designed according to the requirement analysis.Then the specific module is designed and implemented in detail.Finally,the system is tested and verified to ensure that it meets the online requirements.The author independently establishes the machine model used in the module,and adjusts and optimizes the model through optimization algorithm and loss function.After that,the machine model and the core function modules are tested and verified to ensure that the system meets the online standards.In addition,the author participated in the development of three modules of the system core,namely,content processing(content filtering and modeling),content auditing and content monitoring.In content filtering,characters are matched by N-Gram language model,and parameters are optimized by decision tree and Center-Loss function.In content modeling,data are preliminarily classified and supervised by SVM(Support Vector Machine)algorithm and GRU(Gated Recurrent Unit,Gate Recurrent Unit)model,according to different content classes.TEXT-CNN model and FAST-RCNN model are used to partition the model.In content monitoring,in order to obtain real-time data results,Spark real-time framework and MemCache cache technology are used to achieve the performance requirements of monitoring.At present,the system has been put on line,and the average accuracy rate of image and text is more than 95%,and the average accuracy rate of video is more than 80%.Content processing speed is controlled within 120 seconds on average,and real-time monitoring query speed is controlled within 2 seconds.