Design And Implementation Of Data Monitoring System Based On Multiple Data Sources

With the vigorous development of the advertising industry,the complexity of the basic platform service system required for an advertisement in the commercial monetization process is also increasing.Under such a complex distributed system architecture,system dependence leads to inconsistent data between systems.Phenomenon also occurs from time to time,which is a loss of interests for users and a loss of assets for the platform.Therefore,how to ensure the accuracy of business data and solve the problem of system data consistency has become top priority.This system is a project established during the company's internship to solve the problem of data inconsistency between systems.The system combines the company's actual business background and needs to provide unified data verification capabilities for multiple services and multiple data sources,including data source management,unified configuration of monitoring tasks,asynchronous scheduling of tasks,task execution,abnormal data alarms and alarm cause prediction,abnormal data The processing of abnormal data,the analysis and statistics module of abnormal data,through monitoring business data,expose data abnormalities in time,reduce capital losses as little as possible,and improve the company's internal efficiency and accuracy of business data.I participated in the project and completed the preliminary research and project establishment of the system,system requirements analysis,design,development,testing and online process.The main system modules are monitoring task configuration,task asynchronous scheduling,task executing,abnormal data alarm and process,abnormal cause prediction five modules.In the specific implementation of the system,the Web framework uses the Python Flask framework,the RPC(Remote Procedure Call)framework uses the company's self-developed Euler framework,the message queue uses Rocket MQ,the data cache uses Redis((Remote Dictionary Server),and the alarm reason prediction algorithm uses the Naive Bayes classification algorithm.At present,the system has completed the launch of the basic functional requirements,ran stably online,and entered the operation stage.It has instructed multiple users to configure the rules,and the success rate of rule execution has reached99%+.In the process of use,online requirements are continuously collected,and the requirements are continuously iterated.