Research Of Monitor System For Business Layer Based On Storm In Big Data Environment

As the rapid development of Computer and Internet technology,Internet of Things(IoTs)and mobile networks,people's daily life has been fully occupied and significantly changed by data and data services.According to the statistics provided by Tencent and Ali,the daily amount of logged data has reached up to 50 TB from Mobile QQ and Taobao alone.Apparently,how to effectively analyse the collected data and achieve a real-time monitoring and management of corresponding services has become a forefront research topic.Most of the existing systems mainly focus on hardware layer monitoring,leaving the service layer untouched.However,with a fast growth of service requests,the latter now is also challenged by the trend of Big Data.In order to solve the above problem,a general monitoring and management solution based on the framework of Storm and Kafka has been proposed in this work.This solution provides a novel method which only requires the field property configuration of service and target rules from users to achieve the ability of real-time surveillance,detection and alarm.The novelties of this work are as follows:(1)Aiming at the problem of complicated monitor configuration with no visualization of the existing system,a uniform web service is proposed.To design and implement each module pluggable.To make the basic information of the monitoring,input field,processing logic,topology preview and other operations can be visualized operation and generates the code of the monitoring logic automatically according to the configuration information of the page,which greatly reduces the complexity of the monitoring and management configuration.(2)Aiming at the problem of low accuracy of the existing monitoring system,a low false-alarm algorithm is proposed.The algorithm proposed in this work numerically analyses the history data and calculates fluctuation rate of sampling points using hypothesis testing and clustering methods.With an effective modeling and prediction by cubic interpolations and BP neutral networks,the false-alarm of the detection system has been dramatically reduced from ten percent to one percent.(3)Aiming at the problem of low transmission efficiency and high processing delay of the existing system in big data environment,a system design consist of user interface layer,data access/cache layer,Real time computing layer,offline computing layer and data persistence layer is proposed.A hybrid and secondary development of Storm and Kafka.The ultra-low delay and high concurrency property from Storm has been combined with the serialized compressive coding from ProtocolBuffer to satisfy the real-time and multi-dimensional requirement in Big Data scenarios.Finally,the function test and the performance test are done on the system which has been designed and implemented in this paper.The analysis of the test results figures and statistical graphs of several performance indexes shows that the system perfectly satisfies the expected designing target,which has been applied to monitor environment in parts of business in Tencent.