Detecting Anomaly In Large-scale Network Using Mobile Crowdsourcing

In recent years,with the improvement of the functions of smart phones,the mobile Internet has begun to rise on a large scale.Accordingly,various OTT(Over The Top)services have emerged in an endless stream.The OTT service refers to Internet companies directly providing various data transmission services to Internet users beyond operators such as text chat and video calls.Typical examples are WeChat,Skype,etc.However,since OTT provides services globally,many Internet Service Providers(ISPs)are required to cooperate to provide services.As a result,the underlying network is highly complex and difficult to maintain,thus causing many network anomalies and leading to the degradation quality of service(QoS).Traditional network maintenance and management methods can no longer meet the needs of OTT services.Therefore,it is urgent to design a low-cost and efficient network anomaly detection mechanism for large-scale networks.In this paper,based on the challenges raised by OTT services,a high-latency network anomaly detection method based on decision tree modeling is designed with the help of the crowdsourced network measurement data.The structure of the whole method is divided into two parts: rule mining and rule evaluation.Rule mining is responsible for mining all potential anomaly rules from the crowdsourced data.Rule evaluation is responsible for evaluating all the anomaly rules.To deal with the problem of the messy crowdsourced data and the inconsistent feature forms,we first perform feature engineering,and then use instance clustering to merge the samples of the same feature.The preprocessed data is modeled by decision tree and used as an analysis model: potential high-latency network anomalies are extracted from the decision tree's topology and tree nodes' information.The main contributions of this paper are as follows:(1)We propose a data mining method to extract the cause of network anomaly based on decision trees' structure.While existing works mainly use decision trees for predictions,decision trees' interpretability can help ue further exploiting the hidden information of network anomaly.Results show that we can efficiently extract all potential cause of network anomaly from crowdsourcing dataset.(2)We propose“confidence”,a criteria to evaluate the anomaly severity objectively according to the degree of performance degradation and the scale of impact.We choose three factors to quantify the anomaly severity: standard deviation,weight sum and impurity decrease.Experiments in Section 5 also proves the accuracy of the criteria.(3)We propose a robust forest-based data mining algorithm by integrating our single tree-based approach with random forest.We utilize bootstrapping sampling to generate random sample subspace of the original dataset as the input for each subtree.Then we compact all their mining outputs to get final results.By applying the idea of random forest,the variance of the model can be reduced and the generalizaion can be enhanced.We conduct extensive experiments based on a crowdsourced network dataset with five million samples.The dataset involves round trip time(RTT)information from 6226 kinds of applications and more than 5000 users.Experiments show that our approaches can effectively detect all anomalous network conditions and the approach utilizing random forests achieves about 25 percent higher generalization performance than single tree-based approach.