Context-aware Anomaly Detection Algorithm For Big Data Applications

The rapid development of the Internet information industry has promoted the continuous rise of big data mining technology.Big data mining is a complex process for uncovering potential information and knowledge from massive and heterogeneous datasets.In big data area,there may be data objects whose behavior patterns are inconsistent with other data.These data objects are defined as anomalies or outliers.Thus,identifying and mining anomalies or outliers in the dataset is called anomaly detection.As the volume of abnormal data in massive datasets can not be ignored,how to detect anomaly has gradually become one of the research hotspots in the field of big data mining.Anomaly detection is a generic method concept that aims to detect undesired behavior.It has various applications in different fields,such as fault detection,intrusion and fraud detection,and data preprocessing.Anomaly detection in a specific field is largely dependent to the characteristics of data and the purpose of detection.The selection of anomaly detection methods varies from different data and purposes.Therefore,how to choose anomaly detection methods for a specific domain is an urgent problem to be solved.This paper proposes two context-aware anomaly detection methods for Web services recommendation and traffic big data,separately.The main research contents are:First,Context-Aware Web Services Recommendation of True Abnormal Data Elimination(CASR-TADE)algorithm is proposed for the personalized recommendation system of Web services with a large volume of historic Web services records.It obtains Web service historic records in similar contexts through similarity degree mining among users and services.Specifically,the concept of true or false anomaly is introduced,and the identifying and processing module for true or false anomaly is constructed to provide personalized Web services recommendation.Second,the context-aware unlicensed taxis identification algorithm is proposed in the field of traffic big data with a massive Vehicle License Plate Recognition(VLPR)data.In the framework model,the daily behavior and sustainable behavioral features are extracted using the location and temporal contexts for each vehicle from the VLPR traffic trajectory data,based on 336 million VLPR records of 6.2 million vehicles.Later,an unlicensed taxis identification model is constructed based on both the supervised anomaly detection algorithm and the extracted features of vehicles.Third,we conduct intensive experiments and evaluations on the proposed two anomaly detection algorithms mentioned above,and the results prove the effectiveness of algorithms.This paper proposes anomaly detection methods for the characteristics of big data applications and data features in two field.On the one hand,it offers a new idea for anomaly detection applications in the specific fields.On the other hand,it provides big data decision support for anomaly detection systems in recommender system,intelligent transportation system,etc.