Three Way Decision-based Large Data Classification And The Application In Video Anomaly Detection

Data mining technology helps people find much useful information from large noisy data. Because the data contains much noise and redundance, the information is uncertain. Rough set theory is the effective tool to solve the problem of uncertainty in data mining. The research of attribute reduction algorithm is one of the core problems in rough set theory. The traditional rough set attribute reduction algorithm can not deal with mass data. Therefore it is worthy to apply traditional attribute reduction algorithm to big data. Comparing with Hadoop, S park is much more superior platform and is adopted by many research institutions. This paper will use the platform to deal with huge amount of data.Three way decision theory is an expansion of the two way decision theory. It can help reduce decision error rates due to the insufficient information effectively by introducing boundary regions. Meanwhile it meets decision-making process in real life. However, in most of applications, the boundary region is decided by users. When the boundary regions are too large, users will take long time to deal with it. Thus it is a problem to reduce boundary regions as much as possible, but at the same time ensuring the classification accuracy has become a problem. Same as the traditional attribute reduction algorithm, when facing large-scale data, three way decision algorithm can not deal with them efficiently. Thus it is the important research to apply three way decision algorithm to process large-scale data.As cost sensitive classification algorithm, three way decision algorithm can be ap-plied to cost sensitive classification problem. In video anomaly detection applications, the cost of classifying normal behaviors into abnormal sets is higher than the cost of classifying abnormal behaviors into normal sets.Meanwhile, the number of normal be-haviors and abnormal behaviors is severely unbalanced. Thus video anomaly detection is not only a cost sensitive classification problem, but also an unbalanced classification problem. It is of great value to apply the three way decision algorithm to the video anomaly detection.This thesis respectively focuses on the above-mentioned issues and makes im-provements.The main work is as follows:We parallelize the traditional attribute reduction algorithm RSAR and propose a parallelized algorithm PRS AR. We implement the algorithm on Spark. The experimental results show that the parallelized algorithm can effectively process huge amounts of data.We analyze the reason why three way decision algorithm generates huge bound-ary regions and propose an ensemble classifier Tritraining-TWD based on the idea of Tritraining. The experimental results show that the classification perfor-mance has been improved comparing with three way decision algorithm. For dealing the big data, we parallelize the three way decision algorithm and the ensemble classifier Tritraining-TWD algorithm and propose two parallelized algorithms:PTWD and P-Tritraining-TWD. We implement these two algo-rithms on Spark. The experimental results show that parallelized algorithm can process huge amounts of data faster.We adopt parallel LDA(Latent Dirichlet Allocation) to process video data, trans-ferring high dimensional motion features into low dimensional semantic features. Later, we use three way decision algorithm to classify normal behaviors and ab-normal behaviors. Based on the application, we improve the cost partial ordering relation in three way decision algorithm. Experimental results show that com-paring to the general classification algorithms, three way decision algorithm has more advantages. Meanwhile, with the guidance of improved cost partial or-dering relation, the performance of the three way decision algorithm has been improved.