Research On Anomaly Detection And Classification Technology Of Real-time Data Of Internet Of Things Based On Deep Learning

While the Internet of Things(IoT)cooperates with big data and cloud computing to deeply restructure various industries,it also poses more severe challenges to system security,the realization of intelligent anomaly detection is the key step.The combination of traditional statistics and Machine Learning(ML)algorithm provides a solution in the early stage,but they have some inherent constraints,such as difficulty in algorithm and parameter selection,high development cost,over-reliance on good tags,limited data processing capacity and so on.By representing the world as a nested hierarchical concept system,deep learning algorithm shows prior functionality and flexibility in large-scale data processing and multi-scene tasks compared with traditional ML algorithms.However,most of the existing anomaly detection methods based on deep learning are of batch processing,which has be done in an offline fashion and is not suitable for real-time detection in the IoT environment.In addition,their anomaly features mining mostly focused on spatial anomalies,ignoring the detection of temporal anomalies,thus cannot deal with the phenomenon of concept drift.In order to solve the problems above,this paper proposes a real-time anomaly detection algorithm for IoT data based on spatio-temporal fusion,which can simultaneously model spatial and temporal patterns in real-time data streams,mine spatial outliers and temporal outliers,and get detection results through a comprehensive decision strategy.Furthermore,an anomaly classification algorithm based on Convolution Neural Network(CNN)is proposed.The algorithm uses the Gramian Angular Fields(GAFs)to transform one-dimensional sequences into images,and then perform the image classification task by CNN,applying the advanced achievements in Computer Vision to the anomaly classification diagnosis of massive real-time data streams.The main work and innovations of this paper are as follows:1.In view of the lack of temporal anomaly detection in existing algorithms,a real-time anomaly detection algorithm for IoT data based on spatio-temporal fusion is proposed in this paper.The anomalous sample points are excavated from the spatial pattern and the temporal pattern respectively,then a comprehensive judgment is carried out to maximally coordinate the two patterns' judgements on the same sample point.Specifically,the algorithm uses Hierarchical Temporal Memory(HTM)to store the contextual information of the data,and extracts its temporal pattern features to realize anomaly mining in the temporal pattern;a Variational Auto-Encoder(VAE)is used to store the general probability distribution of the data to realize anomaly mining in the spatial mode;the dynamical adjustment of the window size is further adopted for final determination in the comprehensive judgment strategy.2.To address the problem that VAE training and detection are seriously affected by the abnormal points(anomalous and missing points),two improvements are made to VAE in this paper.1)In the training stage,the formula of Evidence Lower Bound(ELBO)is improved through the introduction of threshold judgment and scaling factor,the contribution of labeled abnormal points in the training set to the optimization target is directly excluded;2)In the detection stage,the missing points are reconstructed by Markov Chain Monte Carlo(MCMC),which greatly eliminates the latent variable mapping bias caused by missing points in the test set.The above two improvement methods reduce the influence of abnormal,and improve the overall anti-interference ability of the model.3.To address the problem of insufficient data and excessive dependence in one-dimensional time variable analysis,an anomaly classification algorithm based on CNN is proposed in this paper.The algorithm first uses GAFs to transform one-dimensional time series into two-dimensional images,then smoothly reduces the dimensionality by Picewise Aggregation Approximation(PAA)to get GADF images and GASF images with appropriate resolution,and finally inputs them to CNN for anomaly classification.4.This article tests the proposed anomaly detection method on 12 benchmark datasets of all walks of life.Experiments show that the improvements of VAE in this paper significantly enhance the anomaly detection performance of VAE,of which the correction of ELBO contributes more.Compared with the four algorithms of HTM only,VAE only,LSTM-AE and LSTM-VAE,our algorithm got the highest in both NAB score and Best F-score.For the anomaly classification algorithm in this paper,the famous KDD CUP 1999 dataset is selected for classification and diagnosis experiments.Experiments show a promising result that the macro-P,macro-R and macro-F1 reached 0.838,0.798 and 0.763 respectively after 10 epochs.