| With the application of Internet of Things technology in all walks of life,the number of sensor devices and intelligent terminal devices connected to the Internet of Things surged,and the amount of data increased explosively.Gradually,the Internet of Things system presents the characteristics of various device communication protocols,various types,and very complex interactions.On the one hand,the contradiction between limited physical resources and rapidly growing data has become an urgent problem to be solved in the Internet of Things system.On the other hand,the huge data generated by the large number of IOT devices for anomaly detection in order to monitor the operation of the device has become particularly important.Due to the increase in data volume and complex data structure,traditional data compression algorithms and traditional anomaly detection methods face severe challenges.In this context,this thesis proposes an Internet of Things heterogeneous device access architecture(IOT Heterogeneous Device Access Systems,IHDAS),and makes an in-depth study of data compression and anomaly detection under this architecture.The specific content is as follows:(1)A hybrid data compression model based on LSTM-β-TCVAE-LZW is proposed to solve the problem of high device storage pressure and high cost of edgecloud transmission caused by the massive data of the Internet of Things.The model is based on long short-term memory neural network(LSTM),improved variational autoencoder(β-TCVAE)and Lempel-Ziv-Welch algorithm(LZW).LSTM is used to process multi-dimensional sequence data and extract feature long-term dependencies;β-TCVAE is used to generate low-dimensional latent representation of multidimensional data to complete data compression;LZW further lossless compresses lowdimensional latent variables generated by β-TCVAE to improve Compression ratio.At the same time,in order to balance the generation and decoupling capabilities of the LSTM-β-TCVAE model,the hyperparameter β is introduced to punish the TotalCorrelation of the variational lower bound of the β-TCVAE model,and the hyperparameter γ is introduced to correct Index-Code and Dimension-Wise items.Finally,the model can effectively realize the compression and reconstruction of Io T data,and minimize the cost of data storage and transmission while ensuring data quality.(2)A three-layer Blending anomaly detection model based on the selection of the minimum feature set is proposed to solve the problem that traditional anomaly detection methods cannot comprehensively and accurately detect various types of anomalies in Io T device data,and cannot reduce the features of samples with the large number of features.Firstly,in order to eliminate redundant features and reduce the number of original features,a minimum feature set selection algorithm is proposed in combination with the dynamic correlation formula.This method evaluates the degree of importance between each feature and label,and uses multiple machine learning models to evaluate the accuracy of feature subsets in a summed and average way to select the minimum feature set.Finally,in order to solve the problems of low accuracy of traditional single machine learning models and long training time and easy overfitting of traditional fusion models,a three-layer Blending-based fusion model detection method is proposed,which can train and deploy anomaly detection model based on the minimum feature set.Finally,the experiment proves that the reconstruction error of LSTM-β-TCVAE model in data encoding-decoding is 33% lower than that of LSMT-β-VAE,and the optimal parameter combination of LSTM-β-TCVAE is obtained through experiments.After combining LZW,the optimal compression factor of the LSTM-β-TCVAE-LZW hybrid data compression model is increased by 3-40 times compared with the traditional algorithm;in terms of space saving,the optimal result is improved by 25%-36%compared with the traditional algorithm.From the experimental results,the hybrid data compression model not only has a high compression ratio,but also preserves the important information of the original data,which can make the error between the decompressed data and the original data very small,very suitable for the compression of massive data of Io T.In the anomaly detection of Io T data,the number of features selected by the three-layer Blending anomaly detection model based on the minimum feature set selection is reduced by 4 times compared with the original feature number;in terms of testing accuracy,compared with the traditional machine learning model,it increased by 12.14% on average,and compared with the traditional Stacking and Blending models,it increased by 1.32% and 2.57% respectively.From the experimental results,the model can not only effectively reduce the number of features,but also has higher accuracy than common machine learning models and fusion models,and can better complete the anomaly detection of IOT devices. |
