The Design And Implementation Of Log Analysis System For Internet Of Things Platform

With the development of IoT(the Internet of Things),IoT technology has been applied to all areas of society.As the infrastructure of IoT technology,IoT platform can realize the interaction of equipment,data,and information,and complete unified management and monitoring.Efficient operation,security maintenance and performance monitoring of IoT platforms are very important.Logs are records of a large number of events generated during the operation of various devices,system platforms,and applications.We can quickly find out the problem based on the log.Therefore,log analysis is critical to the operation and maintenance of the IoT platform.This thesis designs and implements a log analysis system for the IoT platform by studying the log solution of the IoT platform.The system adopts a distributed deployment method and a streaming data analysis framework,which can safely store and analyze a large amount of log data in real time and perform visual display.The main content of this thesis is as follows:1.Designed the log analysis system of the IoT platform based on ELK.Aiming at the characteristics of large log quantity,complex structure,and high real-time requirements for log analysis,the corresponding functional structure and system architecture,as well as the workflow and specific configuration of each module of the system,are designed.In order to improve the scalability of the system and reduce the complexity of system deployment,the container-based Kubernetes technology is used to deploy and manage the various modules of the system.2.Aiming at the clustering problem of massive logs,this thesis combines SOM(Self-organizing Maps)and FCM(Fuzzy C-means)into a SOM-FCM double-layer clustering algorithm,and realizes Effective clustering and dimensionality reduction analysis.Secondly,for the timing log of IoT platform,a deep learning-based LSTM-Seq2Seq algorithm was studied to realize the abnormal detection of log data and achieve a good detection effect.3.Finally,the function test and performance test of the log analysis system of IoT platform designed in this thesis have been carried out.The results show that the system has a good real-time processing capability of massive logs.At the same time,the SOM-FCM two-layer clustering model has achieved better clustering effects and the LSTM-Seq2Seq algorithm has a good accuracy in log anomaly detection.