Research And Implementation Of Anomaly Detection System At The Edge Of Industrial Quality Inspection Based On Log Analysis

With the rise of industrial edge intelligence,the edge end carries more complex tasks.In order to ensure the normal operation of the edge end,real-time abnormal monitoring of its operation is required.The log generated by the system records detailed operating information and has become the main data source for anomaly detection and data monitoring.Log-based anomaly detection has become an important research topic of practical significance in academia and industry.This article takes the zipper quality inspection system as the background,aiming at the problems that the real-time and accuracy of the current anomaly detection system applied to the edge of the industry cannot meet the needs and the manual inspection workload is large,designed and implemented a log abnormality monitoring system that combines Flink realtime calculation and K-Means algorithm,which can accurately detect abnormalities at the edge in real time,reduce production losses,and improve production efficiency.The main work results include:(1)It solves the problems of extracting event information from mass production log.The AFT-tree algorithm was proposed to improve the shortcomings of FT-tree algorithm,such as long initial matching time,single divider and large randomness of pruning threshold.Use text editing distance to pre-group logs,add special symbols as delimiters,and use Apriori algorithm instead of fixed pruning threshold for pruning.Experimental results show that the average accuracy of log template extraction based on AFT-tree can reach 95.6%,which is1.5% higher than that of the FT-tree algorithm,and has better performance than that of the FT-tree algorithm when processing large amounts of data.(2)It solves the problems of insufficient labels and low detection accuracy in current logbased anomaly detection methods.The log sequence is comprehensively weighted,and the unsupervised algorithm K-Means is selected for anomaly detection and classification.The traditional k-means algorithm's clustering effect is unstable and greatly affected by outliers.Firstly,the outliers are detected based on density,and then the number of clusters is determined using the Elbow method.Finally,the initialization center point is selected based on the maximum distance,so as to optimize the algorithm.The experiment shows that the improved K-Means clustering algorithm in this paper has an average detection rate of 88%for abnormal events,which is 14% higher than the traditional K-Means algorithm,and it is also better in comparison with other unsupervised anomaly detection algorithms.For the problem of heavy workload in anomaly location,the method of using representative sequences to construct an experience database is compared with the traditional manual method of searching and locating anomalies by keyword matching,which greatly reduces the number of logs that need to be checked and reduces the workload of operation and maintenance personnel.(3)Designed and implemented an anomaly detection system based on log analysis for industrial quality inspection.First,deploy the Docker image encapsulating the Flume log collection framework to each edge of the quality inspection,and the Flume will transmit the logs generated by the edge to the Kafka cluster in real time;Second,Flink will read the messages in the Kafka cluster in real time for anomaly detection,and synchronize the detection results to Elastic Search;Then use the Elast Alert framework to monitor the Elastic Search cluster in real time,and send the alarm information to relevant personnel using Ding Talk or email when an exception occurs;Finally,the Kibana log analysis platform is integrated to realize the visualization of abnormal data.The test results show that the system fulfills the functional requirements of log collection,anomaly detection,log retrieval,alarm notification,and data visualization.The throughput can reach up to 30 w log data per second,and the log from producing to finish testing and locates the total delay of only 9.11 s,which meets the high throughput and low latency requirements of the system,and has good scalability to cope with business growth and changes.