Research On Log Anomaly Detection Methods For Large Scale Software Systems

With the rapid development of information technology,the scale and complexity of modern computer systems are gradually increasing.System logs record key user operations and status information during system operation,which can help technical personnel solve problems and eliminate financial and security risks in enterprises.Therefore,anomaly detection of system logs has become a research topic with practical significance.Traditional anomaly log detection mainly relies on manual detection or semi-automatic detection by writing expressions,but the labor cost is high and the detection accuracy is also subjectively affected by human factors.The rapid development of machine learning and deep learning enables technicians to build detection models,identify the characteristics of abnormal logs,and achieve fully automated log anomaly detection.However,current log detection methods still have problems such as low accuracy in log parsing、incomplete extraction of log features、uneven proportion of positive and negative samples、untimely abnormal feedback and disordered logs.This article conducts corresponding research and analysis on the above issues.This paper proposes an anomaly log detection algorithm based on the Bi GRU-CNN model to address issues such as low accuracy in log parsing and incomplete extraction of log features.The algorithm first uses the FT-Tree method to extract log templates,improving the accuracy of log parsing.Next,based on the spatial characteristics of log parameters and the temporal characteristics of log templates,they are input into CNN and Bi GRU models to extract features.Finally,the extracted features are concatenated and combined to fully utilize the complete log information for anomaly detection.In response to issues such as imbalanced positive and negative sample ratios and untimely abnormal feedback,this article also proposes an anomaly log detection algorithm based on a generative adversarial network model.This algorithm utilizes the game between the generator and discriminator to simulate the distribution characteristics of logs by using Bi LSTM network and multi head attention mechanism in the generator,thereby alleviating the impact of imbalance between normal and abnormal instances.Then,the generator is used as the detection model,and when any log arrives,the generator can make judgments to achieve fine-grained anomaly detection at the log level.Finally,based on the above two algorithms,this article designs and implements an anomaly log detection system in accordance with software engineering requirements.The system adopts a B/S architecture to meet the functional requirements of users in model training,anomaly detection,and historical information retrieval,as well as the performance requirements during human-machine interaction.Through relevant testing verification,the system has good detection performance under different data samples.