| With the popularity of the Internet,especially the rapid development of the mobile In-ternet,Web services have affected all aspects of social lives.Simultaneously,the failures of Web services have caused poor user experience and enormous economic loss.Improv-ing the efficiency of Web service operation is the key to ensuring service quality and user experience.The operation is an essential part of the Web service life cycle,and its purpose is to make the online services run safely and stably.The traditional Web service operation mainly depends on the domain knowledge and rich experience of the operators.However,with the rapid increase in the scale,type,and complexity of Web services,the traditional operation methods that rely on manual analysis have been unrealistic to handle a large-scale service operation’s problems.In recent years,the in-depth researches of machine learning have shown some inspirations to solve the problems of operation.Web services own a large amount of operation data,and machine learning algorithms are good at auto-matically learning and mining rules in large amounts of data.Therefore,to improve the efficiency of failure detection and root cause localization of Web service operation,this study will use machine learning algorithms to detect the anomalous traces among service modules and localize the root cause of anomalous service modules.The main research contents and contributions are as follows:(1)Trace anomaly detection.Facing the problem that it is challenging to detect anomalous traces among Web service modules,this study proposes a trace anomaly detec-tion algorithm based on a deep learning algorithm: Trace Anomaly.The algorithm learns the normal patterns of traces through deep Bayesian networks and achieves accurate trace anomaly detection based on the learned patterns,further improving failure detection ef-ficiency.Trace Anomaly has been deployed on 18 online Web services of We Bank com-pany,both online and offline experiments have shown that Trace Anomaly’s recall and precision are all above 0.97.This work is open-source and published on Git Hub.(2)Root-cause machines localization.Facing the problem that it is challenging to localize root-cause machines in a large number of anomalous machines of Web ser-vice modules,this research proposes an algorithm to automatically localize root-cause machines: Flux Rank.The algorithm can quickly analyze and extract the abnormal pat-terns of the massive KPIs(Key Performance Indicators)of machines through Kernel Den-sity Estimation and clustering algorithm.Then it recommends the root-cause machines’ rankings based on the extracted abnormal patterns.After evaluating Flux Rank using 70 real failure cases from Baidu company,the results show that the root-cause machines are ranked top 1 for 55 cases and ranked top 3 for 66 cases.Flux Rank has been deployed on seven online Web services.After more than three months of deployment,Flux Rank has diagnosed 59 online failures and correctly localized the root-cause machines as the top 1for 55 cases.(3)Root-cause KPIs localization.Facing the problem that it is challenging to lo-calize root-cause KPIs in a large number of anomalous indicators of a single machine,this research proposes an algorithm to automatically localize root-cause KPIs: Flux Infer.The algorithm can automatically build a weighted undirected dependency graph among anomalous KPIs and recommend the root-cause KPIs’ rankings based on the dependency graph.After evaluating Flux Infer on an open-source system,the results show that the ac-curacy of Top3 and accuracy of Top5 in the rankings reached 0.90 and 0.95,respectively. |
