Research Of Data Quality Assessment And Disk Intelligent Diagnosis Method Without Faulty Sample For AIOps

With the development of Artificial Intelligence,cloud computing and micro-service technology,the complexity of IT system structure and the diversity of components increasingly lead to many challenges to the traditional operation and maintenance technology.Artificial Intelligence for IT Operations(AIOps)has become the trend of The Times.AIOps adopts artificial intelligence technology to automatically learn rules from the available operation monitoring data(performance indicators,alarm information,system logs,etc.)to replace the man-made rules,thus improving the predictive ability and stability of the system,reducing the cost of IT system operation,and improving the product competitiveness of enterprises.However,big data is the basis of AIOps,and the lack of data required for AIOps model construction and data quality all affect the accuracy and generalization of the model.From the perspective of data collection and evaluation,online data is often used to construct AIOps models,but the collection of online data will lead to lag in model construction.Meanwhile,the traditional AIOps process lacks data quality evaluation process,and the data quality of online data is difficult to guarantee,which will affect the actual effect of the model.From the perspective of models,most of the existing models default that the training data is complete enough,while in the real scene,the data is often not ideal,there are many problems such as missing data and difference in sample distribution.Aiming at the above challenges,this paper aims at the data problems existing in AIOps and studies the data quality assessment and intelligent disk diagnosis method under zero fault samples.The main results are as follows:1.From the perspective of data acquisition and evaluation,this paper proposes an AIOps agility scheme and data quality evaluation method.AIOps agility scheme is to advance the AIOps model construction stage to the test stage,and use the monitoring data generated in the test stage to replace the data collected online to train the AIOps model,so as to realize the early development and early use of AIOps.The data quality method is based on Maximum Mean Discrepancy(MMD)to evaluate the trend,stage,detectability and diagnosability of the training data for health assessment and fault diagnosis scenarios,respectively,in order to estimate the applicability of the data to the model.Based on the test environment provided by Huawei,this paper sets test cases and constructs experimental data set.The experimental results on the data set prove the feasibility of AIOps agility and the validity of the data quality assessment model.2.From the perspective of model analysis,this paper takes disk fault diagnosis,a common operation and maintenance scenario,as an example,analyzes the real data problems in this scenario,and proposes an intelligent disk diagnosis method under zero fault samples.In the method,the intelligent disk diagnosis model is constructed by using the normal and fault samples of old disks and the normal samples of new disks.Specifically,the Deep Generated Transfer Learning Network(DGTL-Net)is proposed,which combines the generation network with the migration network and solves the problems that the fault samples of new types of disks are not easy to obtain and the attribute distribution of different types of disks is different.Finally,the intelligent disk diagnostic model is evaluated on the Backblaze disk open source dataset,which prove the effectiveness and reliability of the proposed method.