Research And Application Of Inhibiting The Effects Of Concept Drift Based On Machine Learning

With the development in the field of internet technology,big data technology has been applied more and more in various fields,including the analysis of communication data,the help of product data for product development,the application of electronic commerce and financial supervision.Compared with the traditional data mining,today's data mining has the characteristics of massiveness,infiniteness and dynamism.The traditional mining algorithms can not completely adapt to the current data mining.Nowadays,with the development of big data technology and the large number of applications,mining information from data streams has become a research focus.The research content of this paper is about the coping strategies when the concept of credit drifts in credit evaluation and the impact of coping strategies on the complexity of the whole system.First of all,this paper uses an improved ensemble method to improve the adaptation of the model to dynamic data stream with incremental algorithms to reduce the effects of concept drift.Then,using the sliding window strategy,the data segments of different probability distributions are distinguished and the influence of concept drift is suppressed in the other dimensions.Finally,we study the problem of preventing over-fitting when suppressing concept drifts.Specific research work is as follows:First,this paper uses an ensemble method based on mixture model,combined with the decision tree and knn algorithm to reduce the impact of concept drift in a heterogeneous integrated approach.Secondly,this paper uses the sliding window technology to divide the data stream into the same size of the window unit for processing,to improve the accuracy of the sliding window,making the effect of the concept of drift more effective and stable.Thirdly,over-fitting problems can have a more serious impact on the study of suppressing concept drifts.In this paper,by contrasting and imitating the decision tree over-fitting problem handling scheme,to reduce the effect of over-fitting made a certain study.