Transfer Learning Application Research And Implementation On Smart Grid Data Analysis Based On Spark

With the continuous development of the information society,data mining methods have gradually played an irreplaceable role in the research of various industries,and the smart grid system has also entered the era of intelligence.Traditional data mining algorithms such as classification,clustering and association rules have played an important role in the aspects of power equipment fault diagnosis,power consumption prediction,energy saving and emission reduction,and have made many achievements.However,with the continuous expansion of practical applications,traditional data mining methods may face many problems,for example,traditional data mining methods usually require data to be sufficient enough to get the accurate rules,but when the source data information is sparse,traditional data mining algorithms may be difficult to achieve the desired results.As an emerging machine learning method,transfer learning has achieved certain results in recent years.Transfer learning aims to study the coupling between different distributed data.According to the actual needs,the data source which is sufficient is selected as the source domain,and the data to be analyzed is selected as the target domain.Transfer learning aims to use the rules trained in the source domain and apply them in the target domain to obtain higher starting points and more precise data analysis rules.Therefore,applying the transfer learning method to sparse power data has a very high practical significance for practical production applications.According to the difference of data distribution between the source domain and the target domain,transfer learning can be divided as instance-based transfer learning and feature-based transfer learning.The following work has been accomplished in this paper.First,this paper proposed an improved instance-based transfer learning algorithm based on trAdaboost,then this paper use the improved model to forecast the load of the tranformer.Second,this paper proposed a feature-based transfer learning model.According to the existing domain adaptation methods,this paper introduced the balance factor to realize the dynamic balance between domain distributions.Then,this paper introduced a deep transfer network based on CNN to accomplish the learning work.The proposed feature-based transfer learning model is used to complete the fault prediction of transformers and switches.Last,based on the relevant characteristics of the Spark framework,this paper implements the parallelization of the above two algorithm models,and designs the implemented algorithms into components and integrate them in Big data Analysis Platform(BDAP).