Distributed Data Management System For Federated Learning

With the iterative transformation from traditional Internet to mobile Internet,the improvement of mobile broadband quality,the development of Internet of Things,cloud computing and other technologies,data fills the whole world and becomes a new natural resource.Individuals or enterprises pay more and more attention to data,which eventually makes the data barrier problem appear.In order to break down data barriers and fully explore the potential value of distributed data storage,many researchers devote themselves to data sharing solutions,among which federated learning is a technical means to realize data sharing in the field of artificial intelligence.Through research and analysis,it is found that data sharing solutions such as federated learning focus on make the data does not go out of the security boundary and the application code flows.However,data providers lack effective management ability in the whole process of data sharing,and there are security threats caused by data being out of control.Therefore,this thesis studies the distributed data management system for federated learning.First of all,this thesis researches and designs distributed data management in the general scenario of data sharing.Then it provides functional support for federated learning,a special scenario of data sharing.Finally,the system is tested and analyzed.The main research work of this thesis is as follows:(1)Architecture design of distributed data management system.Understand the data island dilemma in the context of big data,and study data sharing solutions.Based on the existing research results of federated learning and data management,this thesis analyzes the technical design,advantages and disadvantages of effective and controllable management of shared data in the case of distributed data storage.On the theoretical basis of the above research,demand analysis and system architecture design of distributed data management system are carried out.(2)Design and implementation of horizontal federated learning subsystem based on distributed data management System.From the user management,node management,data management of the three modules as the breakthrough point,according to the idea of transferring the power of managing data to the data provider,a distributed data management system is designed and implemented.(3)Design and implementation of horizontal federated learning subsystem based on distributed data management system.The horizontal federated learning subsystem is designed based on based on distributed data management system.Through federated data source management and federated task training,this thesis describes the design and implementation of each functional module of the horizontal federated learning subsystem,and the functional adaptation of the distributed data management system to the horizontal federated learning subsystem.(4)Verification and testing of distributed data management systems for federated learning.The designed system is tested for business functionality,and the performance of the federated learning part of the system is tested through a case of horizontal federated learning to verify the availability of the system.