The Design And Implementation Of One Big Data Governance System

With the maturity of distributed storage,distributed computing and other related technologies,the big data industry has also gained rapid development,both governments and enterprises have seen the great value of big data,and began to mining value from the data accumulated by the business system built in the past.But the traditional data management mode and processing methods can not meet the government and the enterprise's business needs,which they must achieve diversified management,standardized governance,high-quality convergence for these massive heterogeneous data,and use these high-quality data realize accurate governance and rapid decision-making.In order to make these massive data release its powerful power,we must ensure the accuracy,reliability and timeliness of these data,and to achieve this goal,if there is no suitable,efficient tools,only depend on labor it is difficult to complete.Therefore,it is necessary to provide users with a set of effective tools to assist them in data governance.The purpose of this paper is to build a set of such data governance system,through which on the one hand,the user can achieve multi-dimensional organization,unified management,rapid retrieval of multi-source heterogeneous data,and at the same time can establish a set of unified data standards and data governance process,can continuously detect and process data to ensure accuracy of them,and finally provide high-value soil for subsequent business decisions.This paper first expounds the market scale of the big data governance system and the current situation of system construction of China,puts forward the challenges faced by the big data governance system,as well as the necessity and significance of building the big data governance system.Second,this article analyzes the user requirements of the big data governance systems,including functional requirements and non-functional requirements,as well as the related application scenarios,and explain it in detail through the flow chart.Based on requirements analysis,this paper will carry out the architecture design of this system.The whole system will be divided into data asset management,data standard,data quality,data repair and resource scheduling five subsystems,and all of them will according to the same design principles and standards when design the technology architecture and functional architecture.The technology architecture includes the technical selection,the project structure,the interaction protocol and the scalability,maintainability,stability of the system etc.The functional architecture includes the module structure,the functional composition of each module and the function flowchart,etc.,which are implement according to the user's needs.Next,this article discusses the detailed design of big data governance systems.This part will describe of the data asset management,data standard,data quality,data repair and resource scheduling in detail,including the user interaction design,interactive usage instructions of the five subsystems,and the technology architecture,class diagram,sequence diagram,data model diagram about the key points of each subsystem,and at last this part will also make some necessary description for the main system interaction interface.The five part of this paper will introduce how the five subsystems of data asset management,data standard,data quality,data repair and resource scheduling coordinate with each other to achieve the system goals of big data governance system.Finally,this paper gives a brief introduction to the test situation of big data governance system,and puts forward some suggestions for how to improvement of the system.In summary,we propose and design a new type of big data governance system based on the needs of the government and enterprises in big data management and governance,and with the help of micro-service technology architecture we have realized a big data governance system which has advanced architecture,high scalability and integral function.