Design And Implementation Of Distributed Governance Component Oriented To Government Multi-source And Big Data

With the development of information technology,government big data information has become an important reference for government departments to make social governance decisions.Due to the incomplete data governance in the early government system,there may be some wrong data in the system;the data model and business process between the historical versions of the government system may change,and there may be invalid data in the system;the secret level of each government system data may be different in actual application;social governance increasingly relies on the coordination of multiple government departments,and there are still various government departments Information barrier;the data model of each government system is inconsistent,which leads to the data with a certain relationship,and the efficiency is low when the data is integrated.Therefore,data governance is necessary in government system.However,the government system data has the characteristics of multi-source,which is a growing mass of data.The existing data governance tools,such as traditional ETL tools,are difficult to apply in the government system big data governance.The data of government affairs system is the data with the highest credibility,which is of great value.At the same time,some data are also sensitive.To give full play to the value of government affairs data and improve the utilization rate of data resources,it is necessary to manage government affairs data.In order to realize the data governance function of government big data,based on the research of big data related technology,combined with Apache beam technology and ETL tool kettle component design idea,this thesis designs and implements the distributed governance component for government multi-source big data.The data governance component designed and implemented in this thesis is universal and extensible.The input and output components support the multi-source characteristics of government data,and the data processing components have rich data processing and conversion capabilities.The task of government big data governance based on the arrangement of distributed data governance components is simple,easy to use,flexible and convenient.Combined with the computing ability of government big data computing cluster,it can realize the distributed governance of government big data,realize the potential value of government big data,realize the effective use of government data resources,and have a positive effect on economic and social benefits.After experimental demonstration,the big data governance task based on the configuration of distributed data governance components designed and implemented in this thesis can be executed in the big data computing cluster environment,realize the distributed data governance function,and meet the governance needs of multi-source big data for government affairs.