Design And Implementation Of Data Map Based On Meta Data Warehouse

With more and more internet industry using data-driven as the tool to achieve decision-making, data is playing an increasingly important role in the internet. And now the order of data has been growing explosive, relying solely on data warehouse modeling theory has been difficult to maintain the data growing in a good way, to avoid the brutal growth of data, enterprise need for an unified management of data urgently. Meta data is called the data of describing data, can be used to describe the relationship between the contents and elements of data, also can describe the data more clearly, one way to manage the large-scale data can be building the system of meta data, it can also undoubtedly promote using data in more ineffective way.Firstly, this paper analyzes the means of universal data management at the present stage, a general management of meta data, through the analysis of the status, summing up the way that using meta data warehouse to promote data management scheme. At the same time, based on the company's current situation and analysis of the actual needs, choosing the data map which based on meta data warehouse to solve the problem of data management, it also can solve the problem to determine the data source. Secondly, based on the above conclusions and combined with the demand analysis, this paper contains the system design, and detailed design of each functional module, including the system architecture, module flow chart, etc. Finally, developing the data, backend, separation respectively to implement the requirements, using ETL based on hadoop and hive to construct meta data warehouse which can ensure data production, using open source python flask architecture as the system development framework, MySQL as the background database, using Flux and React in the foreground enable realization of user interaction, the ultimate realization of the system function.In this paper, the realization of the system finally realize the meta data warehouse on the unified management of the whole company, the data map provides the localization of data source, preview of data, life cycle of data, analysis of kinship to help the analysts find data by themselves. At the same time, developing the system in the agile way, through the process of developing exchanges and communication with product managers and users continue to carry out all aspects of the need for iteration, constantly iterating on the system so that the system performance and ease of use is more in line with user requirements.