Design And Implementation Of A Data Management Platform Driven By Metadata

In the back-end of mobile Internet App,programmers usually adopts a variety of database systems to store business data of different dimensions and themes,with scattered data storage and different structures.App content operation and editing staffs need a back-end data management platform to effectively manage these multi-source heterogeneous data of different dimensions.The traditional development method mostly adopts the customized development method.According to the specific coding of business requirements,the implementation of data addition,deletion,modification and search is not effectively reused.This thesis proposes a metadata-driven architecture design,and only metadata configuration is stored in the production database of the platform.When data management requests occur,the platform shall process business requests according to preconfigured metadata information.The platform conducts data interaction with heterogeneous data sources,and complete the process of data extraction,transformation and loading into the web-side to satisfy data query request.The platform can also conduct data change process from web-side to source-side.The research work of this thesis includes the following aspects:1.Describe and manage metadata of heterogeneous data sources and business functions,build data model,select relational database to store metadata information,and customize configuration rules of metadata.The metadata configuration determines the set of data sources and fields that business functions associate with and how the platform interact with data sources.2.Design and implement the data engine.The engine provides universal interfaces to handle user requests of adding,deleting,changing and searching data.The data engine is capable of handling not only single datasource queries,but also multi datasource relational queries.It decomposes relational queries of multi datasource according to the configuration of metadata into multiple subqueries,merges and transforms the results after the completion of sub-queries by different adapter programs,thus integrating heterogeneous data sources in middleware mode.3.ElasticSearch and MySQL data source adapter are designed and implemented,providing the platform with the ability to conduct two-way data interaction with these two heterogeneous data sources.The adapter establishes the connection with the corresponding data source according to the metadata configuration,assemfies the interactive statement,completes the data interaction and converts the query result into the common structure,shielding the data heterogeneity.Different adapters implement a unified abstract interface for upper level,making the platform extensible.4.Access ElasticSearch index and MySQL relational table respectively by adding metadata configuration to test the platform's metadata-driven data access ability to different data sources and the platform's heterogeneous data integration ability.After actual deployment,the platform has been put into operation in iQIYI company,providing more than 30 data management functions such as content pool management and boutique pool management.These functions are quickly implemented by adding metadata configuration.The platform accesses the data in the data source and displays the data in Web pages.Users can also conduct data modification,deletion and other operations in Web pages.This method of adding metadata configuration to replace coding does not need to write code or recompile and deploy,which is of high practical value for the development of data management Web systems and heterogeneous data integration systems.