Research And Implementation Of Data Exchange And Sharing Platform Based On Microservices

Driven by the rapid development of intelligent cities,it is necessary to comply with the development trend of big data and accelerate the informatization system construction.The city department usually adopts traditional service-oriented(SOA)architecture design to complete massive data exchange tasks.Due to the different data storage methods,increasing requirements,the increasing scale of data sets,data exchange pressure has risen exponentially,posing new challenges to the scalability and stable performance of the data service architecture.In order to support the intelligent city data exchange system to provide efficient and stable services and avoid the problem of high coupling between system modules,this article intends to design a data exchange and sharing platform based on microservices.By setting a unified data access standard and using reader plugin and writer plugin to collect source data and target data to realize the automatic collection,exchange,and storage.By shielding the underlying data source format and data source type to complete the data exchange between heterogeneous data sources and by configuring the data exchange task scheduling strategy through task scheduling management to reduce the consumption of memory and CPU.Data governance by providing blood relationship analysis,metadata view presentation to support metadata management,mine the data true value,and realize data is asset.The main results are as follows:(1)Establish a set of data exchange standards to realize the entire management process of data collection,exchange,cleaning and processing.Break the limitations of geography,data storage,and data format,and provide strong decision support for the construction of smart cities.For heterogeneous data source types,provide corresponding database read and write plugins,and data exchange tasks are completed through a unified data exchange binary channel.(2)Supports distributed tasks,separates the scheduling module from the execution module,communicates through RPC,provides multiple routing strategies and failure handling mechanisms,and the scheduler selects the executor at the specified trigger time to start the full and incremental data exchange tasks.(3)Supporting real-time data push and sharing,through monitoring the database Binlog file,analyzing the data stream and writing it to Kafka,pushing data to downstream consumers subscribing to Topic,realizing data interoperability between departments at all levels.Provide metadata information management,support blood relationship analysis,metadata view presentation,and modify database tables,fields,and partitions.(4)Test the functions and performance of the data exchange and sharing platform designed in this article to prove the feasibility of the system through experiments.