Research On A Multi-Source Heterogeneous Data Collection System Based On Microservices Framework

With the rapid development of the new generation of information technology,technologies such as big data,the Internet of Things,and artificial intelligence are widely applied in various industries.The total amount of information resources generated by different government and enterprise departments is increasing at an astonishing rate.However,the sources of data between different departments are not single,mostly from different systems,platforms,and devices,resulting in heterogeneity between data.Against the backdrop of strengthening the development and sharing of public data as determined in the national "14th Five-Year Plan",there are problems in the practical use of multi-source heterogeneous data,such as difficulty in achieving information sharing,data isolation between different departments,and untimely data collection.Especially in terms of data collection,in the era of "big data",the integration and development of data resources is an inevitable trend.However,the heterogeneity of multiple data sources has led to a complex data collection process,which requires targeted development for different data sources and formats,affecting the efficiency and popularity of data collection.How to separate the logical description of data collection from the code development process,making the data collection process more universal,is an urgent problem to be solved in current data collection.Aiming at the above problems,this paper describes the data collection logic through XML Extensible Markup Language,designs XML description files for the collection process of data resources with different structures according to the characteristics of multi-source heterogeneous data,and designs the corresponding collection algorithm.The above collection process is realized and integrated based on the Microservices framework,and the design and implementation of the data collection system based on the Microservices framework is completed.This paper analyzes the characteristics of multi-source heterogeneous data in the field of data collection,and analyzes the structural and unstructured characteristics of data collection from the data layer and presentation layer in combination with the general Architectural pattern of the information system,The collection process is versatile.By virtue of the advantages of Microservices framework in the field of multitasking,the data collection process is integrated into the Microservices framework.At the same time,the security strategy and visualization module related to the collection process are designed,and the data collection system based on Microservices is realized,including data collection startup service,structured data collection,semi-structured data collection,encryption processing,multi task scheduling,Data and information visualization and other functions.In terms of system design and implementation,the system is developed based on the Spring Cloud Microservices framework,mainly using the Spring Boot mode to design the software layered architecture,and the system is divided into data presentation layer,system control layer,system business layer and data persistence layer.Based on JAVA language,develop six functions logically,design database tables for data collection tasks,and describe and record data collection tasks and processes.The resource allocation of Microservices is realized through load balancing,which is more conducive to the overall operating efficiency of the system.Data security is encrypted based on Homomorphic encryption algorithm;Set the timed tasks of the multitask module through the quartz multitask scheduler.All the basic functions of the data collection system based on the Microservices framework are realized.