Research And Implementation Of Real-Time Data Integration System Which Support To Big Data

Today’s society, with the deepending of information technology development and the developing, in order to adapt the increasingly frequent business activities in each other and improve their competitiveness, companies invest a lot of resources for research and development which adapt to the various departments of the business system. However, these business systems is both different functions and independent.Their data storage and access methods are not the same. As the business growing,internal data showing characteristics that the amount of data becomes more and more larger, data sources and data storage format become more diverse and location of the the data storage become more discrete. For an enterprise, how to effectively use these data and obtain a favorable decision quickly for enterprise business information in the mass data.It’s all directly related to the survival of enterprises. Therefore, how data is logically or physically concentrated in the organic unity together have increasingly recognized by the companies, so enterprises and departments can provide more comprehensive data sharing and rapid changes in enterprise business information.Real-time data integration technology can fully solve the above problems.It describes the current research topics data integration technology, at the background of data integration. It make a brief introduction to the related technology.For the real-time needs of today’s large enterprise data under the data environment, the paper research to achieve a practical and reliable real-time data to support large data integration system on the basis of the analysis of existing data integration technology.System mainly make research on two aspects which is the stability of real-time data integration system and the security of a large number of real-time data integration process.Firstly, based on the research of traditional data integration system architecture and the analysis in real-time data integration features and applications on demand, this paper presents a general real-time data integration architecture. Then it analyze the process of the real-time data integration. Real-time data integration can be divided into three parts that is real-time data extraction, loading and real-time conversion. To solve the problem of real-time data extraction, we analyze the difficults of real-time data extraction in a heterogeneous multi-source environment and present real-time data extraction method that is based on message middleware. To different data sources, it can quickly andefficiently achieve real-time incremental data extraction task. For real-time data loading,it use real-time data loading methods which is based on real-time data cache. In the case which does not affect the performance of the data warehouse, it implements a large number of integrated real-time data loading. Then, this paper proposes a concurrent task scheduling strategy which is based on the methods of data pretreatment and the tasks in real-time requirements of the rules engine to improve data efficiency and safeguard the stability of conversion tasks, the real-time of integrated data.Finally, we tested the real-time data integration system through simulation experiments, and verify the availability and stability of the system by analysis of the experimental results.