Design And Implementation Of Real-time Acquisition And Heterogeneous Storage Of Human Society Data

With the rapid development of Internet and other information technologies,human resources and social security has to change the way of management,simplify the procedure of business and improve the effciency of work.Because human resources and social security business covers many fields,such as social insurance,public employment,labor relations,personnel and so on,the business involves widely a large quantity of work,which makes the business procedure trivial and the efficiency low.Therefore,in order to adapt to the development of the era,and to provide people with efficient and convenient business services,Human Resources and Social Security Bureau proposed the integration of human resources and social security business.And,in the process of integration,there are demands for data sharing,business collaboration,data synchronization between production area exchange area and service area,as well as exchanges between services.That is why a system is required to realize the internal and inter-region efficient incremental conversion and synchronous replication of massive data between production area,exchange area and public service area.Based on the above background,taking full account of the various needs of human resources and social security integration in real-time data acquisition from source database,data conversion,as well as data storage in the target,this paper is going to introduce the development of data heterogeneous synchronization,part of HS big data integration platform.Data heterogeneous synchronization includes many functional modules,for example,heterogeneous replication configuration,conversion rules management,database and service resource management,conversion process configuration and so on.On one hand,it allows users to choose the source database and the database tables according to business requirements of incremental data heterogeneous synchronization.On the other hand,the system provides users with the conversion rules development interfaces,specifications and examples;so that users can customize the conversion rule according to their own conversion needs,and upload the conversion rule through the system in order to be called in the procedure of heterogeneous synchronization.Moreover,the system can convert the incremental record of one table of the source database by calling different conversion logics,and apply the converted contents to one or more targets at the same time.Besides,the system provides monitor modules for the data heterogeneous synchronization process so as to ensure that the process can be well controlled,which realizes the stability,availability and the data consistency of the heterogeneous synchronization system.This paper describes the demands,summary design,detailed design,as well as implementation and tests of the system in full detail.The system uses B/S architecture format,adopt Karaf system which based on OSGI standard as the basic platform,and apply the way of plug-in development to develop.It is developed with EclipselDE.Furthermore,the real-time data collection of the source database adopts Oracle Streams technology;data conversion logic,related database and related services are resources to be used in the heterogeneous synchronization.Also,the Apache Camel technology is adopted to route messages from database queues to conversion logic and then to apply into the target.In addition,the development of system interface uses Vaadin technology.And it can run on both Windows and Linux operating system.Last but not least,the system has been fully tested and realized not only the functional requirements of heterogeneous synchronization of data,but also the non-functional requirements such as easy-to-use,reliability and maintainability of software.