| More than 80% of the world’s data has been generated in the last few years,and the constant generation of this data has made "big data" the most popular IT term at the moment.The massive data produced by the insurance industry every day are characterized by structure evacuation,information deficiency,weak relationship,etc.These characteristics are very consistent with the application of big data technology in the insurance industry.With the rapid development of big data technology,these information can be gradually pieced together and simplified.Based on the research and application of data processing technology in the insurance industry at home and abroad,this paper makes use of the data processing and storage capacity of big data technology,and then determines the main research content of this paper in combination with the data integration needs and technical application needs of the insurance industry.In the system design stage,this paper analyzes the traditional business characteristics of the insurance industry,and combined with the needs of enterprises for data analysis,the data warehouse used by the system is logically layered,and the application of ETL technology in the data warehouse is determined.In the demand analysis stage,through the gradual disassembly of the enterprise’s needs,according to the different stages of data flow to determine access,storage,cleaning,integration,synchronization and other as the main functions of the system.In the overall design stage,the storage technology of the data warehouse adopts HBase,and the relational configuration model is designed for the cleaning and integration functions.Finally,each major function is corresponding to each level of the data warehouse one by one.The data relation configuration model adopted by this system combines customer information,business requirements and system procedures,so as to achieve the purpose of guiding data integration operation through the relation model.This article uses a JSON configuration file to map this process and describes in detail the components of the configuration.Based on the design of data relational model,the storage function of data integration guided by relational model is realized through the hierarchical design of data warehouse,which provides a theoretical basis for the storage of each stage in the process of data integration.In the system implementation stage,this paper carries out the overall implementation of the system after combining the previous analysis and design.Firstly,through the Kafka consumer model,Spark’s streaming technology is used to access data sources from the producer side to the consumer side.When the data source is connected,the system first stores the original value of the data to the BDL layer of the data warehouse.Then,the data is cleaned and stored to the IDL layer one by one according to the cleaning index in the configuration model.Finally,data association query is conducted according to the integration relationship in the configuration model and the results are stored to the ADL layer and DML layer.At this time after the integration of the data also has a clean field,clear structure,data integrity and a series of fine characteristics.In the testing stage after the system implementation,through the functional level and non-functional level tests,as well as the five functions and three performance evaluation of the system,the system has also initially met the basic requirements of the insurance company for the realization of data integration system.The real-time information integration system of insurance customers designed and realized in this paper can not only obtain more customer data associated information,but also provide information services for insurance policy management and insurance system construction,and at the same time provide a large number of reliable,accurate and valuable decision support for insurance design and planning. |
PDF Full Text Request