The Design And Implementation Of Task-based Massive Insurance Data Processing System

With the rapid development of internet, more and more companies have transformed from providing separate service into building a platform, through which they can establish a win-win cooperation with lots of third parties. Those companies can have access to vast amounts of user data and products data through platform. But with the rapid expansion of application platform, the data generated grows exponentially. How to process the massive data quickly and effectively becomes a big problem.The insurance data processing system researched in this paper is based on Alibaba insurance business, it is developed by the insurance core team combined with business goals and processes. This system is aimed at processing new data produced by the insurance platform, then writing those data into files, which will be delivered to according insurance company. To handle massive data, this system adopt odps and hbase to pre-handle and prepare data, and use cron job which is based on Spring frameworks to retrieve and handle massive data. In the design of this system, the feature of expansibility and scalability are fully considered. It can support processing strategy of different insurance companies and different insurance products. What’s more, it can support cluster deployment, which can use multi machines to run task collaborative. In fact, the design of this system is also applicable to other platforms. Through the massive data-processing services, it can not only provide a good user experience for third party companies, but also provide a stable basic services for the business goals.