Design And Implementation Of Big Data Analysis Platform Based On Hadoop And Storm

With the rapid development of Internet,big data becomes the hottest topic in industry and academia,and begins to show the characteristic of big volume,fast processing speed,variety of types,high value and reliable veracity.In order to cope with rapid change and development of big data,people use a variety of approaches,people use batch processing to analyse offline data,use real-time processing to deal with real-time data,use SQL queries to respond to the demand of interactive searching,and use NoSQL databases to store different kinds of data.These approaches have been widely used in many different kinds of areas,such as personality recommendation,finance and insurance,intelligence analysis,how to deal with diverse and changing demands is a great challenge of big data analysis.In addition,with the emerging of various computating frameworks,how to use these computating frameworks effectively is also a great challenge.In this context,we designed and implemented a big data analysis platform based on Hadoop and Storm in this paper.First,we researched the basic architecture of the big data analysis platform,and divied it into data sources,the ingestion layer,the storage layer,the big data analysis layer,the mornitoring layer,the infrastructure layer and the application layer.The seven-layer architecture is the foundation of the big data analysis platform.Second,we focused on the construction of big data analysis applications.We designed and implemented an application architecture for big data applications,which has low degree of coupling and good scalability.The application architecture is divided into UI layer,web service layer,base-platform layer and application storage layer,which separates the front end and back end and standards the process of building applications.Then,we designed and implemented the big data analysis platform based on this application architecture,which contains modules of user management,data management,workflow engine,data exchange and real-time monitoring based on Storm,using Redis and MongoDB for data storage,and workflow engine has the ability to deal with MapReduce and Spark algorithms at the same time.The entire platform has batch processing and real-time processing processing capacity.Finally,it proves the correctness and effectiveness of the design and implementation of the platform by functional tests to make sure that it can deal with various demands of big data analysis.