Design And Implementation Of Data Collection And Analysis Platform Based On Distributed Storage

With the rapid development of the Web 2.0,Mobile Internet and Internet of Things,the data show explosive growth.The research and development of big data platform for huge amounts of data is becoming a hot spot in the industry.Data collection,storage,and analysis are the core problem of the big data platform.Data collection is mainly used to solve the problem of multiple data source access.Data storage is used to cope with the challenges such as diversity of data format and massive data storage.And the goal of data analysis is to support diversity algorithm,and the pluggable analysis algorithm engine should be provided.Based on the investigation,analysis and application of the mainstream big data open source framework,data collection and analysis platform based on distributed storage is proposed in this paper.This solution provides unified interfaces for data collection functions,combines a variety of storage mechanisms for data storage,involves the plug-in algorithm engine for data analysis.In this paper,the investigation on mainstream open source framework and technology platform is given at first.Then the technical characteristics of the distributed system are analyzed and functional requirements and non-functional requirements are put forward to for data collection and analysis platform of big data.Next,the platform interative process is introduced and all stages of the evolution of the platform architecture are specified in detail,followed by the platform implementation,functional and non-functional test.In the end,two concrete applications based on the platform are presented as examples to further verify the effectiveness of the platform.