Research And Design Of Cloud Platform Framework For Beibu Gulf Marine Science Data Processing

With the development and utilization of Beibu Gulf marine ecological resources, mass marine science data had rapidly emerged in large numbers and these data is massive, diverse, complex, dynamic, heterogeneous and so on in characteristic. But at present the projects of marine scientific data are lack of uniform standards and specifications of the storage and collection, without corresponding data sharing platform, so a "data island" had been formed. Now it become one of the key problems for marine scientific research projects that how to manage and store these mass marine science data in order to use these data efficiently. Traditionally, these technologies, such as parallel computing, distributed computing, grid computing, are mostly used of processing large-scale data, but it need expensive computing resources, but also how to put the mass data effective segmentation and allocate computing tasks reasonably requires tedious programming to accomplish. With the rapid development of cloud computing technology, especially Hadoop distributed technology, as a representative of the cloud computing data processing technology, can just provide an effective way for the solution of the above problems. Based on cloud computing technology, focusing on how to construct the Beibu Gulf marine scientific data processing cloud platform framework solution, this paper has carried out the following several aspects of the research work:(1)This paper analyzes the construction demand of Beibu Gulf marine scientific data processing platform, Then the key technologies of cloud computing and the existing frame structure of cloud computing platform are summarized and analyzed. Finally it has a thorough analysis and research on the representative cloud computing Hadoop distributed platform.(2)According to the characteristics of Beibu Gulf marine science data and the special processing demand, this paper clears the functionality of Beibu Gulf marine scientific data processing platform, the platform system operational flow. According to the cloud computing "Software as a Service (SaaS)" mode, this paper designs a cloud platform framework which conforms to the own requirement of Beibu Gulf marine scientific data processing. This platform which consists of three aspects includes distributed database access layer, data processing platform layer and business application layer. It can realize the distributed file system、the parallel programming environment, the distributed system management, the control and scheduling of parallel data storage, the application software development of business application layer, the user application interface functions and so on.(3) The Prototype system of Beibu Gulf marine science data processing cloud platform has been set up preliminarily, and this paper expounds the realization method of key modules and gives specific build steps of platform construction. Using parallel distributed computing technology and Linux cluster technology, based on Hadoop distributed platform, in this paper, Beibu Gulf marine science data has been processed in these data processing methods such as HDFS distributed file system, Map/Reduce parallel programming computing model and HBase distributed database technology, the key modules have been realized by programming, the distributed system management on the cloud platform has been achieved, the storage platform with high reliability, high stability has been provided. At last, the feasibility and effectiveness of the platform framework has been verified and illustrated through the test.In this paper, the research results and the design of Beibu Gulf marine science data processing cloud platform framework provide a viable solution for building cloud platform which can manage and store massive amounts of marine science data based on Hadoop cloud computing technology. So it provides practical application value and scientific significance.