A Data Information And Processing Platform Of Ocean Observation Network Based On Hadoop

The detection for ocean has varied from nautical record to various techniques in human history such as float network and satellite exploration.There are all kinds of ocean observation network,which has greatly enriched the types and quantities in oceanographic data collection.However,the data files can't be effectively extracted into the database as the file formats are different due to the characteristics of datasets such as dispersion,heterogeneity,diversity.At the same time,the traditional database is facing enormous pressure of storage and retrieval for the increasing number of ocean observation data,not to mention supporting analysis and processing of these large data.Hence,it's important to build a platform system that is able to meet the data strorage which format is diversified and process them.In this paper,we develop a robust platform with the capability of data storage and processing for ocean observation network based on the Argo float data,which satisfies the massive data storage and query,analysis and calculation.The main contributions are as follows:(1)A novel system framework is firstly proposed by applying Hadoop tool in data-information-technology platform for modern ocean observation network by investigating the current research situation of existing ocean observation network at home and abroad.The system consists of interactive application layer,platform service layer,the data persistence layer and based support layer,where interactive application layer is responsible for the display and the operation of the user,platform service layer for the analysis and statistics of the data,the data persistence layer for data distributed storage,and based support layer for coordination and failover of distributed clusters,respectively.(2)In the proposed system,based on the analysis of Argo float data and its file format,combing with the characteristics of distributed database HBase structure,and utilizing the idea of space filling curves to reduce dimensions of float files data,we design HBase tables which support the fast query of large Argo data to provide a variety of search methods.Finally,compared with the traditional database,the proposed HBase has efficiency advantage by read and write tests.(3)Utilizing the proposed system platform,the float data can be directly analyzed and processed.We present how to use the platform to implement inverse distance weight interpolation through the float within the search scope in this paper.The KD tree and block encoding is used in the interpolation process to reduce invalid interpolation points.At the same time,the parallel computing framework is adopted in high density interpolation,which solves the bottleneck of single machine processing efficiency.(4)A system with high availiability is designed to implement automatic master backup switching.The proposed system is deployed and the node in the platform is planned.,the system robustness is tested and ensured.Finally,the system application is showed.