Research On Tide-bound Water Level Task Scheduling Pattern Based On Spark

With the advancement and development of computer technology and the ocean observation technology, ocean environmental data takes on diversified features with large amount, multi-scale, complex types, varied forms, distinct space time magnitude and dynamic update frequency. In order to further master the way to processing problem of spatial data in long time series, with China’s current numerical computational mode of cloud computing forecast taken into account, Spark cloud platform is adopted in the numerical prediction model of tide-bound water level so as to change the traditional business computing model.In recent years, the rapid development of cloud computing technology has become an important impetus to economic and social development. Parallel computing provides a new technical route for marine data processing. As a memory type computing framework, Spark parallel computing framework absorbing the advantages of traditional Hadoop framework, successfully attains a more universal abstract programming which can deal with different computing tasks and realize the extension of MapReduce which is called elastic distributed data sets (RDD). RDD has greatly promoted the commonality of Spark parallel computing framework, efficiently increased the data sharing function. RDD can effectively support different distributed computing with a single execution model. Given the above advantages of Spark framework, this thesis carries out a controlling model research on tide-bound water level based on Spark.According to requirements of the national marine public welfare project "Research on Marine Environmental Information Framework of Cloud Computing and Cloud Service System", based on laboratory-developed task scheduling pattern towards tide-bound water level and web interaction system, this thesis explores task scheduling pattern to water level under Spark and Spark primary scheduling algorithm. In the light of the specific characteristics of tide-bound water level computation, the suitable scheduling algorithms being researched and determined, the experiment result indicates that in the framework of Spark, tide-bound water level task scheduling pattern can change the traditional business computing model, introducing Spark platform into tide-bound water level computation, lessening the dependence of tidal level calculation on high performance cluster and reducing the hardware cost of calculation. In the aspect of the flexibility, the newly-developed scheduling algorithms can change the one-piece calculation mode and improve the efficiency and accuracy of the local area according to the requirement of calculation; in the aspect of control mode, the modified control algorithm can improve the rationality and scientificity of task allocation, so as to further enhance the efficiency.The experiment results show that the Spark-based tide-bound water level task scheduling pattern can improve computation task allocation efficiency and speed up the computation.In this thesis, the main research contents and work are as follows:1. Research the current development and application of the cloud computing technology, as well as the advantages of cloud computing technology on huge amounts of data processing. With the typical application on the marine environment, namely, tide-bound water level task scheduling pattern, Spark parallel computing framework is applied to tide-bound water level task control mode in an effort to change the traditional business computing model.2. Allow for the shortcomings of traditional integration of cluster computing, the tide level calculation Web interaction system is designed and implemented, which increases the flexibility, elasticity and interactivity. In addition, it is in favor of the visual interactive browsing and promise zoom of calculation results to achieve the purpose of the ocean environment information processing with cloud platform so as to provide a more efficient means for visual display and data analysis and contribute to disclose the ocean laws.3. Comprehend the various nodes distribution of computing tasks through monitoring calculation process.4. After the study of Spark native scheduling algorithm, combined with the characteristics of tide-bound water level task and the cluster computing power and complexity of computing area, a set of optimal scheduling algorithm is put forward, effectively improving computation task allocation efficiency and speeding up the computation largely.