Tasking Framework Of Flow Field Visualization Based On Spark

With the slogan of "Ocean Power Country" proposed, ocean industry has entered a rapid development period in China.In this tremendous natural resources, energy, marine geological treasure trove, the flow field data play a very important role in revealing ocean phenomena, preventing natural disasters, studying ocean power system and other ocean important theoretical research disciplines. But compared to other areas of environmental observation, the direct observation and measurement of ocean flow field is not only more difficult, but also more expensive. When calculating the flow field variation of the ocean, there are hundreds of time, but at each point of time and the distribution of thousands of data grid, the flow field data for these changes all the time, precisely we study variation of the flow field, looking to avoid the natural disasters, marine mining potential of important data in accordance with the law. Therefore, flow visualization research work, the marine economy of China to promote the process of transformation needed core technologies and key technologies. In the government’s strong support, the State Oceanic Administration intends to use the latest cloud computing technology in the ocean flow visualization for technological innovation. In this method, it is based on the original Hadoop flow visualization technology platform, further improve computational efficiency of the flow visualization by technological innovation.In this paper, flow visualization Tasking framework based on Spark aims to take advantage of Hadoop 2.0’s new resource management system YARN, Spark computing framework will replace the Hadoop MapReduce computing framework by using memory-based computing model Spark, elastic distributed data Set RDD, the intermediate data stored on disk and memory, and thereby reducing the IO overhead costs and scheduling time traditional Hadoop framework to solve existing under Hadoop ocean flow visualization Tasking framework based interactive computing defects, shorten the calculation time spatial sequence field for a long time, and improve scalability and enhanced ability to share many different applications of the cluster, thus further enhancing the flow visualization of computational rate, improved graphics computing capacity and enhance the user experience.For the requirements and characteristics of flow visualization mission, this paper studied the internal core structure and working mechanism of Hadoop-Tasking. For the composition of the new structure Hadoop YARN resource management platform, the internal mechanism, workflow, implementation and how to use YARN embed the Spark computing framework into Hadoop,we make a detailed study and propose a very useful technical solution. We study the internal mechanism and workflow of Spark computing framework to explore its calculation principle and its application in flow visualization computing and present a detailed technical implementation. Finally, according to the technology solution of the above offered, for flow visualization of the specific circumstances of the cloud platform identified the Task handling mechanism to improve the form-based flow visualization Tasking framework Spark, the flow visualization processing speed has been greatly enhanced.The significance of this study is not only to improve the computational efficiency of flow visualization,it but also has good portability and scalability. We can use it as an example for the next Task processing framework applies to all types of the vector field visualization,which laid theoretical basis and practical reference.This method paly a very important rolre in researching cloud computing technology on graphics.