Research On The Storage System And Optimization Of High-performance Cloud Platform Based On MPI

In order to meet the urgent needs of user for computing capacity, the efficiency of resource utilization and resource centralization, cloud computing technology is put forward as a solution and developed rapidly in a short period of time. In recent years, the research institutions and companies are beginning to study the cloud computing technology and providing applications based on the service of cloud computing. But many cloud platforms are not suitable for low delay service, and the task execution efficiency is not high when dealing with compute intensive task. MPI(Message Passing Interface) is good at computation intensive tasks, and communicates quickly, has little delay when transfer message, to achieve a cloud computing system by MPI can make up for the disadvantages. Our parallel laboratory has achieved a high performance computing platform based on MPI before, but it does not have a distributed file system as the underlying storage, the realization of some functional modules is also need to be improved when dealing with tasks that has calculation dependent. The main work of this thesis is as follows:First, it researches on the WCP(Weighted Critical Path) task allocation algorithm suitable for LilyTask computing model, and proposes an improved WCP algorithm suitable for the job in our MPI-based platform, this algorithm takes into account the existence of several communications between two tasks on the basis of the original algorithm.Second, it designs and implements a task dependence definition program based on a visual graphic interface, the user can draw a task dependency graph on the main interface through the button and the auxiliary function, and the dependency graph can be stored in a certain format in a file.Third, it does experiment analyse on the object information query job, and compare the running time of object information query job that applies the improved WCP algorithm, the original WCP algorithm and the task allocation method of previous platform respectively. The results show that the improved WCP algorithm has a performance advantage compared to the other two kinds of allocation algorithm.Last, it focuses on the research of HDFS and HBase, apply HDFS as the storage system of the platform, and integrate HBase into the parallel query system to replace the previous MySQL cluster. And then tests the performance of the improved parallel query system, the results show that the efficiency of data query is higher.