Research On Out-of-core Middleware Key Technologies In Cluster Computing

Many applications in scientific computing require a tremendous amount of data. Often, such applications run on thousands of computing nodes so as to ensure that the data of the application fits in the main memory of the distributed machine. However, for some irregular applications, like sparse linear solvers, running on thousands of nodes typically do not achieve high efficiency, leading to a significant waste of computing time. Out-of-core computing allows to use hard drives to store, schedule and manage the data of the application. Middleware technology makes cluster computing more convenient and efficient.This thesis focuses on the related technologies of out-of-core middleware in cluster for numerical linear algebra computations, and presents an out-of-core middleware, which has been used to constracted serval different fuction..parallel and less energy-cost application. The middleware is general useful,effiecient and easy to use. The main reseaches in this thesis are including:the architecture of data-flow and task-flow in out-of-core middleware, the researches on high efficient task scheduling methods, saving computing resources by out-of-core in green computing, distributed data stream processing system middleware construction method, etc. Specific researches and innovations in the thesis are as follows:(1) An partitioned Out-of-core middlewareThis thesis proposed a new distributed out-of-core middleware om top of a data-flow middlware which named DataCutter, for large-scale numerical linear algebra computations. The middleware achieve real time communication and collaboration between multiple nodes. In the global address management mode, the middleware manage all the data including which much more larger than memory system by partitioning data into numbers of blocks. The middleware manage tasks in global and local level. Compared with DataCutter, this middleware has better achitecture, better data flow and tasks flow management, could bulid paritioned out-of-core computing system much easiler in cluster. By tasks partitioning, it achieve a separation of the data-flow and task-flow, process-level parallelism between nodes and thread-level parallelism within node. The effieciency of middleware has been proved in hard disk reading tests and in-core tests.(2)Scheduling Methods based on Out-of-core cluster computing For the task management in the distributed out-of-core middleware mentioned above, this thesis provide a data-aware based scheduling strategy. By using Direct Acyclic Graph, the strategy sets data dependencies between tasks, partition tasks and establish the task sequence according to each tasks’ data dependencies. When part of graph, which represent a part of tasks, arrive computing node, all the tasks are break up in task pool, then compose a task sequence. Through dynamic perception of existing data in local memory, it adjust the order of task sequence, change the priority and trigger the current task which have meet the data dependent conditions already. In this algorithm, it result in reducing the number of external memory access and trigger task which has enough data as much as possible, in oreder to improve whole system efficiency. Tests have proved that the data-aware scheduling method has much better efficiency than MPI scheduling method.Additionally, in this thesis the data-ware scheduling has been improved by several method such as prefectching function, synchronization controlling, etc.. The prefectching function allows system load data in advance while computing cuttrent task, and new synchronization controlling allows trigger more operations at a same time, which has data dependency with each others.(3)DOoC middleware in green computingFirstly, this thesis test the ability for green computing of the DOoC middleware for in-core and out-of-corecomputing model. Then implement a system similar with MFDn system by this middleware in a high performance computing (HPC) platforms equipped with solid state drives (SSD). In the results of experiments, The similar system achieved apparently better energey efficiency than MFDn. By comparing with MFDn system, it demonstrates that the out-of-core implementation on the SSD test-bed can compete with an in-core implementation in terms of total CPU-hours and energy efficiency.(4)Linear algebra system programming frameworkThe implementation of parallel system in cluster system is always extrmely difficult, in which the programming is always super complicated. In this thesis, we present a novel distributed out-of-core linear algebra framework for general distributed data-flow computing construction. By using the interface applied by this framework, the programmer could use C or C++programming language, complete the large-scale implementation of distribute out-of-core solvers in cluster environment.. This framework reduce the workload pretty much in architecture of such a system, and provide a conmen, easy-to-use, and efficient way to resovle.