Research On Distributed Complex Task Tree Scheduler For Big Data Storage System

Recently,big data has been gradually become the research focus of many industries currently.At present in the field of Information Science,the research of big data mainly focuses on the collection,storage,mining,display of big data.The storage and management technology of big data is the critical scientific problem that precedes the analysis and processing technology of big data.Therefore,it has important theoretical and practical significance to carry out the research on storage and management of big data.After the analysis and study of the current relevant storage management technology systematically,this thesis proposed a framework of distributed task management based on message queue technology in big data storage system.Then a big data storage system prototype was implemented using the framework.The main work of this thesis can be summarized as the following aspects:(1)Studied the architecture of current common storage system,proposed a new type of task processing engine for the characteristic,which decouple the task description from the execution.Implemented a new type of data structure that called complex task tree,which can be used to describe a set of complex task.(2)Designed the task scheduler of Complex Task Tree in big data storage system,which can be used to determine the execution order of tasks which there is a dependency relationship between them.(3)Implemented the across-node task type that called distributed complex task tree by expanding complex task tree.A complex task tree that can be physically distributed on multiple machines amounts to a complex task that can be executed on multiple servers.This practice solves the performance bottleneck problem causing by common rich client by utilizing idle CPU resource fully.Provided a uniform service model which makes the different roles in the storage system apply the same execution logic.By using uniform service model,all tasks in big data storage system can be described by distributed complex task tree.(4)Implemented a big data storage system prototype by applying asynchronous non blocking mechanism based on message queue,and describing the tasks based on distributed complex task tree.The study of storage management mechanism of distributed complex task tree scheduler in the big data storage system has some reference value on the research for further optimizing the performance of big data storage system.