Research On Distributed File System Of Supporting Gpu Acceleration

Along with the rapid growth of social information and diversified development of the Internet, all kinds of large-scale data, which is changing our life, gradually get popular in public. Facing such large-scale data, the management, retrieval, analysis, safety and other requirements have been challenges to traditional data storage. As a distributed file system, HDFS can efficiently solve data store problem. HDFS, which provides storage service and is integrated with MapReduce programming model, has advantages of easy to program, good extensibility and high fault-tolerance.Compared with CPU, GPU has a different structure and develops much faster,on the other hand, GPU does better at computing and storage broadband. Not only is GPU used in graphic and image processing, it is used in general computing and attracts more and more attention from scholars and exports. Distributed file system algorithm which supports CPU acceleration was brought forward in this thesis, it promotes the efficiency of processing large-scale date in parallel and storing files, on the other hand and has great theoretical and practical significance for big data analysis and processing.The main work of this thesis is as follows:1. Optimize and improve the architecture, file storage and caches of current distributed file system to store data processed in parallel by CPU on the basis of analyzing current distributed file system, then I did some experiments which proved that this algorithm can reduce the burden of HDFS and improve the efficiency of storing and performance of read and write Intensive files.2. Research theoretical knowledge of GPU programming model and MapReduce programming model, analyze current GPU acceleration MapReduce programming model, on this basis, I brought forward GPU acceleration MapReduce programming model algorithm and verified its feasibility to process and analyze big data after comparing it with Mars.3. Put forward a GPU acceleration distributed file system model and improve its ability of parallel processing and analysis. After comparative analysis of experiments, I proved the high-performance parallel computing ability of GPU in distributed file system.