Based On The Distributed Mass Image Data Storage System In The Research And Design

With the rapid development of the Internet and the rich development of information resources, all works of life will produce a huge amount of raw data, especially data represented with the form of pictures. How to manage and store information resource is becoming more and more difficult. Under the Internet age in which information grows explosively, the traditional way of file storage is not able to meet our requirements in processing time and storage space. The study of High-performance distributed file system is becoming a research hotspot.Based on the analysis and comparison of the distributed file storage system at home and abroad, FastDFS is selected to be the storage system in this thesis. FastDFS is an open source lightweight distributed file system, which can solve the problems such as data storage and load balancing. The grouping and file naming mechanism of FastDFS makes itself has the advantages of lightweight, high scalability, and ease of use. As FastDFS is an open source distributed file system, this thesis tailores a picture processing system to meet the requirement of the enterprise. The works of thesis are listed as follows:In this thesis, the first job is to improve the synchronization mechanism of storage server of FastDFS. The original method synchronously updates files at regular time. However, the improved one can synchronously update files during upload of them.The second work of this thesis is to improve the load balancing algorithm of FastDFS. The factors considered are extend from one factor(total storage space) to three factors(storage space utilization rate, the number of connections and performance). Experiments are conducted in three cases: One is only considering the utilization ratio of storage space. The second one is to consider the storage space utilization rate and the number of connections, and the last one is to consider the storage space utilization rate, the number of connections and performance. By comparing imbalance degree of the load for the three cases, it was found that the load balance effect of the improved load balancing algorithm is more obvious than that of the original algorithm.The third work is to apply FastDFS into picture data processing system, so that the system can be more efficient and effective in file management by detecting and eliminating duplicate data. This thesis starts from the system structure design to the realization of key technology to introduce this system. By environmental testing, the system can well meet the needs of image data system on acceptance ability, concurrent processing, real-time response ability, and high extensibility.This thesis improves the synchronization mechanism and load balancing algorithm of Fast DFS, and applies them to picture data processing system. The experimental results show that, compared with the traditional single Fast DFS file system, a series of mechanisms designed in this thesis can better meet the needs of the business enterprise on multiple demands, high utilization, and easy extension.