The Design And Implementation Of Performance Optimized Distributed Storage Subsystem

With the rapid development of the Internet, people are stepping into an era of blooming Internet today. Data is growing in hundreds of times the speed in the past, which leads to an exponential rise of the data quantity. Problems which we are facing have largely gone beyond the capability of several persons or computers. The Internet ties the wisdom of people all around the world, and meanwhile, raises a tough challenge to us: how to handle these explosive big data. To solve this problem, the conception of “cloud” is proposed, which in turn gives birth to the distributed theory.In addition, various different applications of large companies lead to increasing expectations on cloud data storage, especially for the different scenarios proposed different requirements, which make the traditional distributed framework deal with the problem weakly. To meet the changing needs of users and provide a better experience for users, the existing distributed file system had to do change with its corresponding. Our lab is also in line with the purpose of scientific research, learn from the classical framework of distributed systems, designed our own distributed file system C_STORE. The paper first introduces the popular distributed file system at home and abroad, and studies them, then gives us the design of the system data management module, after that proposed performance bottlenecks and has improved on this basis.C_STORE uses load-balance and recovery strategy, it has good scalability and reliability. It separates the management of data and metadata, which allows the client access to the server more efficient, but also to facilitate the management of resources, both full use of resources, but also reduce the pressure on the side of server. On the client side, we implement client-based software for different platforms, such as UNIX, ANDROID and WIN platform client. We also realize the user authentication module, which facilitates the user login and assigning user storage quotas, and ensure the security of user account.The data storage module provides good data storage for users, in order to ensure the availability of users’ data, it takes a redundancy policy, each data would be backed up, recovering mechanism to ensure the reliability of data, using logic clock to ensure data consistency. Between duplicates, the model used a backup copy of the heartbeat connection push slice data, once appear machine hotspots, migration mechanisms and expansion mechanism will guarantee load balancing of data between modules.Finally, we will give a basic functional test modules and the basic performance test for the size of files, and then we give the consumption indicators of the hardware’s parameters for the machine after we optimize the performance, which will proof our system that is real available.