Design And Implementation Of Key Modules In Cold Storage System

With the rapid development of the Internet industry,a large number of data are generated at every moment.The storage of large data has become a hot issue in the field of storage.According to the survey,while the amount of data has increased dramatically,there has also been a phenomenon of data access strategy stratification.Cold data accounts for about 80%of the total amount of data.Traditional data storage methods solve the problem of large-scale data storage,but there are still problems of high energy consumption and high cost in cold data storage scenarios.In this thesis,we optimize the cold data storage scenario.By controlling the power-on and power-off of the hard disk of storage server,we can make the hard disk with low frequency access sleep,which greatly reduces the power consumption of the system.Thus,a low-power,low-cost,high-availability and scalable distributed cold storage system is designed and implemented.We mainly study the function modules,status monitoring modules and storage client of the system in this thesis.Firstly,we describe the research background and status of cold storage system,and compare it with the existing storage systems,and put forward the design idea of cold storage system.Next,we introduce the system architecture from the hardware and software level,and explain the composition and function of the key modules of the system in detail.We use Erlang/OTP framework to design and implement the disk power control module and health checking module.In order to improve the reliability and fault tolerance of system data,we design and implement data scan module and file rebuild module.By combining open source logging platform ELK with the cold storage system,we design and implement the monitor module of servers and modules in the system.Based on aggregated queries of monitoring data,we design and implement the system alert module.In addition,a storage client program based on command line is designed and implemented in Golang.It can be easily used to list,upload,download and delete files in the cold storage system.The detailed process of file upload,download and data transmission in the cold storage system is introduced.Then,we make test for system performance and power consumption.Finally,we summarize the main work of this thesis,and propose the directions of system optimization in the future research.