Research On The Optimization Of Cloud Data Access And Storage Based On FUSE

In the traditional Internet model,companies need to estimate the required computing and storage resources in advance.Due to the different scales of different businesses,the demand for resources is often quite different.At the same time,companies will reserve certain additional resources for the business growth that may occur during the peak period,and these resources are often not fully utilized,which increases the hidden costs of the company.In the era of cloud computing,with the advent of cloud storage technology,resource utilization has been greatly improved.Based on the user space file system FUSE,using HDFS as the underlying storage mode,this paper proposes a new cloud data access and storage technology scheme,and performs a series of optimizations,which include: 1)For the problem of large-scale data concurrent access in cloud storage system,this paper designs a new load balancing algorithm.The algorithm calculates the degree of load based on the CPU,memory,and disk usage of the nodes in the back end server cluster.At the same time,the algorithm feedback it to the scheduler in real time.The scheduler then dynamically adjusts the weight of the node to achieve efficient utilization of the server resources.2)Aiming at the performance bottleneck of using FUSE to access HDFS,by analyzing the FUSE source code,combining with the Linux file reading and writing process,this paper optimizes the reading and writing performance of the scheme from three angles,which include reducing the CPU context switching time,using direct I/O instead of cache I/O,and caching some metadata.Experiments show that this article's optimization brings about 20% performance improvement.3)In order to improve the security of the cloud storage scheme,this paper implements fine-grained file access control mechanism based on file attributes.A series of experimental results show that the optimized scheme does better in terms of performance,availability and security.In addition to the analysis and optimization of existing solutions,this paper also implements a complete cloud storage system based on the optimized solution,and applies it to a big data analysis platform to deal with the daily data access and storage at hundreds of TB level.In general,the solution proposed in this paper fully resolves the pain points of the business side in data storage during the actual application process.And it is also suitable for a variety of different application scenarios.