Research And Optimization Of High Availability Management Technology Based On Alluxio

As storage hardware costs are declining,big data ecosystems are undergoing complex changes,and there are significant characterist ics of diversit y and heterogeneity on calculating framework and storage systems these days,a series of memory-based products such as distributed file system and database appear,helping to integrate the big data ecosystem,and provid ing better services for the outside world.Availability is an important ind icator for evaluating the performance of a mass storage system.The paper research on the current open source me mory-based virtual d istributed storage system Alluxio based on the perspective of improving the availability of a mass storage system,and mainly study the availabilit y optimization technology of data management mechan ism,in order to improve the availabilit y of a mass storage system which combined Al uxio and the underlying storage in a remote environment.In the paper,the availabilit y of a mass storage system which combined Alluxio and the underlying storage is taken as the research point.we base on some of the current availabilit y techniques of other distributed file systems or memory-based database systems,analyze the unava ilable state of the data that is not accessib le in the underlying data in a remote environment because of the formation of unpredictable factors such as the network,and data is not available under asynchronous storage because of asynchronous mechanis m and other reasons.Then,we put forward two optimization strategy: first,cache prefetching and replacement,We should pre-fetch data to Alluxio,increase the thermal data capacity of Alluxio,reduce the network congestion when the data transmission pressure,reduce the number of access to the underlying storage,and extend the external service time when the underlying data is inaccessib le.Second,optimizing the process of asynchronous storage,proposed asynchronous operation optimization strategies which are combined with operation?That is,when the operation is clear and idempotent,the underlying simultaneously has the corresponding comput ing resources,you can use Alluxio to send commands directly to the underlying storage rather tha n data,whic h relieves the network pressure from transmitting large amounts of data.Besides combining asynchronous and synchronous to further ensure the availability of persistent data.Based on the above optimization ideas,we proposes the follow ing strategies: data prefetching and replacement strategy based on association rules between data blocks,and asynchronous storage optimizat ion strategies combined with operation.In addition,We have a more complete solution to the above issues.Finally,a comprehensive analys is of the relevant optimizat ion techniques is carried out through experiments.According to the experimental results,it is concluded that the data prefetching and replacement strategy based on association rules can carry out data prefetching in remote environment,avoiding the unavailable external servicwhich is caused by network and other reasons,and keep the hot data in Alluxio for a long time.At the same time,the thermal data rema in in A lluxio for a long time,which reduce the latency of application access data,reduce the number of vis its to the underlying storage,mit igate the communicat ion pressure at high load of the network,and reduce the failure rate of the entire system downtime,thereby improve the availabilit y of external services.Asynchronous storage strategy can ensure the availability of data as much as possib le in the case of asynchronous,it's not only relieve network pressure of data transmission,but also ensur ing the integrit y and consistency of data and other performance requirements,in such a way that it ensures both the performance that program requires and the data availability.