A Research On Automatic Deployment Of Distributed Storage Resources For Big Data Systems

With the growing popularity of Internet-based services and the trend of hosting them in the Cloud,more powerful back-end storage systems are needed to support these services.On one hand,the storage system itself should be capable of handling workloads with higher concurrency and intensity.Distributed storage solutions are proposed.They are expected to achieve some of the following properties: scalability,availability,durability,consistency,and partition tolerance.It is difficult or even impossible to have a distributed storage solution that fully satisfies all of these properties.Thus,there are researches that focus on pushing the limit on achieving those properties in the designs of distributed storage solutions satisfying different usage scenarios.On the other hand,the increasing trend of Cloud computing brings new challenges to the design of distributed storage systems.In order to better enjoy the benefits from the pay-as-you-go pricing model in the Cloud,a dynamic resource provisioning middleware,i.e.,an elasticity controller,is needed.An elasticity controller is able to help saving the provisioning cost of an application without compromising its performance.The design of an elasticity controller to auto-scale a distributed storage system is not trivial.The challenging issue is the data migration overhead(data consume time and resources to be transferred to added instances or transferred out from removed instances)when scaling the system.This thesis design automatic deployment of the system is mainly adopted the workload forecasting method to realized,through monitor storage system performance index(eg request latency?CPU utilization?VMs),To forecast the next of PW storage system's workload by monitoring the current and past workload data,as well as use Monitoring data derived storage system minimum configuration under satisfy SLA,so we can re-configuration storage system before the next PW.The data is migrated when adding or removing instances,so that the request is more evenly accessible to each storage instance.For Predicting Workload,This thesis usesa specific type of workload.The prediction of the workload is the core of the whole system,this thesis uses the Wiener predictor based on the Wiener filter principle to forecast the workload of each workload predict window.According to its periodic workload will be divied into a specific length of the predict window.Forecast the wordload of the next predict window at the beginning of each predict window.Finally,Through a large number of experiments and tests on the proposed algorithm and verify the performance of distributed system,it is confimed thar the Wiener-based forecasting scheme can be used to ensure SLA,saving platform resources alse saves configuration costs.