Research On Storage Optimization Technology Of Big Data Processing Platform

For big data analytics platforms,storage is crucial for system optimization.Although researchers have proposed excellent work on the management design of storage,the existing optimization methods statically allocate resources and manage the storage according to the workload without capturing the characteristics of different workloads and dynamic memory requirements,what's more,the replacement of existing strategy cannot obtain corresponding task scheduling information from the system,lead to lose the chance of optimization.To solve above problems,this thesis proposes embedding the supports of data analysis to the design of operating system,which discovers a new automatized way to adaptively optimize the system.Our work can be summarized as follows:1.Dynamically allocating resources for computation and cache based on calculating workload.Since the system characteristics and the load may change time to time,we propose detecting real-time system process and parameters.According to the corresponding task scheduling,using data mining to analyze the relationship between behavior and performance of the system,and C5.0 decision tree algorithm to make decisions,and realizes the optimization of the storage system.When the framework applied to the existing system architecture,it can analyze detected system behaviors through kernel module,without changing or recompiling the program source code.2.Combining with the correlation analysis and data mining algorithms,to analyze the relationship between system behavior,system settings,and the system performance indexes,select the close indicators to track and analysis the system behaviors,in order to reduce the system consumption.3.Applying the proposed SDAF on caching system,by monitoring the current's behavior of the system,dynamically controlled the corresponding replacement algorithm of Cache,result in automatized adaptively optimize storage system.To validate the proposed ideas,we choose the cache design on Hadoop as experimental background.The efficacy of the considered case is verified by a series of experiments,where the results are quite encouraging.