The Research On Storage Algorithm Of Optimization For Big Data Analysis

With the progress of science and technology and the popularity of the network,People's daily work and life will produce huge amounts of data information.How to use these huge amounts of data information effectively by mining the hidden information,that is the key to the future development of enterprises and government agencies,which requires big data analysis technology.The premise of big data analysis is efficient storage and queries.In the distributed storage system,the higher the efficiency of data analysis,the query efficiency of the corresponding storage system must be higher.In this paper,the application of scenario is the situation in which data types are known,the content characteristics of the data file is clear,the content of the data file size is uncertain,the most of the file size is small,and the user is particularly high demand for data access.In this paper it puts forward the solution combining with engineering practice.Based on the structured data model,we propose a type of the file system data structure for data storage through the establishment of data storage unit,data file,data page,data line,data fields,such as multi-level storage structure,to achieve the level of the data storage.This storage structure improves query efficiency by establishing a multi-level index,and supports row storage,column storage mode and data compression.The user can choose compression mode.According to this model,the paper gives the corresponding storage algorithm.On this basis,in order to manage the cost of storage memory,this paper introduces the fixed memory pool to improve storage algorithm.In order to keep the efficiency of query,this paper introduces a dynamic index algorithm in constant memory pool,and makes full use of memory,and improves the query speed at the same time.Test results show that the data storage model improves greatly the storage and query speed compared to traditional storage model,but the compression efficiency is dropping inevitably.Constant memory pool dynamic index algorithm,which is the improved the LRU and LFU hybrid algorithm,improves greatly the query efficiency and effective ratio of files.