Research On Storage Method And Technology Of Complex Large Data Based On Density Partition

Since the birth of Internet,especially since twenty-first Century,with the rapid development of Internet and Internet of things,huge amount of data is generated everyday.With the rapid development of technology of machine learning,people can make use of existing technology can be a complex event pattern between data mining from large data,through the data mining a set can better serve the daily life and production of us.The data source object that this subject deals with is the mining of complex event relational data set.With the complex event relational data sets is larger and larger,and the storage space of the existing storage equipment has been unable to meet the large data storage,and for the development of speed related technology to improve the memory ability of the hardware equipment of the data set can not catch up with the expansion of the speed and scale of the effective storage of big data is an important problem that need to to solve the.As one of the most important technologies for data storage,data compression technology has become the focus of this paper.Aiming at the complex event data set with the repetitive nature of the relationship between the data ratio is too high,too much redundancy problem,proposes a data source density region partitioning algorithm based on density distribution,extract the high density data area in a data source,the high density area in a lot with the repetitive nature of the data to achieve uniform erase operation,the purpose of data compression,and the traditional classic LZW compression algorithm to make a horizontal comparison,do further analysis and Study on the compression performance of the data compression strategy.Finally,in order to solve the existing single data storage device for large data storage capacity,storage efficiency is low,the advantages of distributed file system for large data storage,distributed file storage system in HDFS(Hadoop Distributed File System mainstream)based on the proposed data compression strategy,index structure the use of B tree algorithm,the design and implementation of a data compression tool to test the feasibility study of the work and the optimization of complex event relationship data storage method.