Research And Implement Of The Big Data Storage System Based On Data Attributes

The development of computer and Internet technology has led to the rapid growth of the amount of big data. Big data is not only reflected in the increase in the amount of data size, but also in terms of the kind of data increases. Using traditional database and file system to manage large data exist many problems. Therefore, the study of large data storage and management has become an important research topic.Based on this situation, After in-depth analysis and study of the current big data storage technology, this thesis proposed an Attribute Vector-based Unified Data Model(AVUDM) and implemented a big data storage system prototype using HDFS, Fast DFS and Mongo DB as the underlying data support.The main work of this thesis can be summarized as the following aspects:1. Studied the big data research status and the problems faced by big data storage. Analyzed big data storage requirement and proposed an AVUDM model2. Studied Mainstream distributed file systems and distributed NOSQL databases. Contrast the read and write performance, fault tolerance, availability and scope of these distributed file systems and NOSQL databases3. Implemented a AVUDM model big data storage system prototype using HDFS, Fast DFS and Mongo DB as the underlying data support.4. Including adding file- level data de-duplication to the system, improving the load balancing algorithms of Mongo DB shards and some parallelization optimization in the system.Finally, deployment experimental environment, then test the upload data file performance, data retrieval performance of the big data storage system and compared the time consuming of Fast DFS file attribute extraction between using parallel methods and serial method. The test results proved that the storage system prototype can efficiently store and manage large data.