Design And Implementation Of Big Data Storage System Based On Hadoop

With the quick development of the Mobile Internet, the data quantity increases to TB even PB level everyday. Traditional data storage technology can not handle the mass data any more. So people pay more attention on how to store and read data efficiently. Recently, Hadoop which has the advantages of high reliability, high scalability, high fault-tolerance and high efficiency,etc.is a mature way to store mass data. Furthermore, Hadoop is a kind of open source frame, which is very suitable for science research. So, this thesis chooses Hadoop platform to build a mass data storage system.Hadoop is a kind of framework which can efficiently distributed process mass data. It mainly consists of HDFS(Hadoop Distributed File System) and Map/Reduce(Hadoop Distributed Computing Framework). HDFS can divide the document into several file blocks with same size which men are stored in different nodes. And Map/Reduce is used to calculate based on the mass data.Firstly this thesis uses HDFS file document as the main object to make a deep study and analysis of its structure and way of data organization. And this thesis also gives prioritization scheme of MongoDB Database to solve bottleneck problem of performance of the single namenode. Then this thesis builds a mass data storage system based on Hadoop platform by using cheap PC machines. And this system can achieve file upload, download, share, file safe, file browsing on line, recycle bin and so on. Meanwhile, this system also uses Ganglia to monitor mass data storage system in real time, because it can easily check the working condition of each node. Practical application shows that this system can meet the needs of mass data storage well and can be used as a data storage center in enterprises and universities.