Information Storage Technology Research For The Internet Of Things

With the rapid development of the Internet of Things (IoT) technology, more and more applications of IoT are applied in life and industrial production, and the daily amount of data produced by IoT increase rapidly. The characteristics of IoT’s data are polymorphism, mass scale, real-time and so on. The data produced by IoT at a time are in type of mass scale of small files. Hadoop is a kind of open source cloud computing technology widely used in dealing with large scale of data. However, Hadoop has some drawbacks such as low processing efficiency and high rate of system resources occupation to deal with mass scale of IoT’s small files, so it is very meaningful to improve the processing of storage of IoT’s data in the Hadoop system.According to the characteristics of IoT’s data and the insufficient of storing large scale of files in HDFS, this paper uses Hadoop cloud technology to clustering the mass small files of IoT which have two-dimensional coordinate attribute, and proposes some relevant storage optimization strategy. Firstly, this paper uses G-means algorithm which can choose a best number of clusters to clustering the files, and uses quad tree structure to index the files in each cluster. Then, according to quad tree structure, the paper uses a file merge strategy which based on the distance of files’ two-dimensional coordinate to merge the files in same cluster into a big file which consists of several blocks. Lastly, the paper uses the preloading of local index strategy to improve the efficiency of files access and reduce the burdens of NameNode.At last, the paper has the experimental tests of G-means algorithm based on MapReduce. The result indicates that G-means algorithm based on MapReduce has advantage to clustering mass scale of data. The paper also has the experiment of merging large scale of small files. The merge strategy can reduce the usage of NameNode. The paper also has the experiment of preloading strategy. The result indicates that it will have a good effect when files in the same blocks are accessed frequently.