Research And Implementation Of Data Processing Framework Of IoT Based On Storm

Much attention has been paid to the development of the Internet of things in recent years. As cloud computing technology is increasingly mature and developing smart service-oriented society, the development and application of the IoT technology plays a pivotal role today. With the expanding range of the IoT applications, the research of the IoT technology is gradually in-depth. In the architecture of IoT, due to the diversity of sensors and perception objects resulting in the data in the IoT presents some characteristics, such as multi-source heterogeneous, mass, the spatial and temporal correlation and highly redundant, it will bring some difficulties for data processing. At the same time, the data management layer provides support for the upper application, therefore, accurate and efficient data processing has become the key to ensure the accuracy of the IoT application.The processing technology used in the data processing of the IoT were data fusion and MapReduce currently, as well as used based on the file system or relational database storage, based on the non-relational data storage used only in a small number of applications. Studies have shown that the efficiency of data fusion will be gradually reduced with the increase of amount of data, MapReduce has no longer competitive in dealing with small files and cannot meet the demand of real-time performance, file system’s storage way has a high degree of customization, low degree of data sharing, relational database performance is a bit poor when storing unstructured data.According to the characteristics of the data in the IoT, this paper designs and implements an IoT data processing framework based on Storm, including data processing and storage. Firstly, the processing of the data is obtained by using the optimization after the Storm, this "real-time MapReduce" approaches to further to the characteristics of data in the IoT. As much as possible to save system resources, this paper puts forward the optimization scheme of Storm, namely RIPBS (Resource Isolation Policy based on Storm) strategy and elaborated implementation scheme. Secondly, data processing model based on RIPBS strategy is proposed, and the realization of two common modules are introduced in detail:Spout reads the data source and the Zookeeper optimizes application in the data processing model. Thirdly, the data storage center based on HBase is raised in this paper, putting the IoT data stored in the cloud platform and NoSQL database, and putting forward the specific implementations of storage center, maximum limit satisfy the characteristics of the IoT data. It is also evaluated for RIPBS strategy and the storage center based on HBase proposed in this paper, experimental results showing that RIPBS strategy can be well controlled system resources, and storage center proposed by this paper exists more advantages.