Design And Implementation Of Data Storage Module Of IoT Resource Access And Intelligent Processing Platform

IoT resource access and intelligent processing platform involves modeling description for heterogeneous devices by unique uniform resources,and manages the connection of the devices and resource management platform;uses protocol to analyze the data collected by the devices,and deliver the analyzed data to upper applications through message distribution network.The data storage module is in charge of the data storage and query operation in the IoT resource access and intelligent processing platform.The data stored in the platform mainly includes sensor device data and data collected by sensors.Sensor device data are structured data with fixed format.Data collected by sensors are unstructured data.Data collected by different types of sensors are in various formats.Meanwhile,due to the large number of sensors and sensor data generated continuously,there are a huge amount of data collected by sensors.In the existing platform,data storage module uses single thread to insert data into MySQL database.As a relational database,the table structure of MySQL database is fixed and the length of the data field is limited.It cannot meet the need that we want to store unstructured data in the networking resources access and intelligent processing platform.At the same time,due to its characteristics and the low efficiency of single thread data storage,MySQL database is not suitable for mass data storage.How to store and process the massive data effectively becomes a problem that must be considered.On the basis of analyzing characteristics of the IoT resource access and intelligent processing platform and the key technology of the cloud storage,this paper proposes a scheme of data storage module based on IoT resource access and intelligent processing platform.The main work of this paper is as follows:(1)The data storage module is designed as three layers:database layer,access layer and data buffer layer.The database layer is composed of MySQL database and HBase database which meets the requirement of storing massive unstructured data in the platform.(2)HBase database uses distributed cluster implementation and the data redundancy backup.MySQL database uses master-slave replication mechanism,so as to ensure the reliability of the data.(3)The access layer provides a unified package for the operation of two kinds of database,and provides the interface of database selection,table operation,data operation and so on.(4)The data buffer layer uses ConcurrentLinkedQueue to store the sensor data subscribed from the message distribution network.It also uses multi-thread to process data and insert data into database.(5)An HBase load balancing algorithm oriented to Internet of things is proposed and realized to ensure cluster load balancing and improve the reading speed.(6)The platform is built under the experimental environment,including the construction of the HBase cluster and the construction of the MySQL master-slave replication cluster.Then I test the data storage module.(7)Finally,I summary the data storage module design scheme and propose further research.