Design And Implementation Of Data Ingestion And Storage System For IoT Applications

Internet of things(IoT)applications are gradually moving towards platform construction after making important progress in various vertical areas.As an extremely important data acquisition method and combined with big data,data mining,mobile communication and other technologies that have developed vigorously in recent years,it contributes to building an intelligent society and improving social production efficiency.However,with the rapid growth of application scale and the rapid development of platform construction,the core basic servers,data ingestion and data storage in the IoT application are facing challenges.How to deal with the network communication of a large number of terminal devices with low latency and how to quickly store small and high-frequency heterogeneous data uploaded by terminal devices are urgent problems to be solved.For this reason,the following work has been done in this thesis:1.Data ingestion system:Data ingestion service is essentially a network communication service.After comparing and analyzing various common network communication libraries,Netty is selected to construct a network communication service subroutine.In order to reduce the time delay of data processing,time-consuming operations in network communication services are stripped out and then push the data to Kafka,a streaming message middleware system.Then Kafka consumers are responsible for the subsequent processing of the data.In order to make the data ingestion system work normally,a communication protocol for both terminal and server is designed and the serialization algorithm is introduced to ensure the availability of the communication protocol.The experimental results show that:The data ingestion system work normally and can complete the basic data transmission;The single machine has a high degree of concurrency,and the data processing speed is very fast,the delay is small;The application of Kafka to decouple the program has got the result,improved the thread blocking of network communication programs.2.Data storage system:First of all,the IoT data is a kind of heterogeneous data.It is divided into two types:structured data and unstructured data,and then adopts different storage schemes.Considering the multiple constraints of availability,cost,efficiency,and other factors,the distributed NoSQL database HBase is used to store the structured data,and the lightweight distributed storage system FastDFS is used to store the unstructured data.In order to improve the system efficiency,the row key,region splitting strategy,aggregate operation were optimized in the process of applying HBase.In the application of FastDFS,file relational mapping and caching services are implemented with Redis.The experimental results show that:The write performance of HBase is better than that of MySQL,and various optimization methods achieve results;Using FastDFS to store small files is better than HDFS,and the hot data stored in Redis cache can improve the reading efficiency.