Design And Implementation Of A Hybrid Storage System For Big Data On The Internet Of Vehicles

Since the MOT of China included the construction of the Io V in the national major special projects in 2010,all parts of society began to pay attention to promoting and participating in the deployment of the Io V in the urban transportation system.In this context,I participated in the development of the Urban Vehicle Intelligent Prevention and Control System(UVIPCS).Against the background of the UVIPCS,the paper focuses on the storage problems of the Io V data in the UVIPCS.In order to complete the task of storing the Io V data,and meet the writing requirements of high concurrency and high throughput and highly real-time query requirements,the paper designs and implements a hybrid storage system based on the reading and writing features of the Io V data.With reliability,availability and scalability of the underlying engines,the storage system provides a robust data service for the UVIPCS.The work of this paper is mainly composed of the following three parts:1.System demand analysis.It is determined that the functional boundary of the center platform of the UVIPCS by the hierarchical domain characteristics of the Io V,and its task is to deploy the functions of the upper five layers in the Io V.The two main data streams in the platform are defined as the data production chain and the information processing chain.Following the data production chain and the information processing chain,interactions and collaborations between the hybrid storage system and other components of the platform are studied to extract the functional requirements of the hybrid storage system.We summarize the reading and writing features of the Io V data from the business functions and the production environment of the UVIPCS,and divide these data into three categories: perception data,basic information and statistical data.It analyzes the non-functional requirements of the hybrid storage system at the same time.According to the above requirements and the storage technology,HBase,My SQL and Redis are selected as the underlying storage engines of the hybrid storage system.2.System design and implementation.The hybrid storage system is composed of data objects,operation interfaces,connection pools and physical engines,which is divided into the perceptive data storage subsystem,the basic information storage subsystem and the statistical data storage subsystem.With the powerful writing ability and the distributed management logic of HBase,highly concurrent reading and writing of the perceptive data can be realized.This paper provides the detailed design of the storage mode of the perceptive data,the operation interfaces,the connection pool of HBase,the program for maintaining indexes,the optimized parameters of the cluster configuration and the method of estimating the storage scale in the cluster.With the help of the b-tree indexes and the transaction processing strategies in My SQL,it manages the basic information with complex relationships.Analyze the problems of the statistical data with the actual application scenarios,such as write conflicts and query delay.Take advantage of the high performance of Redis,it records the small amount of statistical data with lots of reading and writing.3.Testing and summary.Check the node usage in the hybrid storage system,make statistics on the storage throughput,and detect the information feedback delay of the platform.The online tests show that the cluster works well,the daily throughput exceeds 400 million records,and the information feedback delay is limited in 7 seconds.The hybrid storage system can provide stable,reliable,scalable,highly concurrent,highly real-time,and high-throughput data storage services for the UVIPCS.