Research Of Massive Data Distributed Storage And Processing System In Internet Of Vehicles

With the explosion of vehicles,the concept of Internet of Vehicles(IoV)has drawn increasing concern among academics.The IoV refers to extracting and processing information from massive data which are gathering from vehicles and their surrounds.With the development of the Internet of Vehicle,the massive data which collected from some sensors for the equipment monitoring requires higher performance of data storage and inquiry.So an integrated IoV system which is reliable on data storage and efficient on data processing is indispensable.The paper designs a distributed storage and processing system depending on Hadoop,HBase and Spark to store and process IoV data.The system can not only overcome the single point of failure problem,guarantee the reliable data storage and meet the efficient computing requirements,but also can realize the clustering algorithm of parallel calculation.The contributions of this paper are as follows:Firstly,the paper studies the characteristics of the IoV data,the study finds that IoV data has the some characteristics of the big data,such as large amount of data and fast processing.So the paper combines the big data technologies,proposing a distributed storage and parallel processing system depending on Hadoop,HBase and Spark to store and process the IoV data.And the construction of the system environment and deployment are completed.The study on the system is focus on the data storage layer and the data processing layer.The HDFS and the HBase are used for the data storage layer that resolves the problem of single point of failure.The Spark parallel computing framework is deployed as the data processing layer that benefits the efficient analysis of massive IoV data.A variety of big data technologies are combined together,constituting a complete,reliable and stable IoV data distributed storage system.Secondly,according to the user requirements,the paper combines the Spark parallel programming model with the clustering algorithms,realizing the parallel clustering algorithm.The anomalous data is filtered out before the data processing and analysis via Spark computing.Then the parallelized K-means algorithm and DBSCAN algorithm are realized on Spark.The experimental data comes from practical taxi data in Fuzhou and the clustering results are shown in Fuzhou road network map.The experimental results can facilitate the public to take a taxi,make it possible for the taxi driver to shift or have a rest.Finally,the thorough tests are done to verify the performance of stability,distributed storage and parallel computing on this system,which use the taxi data in Fuzhou.The results show that the system can satisfy the requirements of the reliable data storage,the load balance and stable data processing.