The Design And Implementation Of The Big Data Processing System For Telematics

As a branch of the Internet of things,the telematics focuses on the interoperability of vehicle systems and real-time collection and processing of on-board sensor data.Due to the special security requirements of the vehicle,the system needs to do the hard or near real-time processing of the important on-board sensing data,otherwise it will bring fatal security consequences.Therefore,the telematics in the field of traditional Hadoop represented the massive distributed batch data base,not only for the massive data throughput capacity has strict requirements,and high concurrent real-time processing of massive data has strict requirements,and put forward the special requirements of real-time data processing,consistency and robustness.To resolve the present situation in the field of vehicle networking and problems,the thesis adopted the existing technology based on the concurrent processing of multi thread Netty,non-blocking,integrated Hadoop,Spark and Storm technology,and realized the huge high concurrent data real-time processing and load balancing.The thesis extracted a more general real-time processing and batch processing framework for high concurrent data based on a general framework.The main work completed in this thesis includes:(1)Realize an overall architecture design of integrated data processing framework for telematics and build the integrated intelligent processing model for storage,processing and computing.(2)In order to meet the demand of mass,high concurrency and real-time processing of telematics,implement a hybrid storage and intelligent adaptive optimization of telematics for structured and unstructured data.(3)Improve the existing data processing framework,and realize the integration of intelligent computing framework,dynamic adaptation,batch processing and real-time processing,and the large data hybrid processing engine.The multi granularity intelligent scheduling optimization greatly promoted the utilization rate and run speed of the resources scheduling for storage and computing.The design of telematics has been applied in many vehicle networking project,which has stable carrying tens of millions of vehicles online real-time data acquisition and real-time monitoring and early warning,and has the ability of dynamic additional vehicles and dynamic expansion server.