Research On The Key Technolotgies Of Real-Time And Fast Query Of Traffic Security Big Data

With the acceleration of urbanization and the rapid increase of urban population,the traffic and security problems are becoming more and more serious.Intelligent Transportation System(ITS)can analyze traffic conditions and urban security,and help control the city's traffic and security with large amount of data collected by city surveillance centers.HBase can be used to store large-scale vehicle information,but can not get fast retrieval in real-time.This paper is mainly aimed at running query of real-time records stored in HBase at faster pace,and proposes an optimization method to reduce the overhead of interconnect networks and magnetic disk read operation.Firstly,this paper creates the secondary index in HBase to reduce the number of scanning car records during query and calculates the secondary index to obtain the car records that match the query conditions.The experiment results show that HBase with the secondary index can reduce the response time of the first record by 45 times when compared to the non-secondary-index HBase.Secondly,the time of transferring the secondary index from the storage node to client through interconnect network is 20%of the total query time.Based on the near data processing idea,this paper constructs an acceleration framework through the HBase coprocessor to move the acquisition and calculation of the secondary index to the storage node so as to reduce the overhead of network transmission.After integrating the proposed acceleration framework,the cluster's network overhead is reduced by 12 times,and the retrieval response time achieves a 1.4 speedup.Finally,during query we found that the time for the storage node to read the secondary index from the magnetic disk accounted for 70%of the total time.According to the cardinality of the secondary index,we reduce the read time of the magnetic disk by compressing the secondary index and computing the compressed index in the HBase coprocessor.The experimental results show that the magnetic disk read time and query response time reduce by 73%and 80%,respectively.