Design And Implementation Of A Hybrid Storage System Based On SQL And NoSQL

In this era of Big Data,people's lives are constantly convenient and rich at the same time,but also face the challenge of how to handle and store massive amounts of data.In the field of transportation and logistics,faced with the rapid growth of traffic monitoring data in type and volume,the traditional relational database solutions although has the advantages of supporting in sequential scan and complex query on data,and powerful function of index equipped.But the lack of big data processing performance and scalability has become his bottleneck.The NoSQL database as a new storage solution,has the characteristics of a highly scalable,highly available.Therefore,how to combine the advantages of both database so that it can fulfill the massive data storage and mining,is of great significance.Aiming at the problems above,this paper designs and implements a hybrid storage system based on the relational database MySQL and NoSQL database HBase.Firstly,through the deep research on the HBase query process,propose a query optimization of hybrid architecture based on MySQL econdary index.The method is that,by indexing HBase tables on MySQL,reduces the number of queries accessing HBase.Meanwhile,taking advantage of the MySQL supporting SQL language and fast query feature,the method improves the performance of HBase data query and retrieval.Then based on the above findings and actual scenarios,this paper design and implement a hybrid storage system for massive traffic monitoring data.In addition to the realization of storage and query functions,the system also uses message queue Kafka and real time stream data processing tool Storm to achieve data preprocessing.At the same time,based on the combination of Sequence files,the paper solve the problem of HDFS small file storage.At last,the paper built experiment platform,the data writing and query of the hybrid storage system is written for a detailed comparison of the performance test.Experimental results show that the hybrid storage system proposed in this paper not only has high scalability and high availability,but also provided with the characteristics of supporting SQL language and better performance in reading and writing,providing an alternative solution for massive data storage.It has certain engineering value and practical significance.