Research And Application Of Big Data Management Technology Based On NoSQL

With the development of sensor,Internet of things and other technologies, the amount of substation equipment state monitoring data is growing at an exponential rate, the data types are diversified, and gradually form the state monitoring big data of substation equipment. Using big data technology to query state monitoring data of substation equipment can quickly get the target data, then early defects of the substation equipment can be found, and important information for fault diagnosis of substation equipment is provided. The data set that the traditional relational database can handle is relatively small, and it is difficult to store and query for big data. The NoSQL database is a new method for big data storage in big data environment, which is famous for the high scalability and the high performance of data reading and writing.In this paper the research of NoSQL database at home and abroad is analyzed, a data storage and query method for state monitoring data of substation equipment based on NoSQL database is presented.According to the defect that it is difficult to determine reasonable relationship model for semi-structured and unstructured data in the relational databases, and the real-time requirements of high concurrent processing is hard to meet. Considering the characteristics of the state monitoring data, the HBase database is selected to store state monitoring data, and the rowkey and the column family are optimized designed to query faster by rowkey. The secondary index based on coprocessor is replied to solve the problem of low efficiency of the non-primary key data query.To solve the performance bottleneck of state monitoring data of substation equipment encountered when based on traditional relational database system, we use NoSQL as the state monitoring data of substation equipment storage engine,and study the storage and query problem on this basis. The experiments on storage, read and write performance test of the state monitoring data of substation equipment is did.The experimental results show that, This method compared with conventional storage methods, storage performance improves about two to three times, The query method based on coprocessor is two times quicker than that based on IHBase. It can satisfy substation equipment state monitoring data mass storage and quick query needing in the big data environment.