Research On Metadata-based Early Warning Data Integration And Sharing Technology

With the diversification of information acquisition methods and the growth of data volume,extensive attention has been widely paid to the construction of data management in the field of early-warning in China.The main problems currently facing include: First,it’s difficult for traditional data storage methods to realize efficient and secure storage of massive amounts of heterogeneous data resources.Second,early-warning information systems established by various departments have formed barriers to data management and sharing.Last but not the least,tons of data resources accumulated in the long-term development makes it difficult to integrate them efficiently.Therefore,it’s of great necessity to establish a basic platform for unified storage,management and utilization of data resources.Based on the practical engineering demands of the joint training unit,this thesis systematically studies the data integration and sharing technology in the field of early-warning.The main research results are shown as follows: 1.The storage technology of massive early-warning data is studied.By analyzing the drawbacks of the existing structured database in the face of massive data,the HBase database is expanded and established based on the original My SQL and Oracle databases,realizing the storage of massive data such as the track;the visual management interface of the HBase database is developed,realizing the migration of data from traditional database to the HBase database with the help of Sqoop,and the secondary index of HBase is constructed by using Solr,so as to achieve non-primary key query and combination query.These improve the system’s storage capacity to deal with massive data,and improve the query speed of massive data.2.The metadata description of the database is studied.A metadata-based query and retrieval method for heterogeneous databases is put forward.By collecting metadata in the database and establishing a corresponding metadatabase,the differences in the underlying data source can be removed,providing users with transparent access to data and realizing data integration.This method is more flexible with which the database can be added or deleted at any time,bearing greater advantages in engineering applications.3.The integration and sharing mode of early-warning data is studied.By studying the existing data integration and sharing model,learning service-oriented architecture ideas and Web Service technology,this paper achieves the encapsulation and integration of existing data system services,and achieves cross-platform,cross-system,cross-language data storage and integration.At the same time,user rights are managed hierarchically,which guarantees the security of data to a certain extent.4.Based on the above studies,the prototype of the early-warning data management system is designed and developed with the characteristics of loose coupling and expandability,which can store massive structured and unstructured data,realizing the integrated management of early-warning data,and providing users with underlying transparent storage,browsing,and query services.The results obtained in this study are in line with the development of current new technologies,meeting the requirements for distributed heterogeneous data storage management in the systematic construction of early-warning system,which can provide technical support for engineering applications.