Design And Implementation Of Knowledge Graph Storage Access System Based On Big Data Platform

The Knowledge Graph applied entity and relationship needs scalable storage schema and distributed parallel queries in search,question and answer(Q&A)and analysis scenarios.In this thesis,we design a distributed-aggregated storage schema based on Big Table model,which has the characteristic of load-balanced storage and local node clustered storage and we design a distributed parallel query engine based on Group-By mode by scanning query tree in parallel.The experiments show that the storage schema and query engines we design have good horizontal scalability.The specific work is summarized as follows:(1)Distributed-aggregated storage schema.We perform random prefix and pre-partition operations on row-by-row stored entity sets based on Big Table model.After these operations,we divide and store the entities evenly to achieve load-balancd.The random prefix can also distribute the same type of entity evenly to the node storage and aggregate them by the entity category on a single node.(2)Distributed parallel query engines: We design two distributed parallel query engines that use different schemes based on the distributed-aggregated storage model: Memory Iteration Query Engine(MIQE)and Inverted Index Query Engine(IIQE).MIQE uses a distributed memory iteration technique to query entities that are represented in memory abstract set based on filter and join operations.IIQE combines inverted indexes with coprocessors in the cluster to query entities in parallel indexed queries.Both of the above query engines we design can improve the reading performance of the Knowledge Graph by reducing disk I/O and parallel query methods.(3)Prototype system implementation and performance verification: We design and implement a Knowledge Graph storage access system based on big data platform according to the above research.Experiments show that the Knowledge Graph storage access system based on distributed-aggregated storage schema and distributed parallel query engine has good horizontal scalability.