Research On Distributed Storage Of Lattice Structure Data

Data cube lattices and concept lattices are two kinds of important models in data warehousing, data mining and knowledge discovery etc. fields. Their instances are lattice structured. It’s still a big challenge on how to storage and query massive lattice structured data. To address this issue, lattice structured data are seen as graph data and its intrinsic statistics and laws are firstly studied. Then the model and mechanism are discussed. Based on these hypotheses, partitioning, storing and querying across multiple nodes are designed. Based on the random division, divided by layer, graph partitioning, the massive lattice structured data is partitioned. The storage and query of massive lattice structured data is completed based on Spark that is a distributed memory computing framework.The main contents are as follows:(1) partitioning massive lattice structured data, in order to storage and query massive lattice structured data effective, this paper puts forward three kinds of partition method that is random division, divided by layer, graph partitioning depending on the statistical characteristics and laws of the lattice structure data;(2) storage and query massive lattice structured data, Based on the results of lattice structured data division and considering the cost of communication and load balancing, this paper design distributed storage model and distributed query algorithms of massive lattice structure of the data to achieve efficient query and analysis of the massive lattice structure of the data;(3) Building a distributed computing framework based on Spark which is an in-memory computing structure to implement distributed queries prototype, also complete experimental verification and analysis.