Based On Predicate Classification The Hive Data Warehouse Of The Design And Implementation

Big data Hive of traditional data warehouse design mainly depends on the designer’s experience,the embodiment of the lack of previous user query analysis,it is difficult to meet the needs of the user personalized data query analysis.Thesis by analyzing the user query past conditions(predicate)analysis,to study and put forward the big data environment Hive data warehouse design method based on predicate classification,in order to improve the data warehouse to the user personalized data query the validity of the analysis.The main work of this thesis:Firstly,this thesis proposes a row storage mode of Hive data warehouse based on predicate classification in big data environment.Based on the association frequency between the predicates,this pattern constructs an equivalence relation on the predicate set.Through the classification of predicates and the selection of key predicates,the corresponding real view is generated,and the user-oriented and personalized row data organization pattern of data warehouse is formed.Secondly,this thesis proposes a Hive data warehouse column storage mode based on predicate classification in big data environment.Based on the association frequency of attributes in predicates,this pattern uses traditional association rule mining algorithm to form column cluster and builds a customizable data organization pattern for data warehouse columns.Thirdly,the architecture of Hive data warehouse system based on predicate classification is designed,and the process and flow of ETL processing and related data organization are analyzed and discussed.Fourthly,based on the data of a foreign-related enterprise,the table of predicate-classified hive data warehouse is imported into the relational database.Combined with the framework springboot + mybatisplus and the front-end display framework Vue,the big data application system is realized and user management and data visualization are completed.