The Design And Implementation Of A Data Service System

With the development and popularization of the mobile Internet,more and more users choose network service as the first choice for travel,and the Internet companies collect huge amounts of data.Data in the mobile travel service is stored in a data warehouse to support analysis and decision-making.However,data warehouse based on Hive has such problems as slow query speed,complex operation,unclear metadata and index meaning,and data quality can not be guaranteed.Data service system in order to make data better serve users,allow users to make better use of the data warehouse has been constructed,to provide the following services: use HBase fast query but do not support complex Join features,to provide real-time query data mart online analytical processing(OLAP)services to solve the problem of slow speed Hive user query;data warehouse metadata query and index,so the user does not have to in the report index and ambiguity problems of metadata.At the same time,because the data in the data warehouse needs to be developed on the basis of the business data for a second time,the system provides the following service functions for data warehouse developers: ensuring data quality,maintaining data consistency and accuracy,creating and maintaining metadata,HBase data cube construction,real-time query to support OLAP.To ensure data warehouse developers in their daily work smoothly output data.According to the requirement analysis and function division,the system consists of four modules: the metadata center,the index dictionary,the data service configuration center and the report system.The system uses a four-tier architecture model to reduce the coupling between the various modules of the system,using My SQ as a business database and HBase as a No SQL database,and finally iterative optimization based on test results and user feedback.The data service system has been run on-line in the enterprise,helping ordinary users solve the problems encountered in data warehouse in daily work,and help data warehouse developers to develop data more efficiently and ensure data quality.Let the data be better service for the user.