Design And Implementation Of Agricultural Product E-commerce Data Warehouse Analysis And Evaluation System Based On Hive On Spark

With the advent of the era of big data,E-commerce platforms have successively entered a data value-driven operation mode.For agricultural product E-commerce platforms,the quality of suppliers is related to the competitiveness even the lifeline of the platforms.Therefore,the use of data to evaluate and select suppliers is an urgent requirement for many companies.However,small and medium-sized agricultural product E-commerce platforms are constrained by insufficient data and high cost to independently build and maintain data analysis systems.As a result,these platforms are unable to enjoy this wave of data dividends.Based on this,in view of the excellent data integration and analysis capabilities of the data warehouse,this paper is committed to apply and implement an existing theoretical model to supplier evaluation,which can design and implement a warehouse system that can evaluate and analyze agricultural product suppliers.The system integrates the data of the relevant small and medium-sized e-commerce platforms and the public data of the network,to provide targeted services and help them evaluate and select suppliers.The Hive on Spark technology framework selected by the system has excellent big data processing and analysis capabilities,which can analyze multidimensional complex data and efficiently obtain calculation results.Therefore,this paper focuses on how to establish a set of data warehouse models and architectures and use these big data tools to analyze and process data.The specific work includes the following three aspects:1.Based on demand analysis,establish the theme of the supplier portrait of the data warehouse.Based on the theme and data sources,the system is divided into public analysis modules and customized analysis modules.The dimensions and models of each module are designed to determine the data to be collected.2.Follow the data flow to divides the system into Data Collection part,Data Analysis part,and Visualization part.The first two parts contain the entire process of ETL.The STORAGE layer,the ODS layer,the DWD layer,the DWS layer and the ADS layer are designed according to the data warehouse layering theory.The one has a five-layer architecture.The layered architecture guarantees the stability of the data channel and system.3.The design concept was implemented by the Hive on Spark technology framework,and the system environment and function tests were performed to verify the correctness and usability of the solution and to prove that the system can meet the requirements.This paper implements the integration and analysis of supplier-related public data of the entire network and enterprise data in response to the problem of the lack of supplier evaluation and analysis capacities for small and medium-sized e-commerce companies,which helps solve the pain points of the industry.