E-commerce Supply And Demand Analysis System Based On Big Data

Supply and demand always have a gap in e-commerce platform. The supply and demand gap refers to the seller's selling goods doesn't match the buyer's needs. In e-commerce platform, the buy's needs could be found in their shopping behaviors, such as the searching keyword behavior, the click behavior and the searching constraints. How to get the buyer's needs and provide the information to seller is an important issue in e-commerce platform.To address problems mentioned above, this paper base on an internationally renowned e-commerce company project as the background to design and implementation an e-commerce supply and demand analysis system. To get the real demand in TB-level data base on user behavior, this paper analysis and research the Hive-based data storage technology, rule-based data cleaning techniques and the Map Reduce-based modeling technology and other related technologies. In the actual test on the company's e-commerce platform with running 3TB data in three months, the feasibility and effectiveness of the system was verified.This paper has the following content:(1) For the problem of extracting and clean up the large-scale user behavior data, proposed to use the ETL(Extract, transformation and load) technology and a rule-based data cleaning on HADOOP, and use Hive data warehouse for data storage. Solve the problem of large-scale data for data processing.(2) For the problem of get buyer needs in large-scale behavior data in data mining and information extraction. In the T-level data, design and implement UDF(user define function) on Hive for different applications scenarios.(3) Depending on the business needs, design the index to describe the relationship between supply and demand and then use them to modeling. Implementation the model on Hadoop Map Reduce, meet the requirements of both performance and efficiency in large-scale data.