Design And Implementation Of Agricultural E-commerce Big Data Development System Based On Big Data Technology

Agricultural e-commerce big data contains a wealth of characteristic information.It is the data basis for analyzing unique user consumption characteristics,evaluating the service capabilities of e-commerce platforms of different scales,and predicting future agricultural production.It is the direction of agricultural industry corporate management and next year's production planning.An important basis for design.Today,with the large-scale popularization of 5G communication technology and the rapid development of computer technology,e-commerce platforms will inevitably generate huge amounts of data.To this end,this research takes agricultural e-commerce data as the main object,and uses open source tools and coding t o design and implement a set of agricultural e-commerce big data development system.The following six aspects are mainly carried out.(1)Design the hardware and software architecture of the agricultural e-commerce big data development system.According t o the list of major big data technologies that are expected to be used in the big data technology documents published by major companies,and after actual testing of multiple versions of these big data technologies,after determining the technologies and v ersions used in the agricultural e-commerce big data development system,Determine the hardware standard and cluster structure of the server according to the resources used;(2)Research the user behavior data record structure available in the system.Behavioral data is the most important data source of a big data development system,and its structure directly affects all subsequent calculations.According to the public technical documents of domestic e-commerce companies and the experience of internships in enterprises outside the school,it is decided to map the user equipment information and user behavior information in the form of key to the log file in the Json format in the form of Json;(3)Build real-time and offline data acquisition and transmission channels.First,by writing the Spring Boot data preprocessing program,the data is cleaned and divided into real-time and offline according to the needs,and then use the distributed message intermediate and acquisition tool to encode the data to the calculation engine to build the channel;(4)Realize special functions such as early warning of abnormal purchase behavior and daily activity of users.Based on Spark's distributed computing technology,design and code to complete the characteristic functions of real-time computing;(5)Use the log retrieval engine to improve the visual response speed.Provide traditional JDBC and log retrieval access methods for the result data of real-time calculation function,and connect the traditional front-end visualization module and log visualization tool for the two methods respectively,and compare the response speed while displaying the results;(6)Complete the construction of agricultural e-commerce big data warehouse.Based on distributed storage technology,the c onstruction of data warehouses and automatic calculation data modeling are completed through mixed programming of shell and SQL-like languages.The agricultural e-commerce big data development system based on big data and distributed computing technology i ntegrates offline and real-time computing frameworks,and establishes a reliable big data warehouse for agricultural e-commerce enterprises while realizing special functions such as early warning of abnormal purchase behaviors.Provide a solid foundation f or data backup and analysis,which can better provide strong support for decision-making.Because this system uses a low-cost server to meet the functional requirements,it reduces the cost pressure of the big data system on small and medium agricultural e-commerce companies.