Research And Implementation Of Logistics Data Resource Management System

In the development of informatization,many companies have generated data around various businesses.In order to be able to discover the value of these data,it is necessary to process and analyze these data through some technical means.The data volume of a special traditional enterprise logistics department is increasing rapidly during the informatization process.The data types are rich and the business systems are numerous.Therefore,a high-performance data resource management system is needed to store,manage,and analyze these data.This paper analyzes the problem of complex data and complicated business encountered by enterprises in the process of developing logistics informatization,and puts forward a plan to construct a logistics data resource management system to solve this problem.First of all,the key technologies and methods required in the establishment of a logistics data resource management system were studied and analyzed,including the distributed file system and MapReduce computing framework based on the Hadoop platform,Hive processing tools,and Spark computing for optimizing data processing effects.engine.According to the business requirements,the system architecture is designed,and three modules are designed and implemented:data integration module,data storage analysis module and data visualization module,which is mainly based on MapReduce computing framework and combined with Sqoop technology to extract and store source data,Then use HiveQL to query the data and display it.The storage and analysis module of the data is the focus of this research.Therefore,this paper proposes a data processing method based on the Spark computing engine to optimize the system.By building a MapReduce and Spark dual engine To improve data processing efficiency.Finally,two types of experiments are designed.One is to perform query analysis on part of the data,and the results show that the system can meet the actual needs of the data query analysis of the logistics department.The other type of experiment compares the MapReduce-based computing framework and the Spark cluster The efficiency of data processing in the two environments verifies the feasibility of the system optimization scheme.