Research Of Analysis And Processing Technology On Large-Scale Data From Commercial And Logistics

In the background of cold chain logistics Large-scale data long-term storage and real-time process applications,the big data from the production and transport process was selected as research object in order to reduce the information management and improve the decision-making of logistics.The theory models and key technologies of logistics Large-scale data were analyzed from low-cost and high-performance storage,secure data access and computing aspects.Data distributed storage technology is the foundation of research,and focuses on solving high-performance storage and access of logistics big data in multi-level hybrid storage system.The prediction model of hot data in the multi-level hybrid storage system was built in order to implement data migration.The distributed security storage model is proposed to integrate data management and security control planes based on multi-level hybrid storage system.Then we focus on the computing Large-scale logistics data with the heterogeneous clusters based on distributed storage system.The goals are to logistics recommendation model and goods sharing transportation model based on Large-scale data processing.In the research process,we emphasized the characteristics of data access rules in logistics industry.The distributed hybrid storage system and typical parallelization algorithms were designed by combining the logistics information management,distributed computing theory and simulation technology.The main research work and innovation are as follows:(1)Based on the analysis of the rules and characteristics of data access in the logistics industry,a multi-layer hybrid storage model was designed.In order to effectively improve the performance of distributed multi-layer hybrid storage system,the key issue is the prediction model that can distinguish the hot and cold data.The accessing features of data in the hybrid storage system were extracted.Then the wavelet neural network was used to build predictive classification model.The prediction model was trained by historical accessing data.Finally,the prediction model that is the foundation of data migration was used to implement the classification of unknown accessing behaviors.(2)A novel transparent security data model was proposed for the application of multi-tenants distributed scenery based on the multi-level hybrid storage system.The security data accessing management architecture was designed that can be used in cloud storage environment in order to integrate the data accessing management and security control planes.The data security management algorithms were proposed to deal with the authentication and data synchronization process.Then high-performance and security storage system can be achieved based on the data security model and algorithms.The simulation results show that the purpose of data security management is achieved while the performance loss is controlled within acceptable range.(3)The personalized recommendation of logistics distribution was built based on big data processing technology,which adopts the label optimization and enhanced learning.The parallel personalized recommendation algorithms were designed to solve the recommendation accuracy and computing latency.A novel label-optimized matrix decomposition recommendation algorithm was proposed according to the social network context information.This recommendation model adopts the enhance learning on the basis of the tripartite graph network to optimize the tag sorting.Then the preference matrix between users and resources was constructed after deleting the junk tags.Finally the recommendation results were generated by matrix decomposing.The scheme has good recommendation accuracy and is suitable for porting to heterogeneous clusters.The two logistics delivery recommendation algorithms were realized in the logistics big data analysis platform to solve the performance bottlenecks.(4)The shared transport models were proposed on the basis of real-time Large-scale data processing and query technologies.The non-linear spatio-temporal mapping problem of large-scale parcel sharing scheduling was solved by the similar trajectory matching and optimized spatio-temporal index technologies.The full-cargo shared transport model was designed by improving the maximum similarity trajectory matching method.And the parallel scheduling algorithm suitable for GPU heterogeneous clusters based on this model.The incremental cargo-sharing transport model was built by using the optimized distributed spatial-temporal index and massive cargo real-time query algorithm.And the corresponding parallel processing algorithms were implemented.The experimental results show that the two shared transport models achieve good performance and accuracy.