Realization And Optimization Of Dairy Traceability System Based On Hadoop/Hive

In response to the calling about government's attention of strengthening dairy traceability,more and more dairy enterprises have participated in the project of traceability,which brings pressure on storage and process of data.In order to solve the problem above,we take the fellowing steps.Firstly,we analyse the demand of dairy traceability and conclude that it is urgent for dairy traceability.Secondly,we introduce the Hadoop/Hive technology to design and realize the traceability system which can expand the system storage capacity,deal with large-scale data performance and improve the speed of the system traceability.Finally,we propose a GA-PSDD-1(Parallel SDD-1 Algorithm Based on Genetic Algorithm)Algorithm to optimize the traceability query and test the performance of large-scale data import,query and interactivity.The main contents of this paper are as follows:(1)Design and implement dairy traceability system on the base of Hadoop/Hive technology.Firstly,we analyze supply chain of Weigang dairy company and its key traceability information.Secondly,we build a dairy traceability framework and develop traceability system application architecture based on Hadoop/Hive to solve the problems of slow large-scale data uploading,huge data storage pressure and slow data processing in traditional traceability system.Finally,we design the database and function modules of the traceability system and realize the dairy traceability system based on Hadoop/Hive.(2)Optimize the performance of the traceability query about dairy products.A GA-PSDD-1 algorithm is proposed and implemented to solve the low speed problem of traceability by combined with dairy traceability system.In order to compare the performance of strategy generation under the condition of actual data about the traceability process,we carry out a test between GA-PSDD-1 algorithm and the traditional SDD-1 algorithm.The experimental results show that the cost of GA-PSDD-1 algorithm generation strategy is reduced by about 26.64%.We further test their performances by applying both algorithms in the application of dairy products.The results show that the GA-PSDD-1 algorithm takes an average of 600.66ms shorter than the traditional SDD-1 algorithm,and the query speed is increased by 24.48%.(3)Test the overall performance of dairy traceability system.Hadoop/Hive technology and GA-PSDD-1 algorithm are applied to the dairy traceability system for large-scale data import,query and interaction of traceability system.We test the performance of the dairy traceability system before and after optimization.Experiment with 30 million records as test data,the results show that the average speed of induction is 88.89%,the average speed of inquiry is 27.66%,and the average speed of interaction is 60.32%.In addition,after combined with GA-PSDD-1 algorithm and traceability query,the system traceability speed increase about 48.35%which is a remarkable result.