Researche On Key Technology Of Hadoop-based Agricultural Information Resource Management

At present, our traditional agriculture is transforming into modern agriculture, digital agriculture and information agriculture step by step. Internet has become the main carrier of agricultural information resources, and agricultural data belongs to typical large data. In such a background, realizing the effective management of vast agricultural information resources has become a key problem of massive agriculture information integrated application.This thesis is to realize the effective management of agricultural information resources for the remarkable size difference, analyst, and transplant and optimize the cloud storage architecture of Hadoop and develop the agricultural information resources management system based on Hadoop. The main research results are:(1)After thorough analysis of Hadoop cloud storage architecture, regarding Hadoop distribute file system and Map Reduce as the core, I analyze the core subproject of the Hadoop mechanism from the angle of internal mechanism. Furthermore, according to the in-depth study of the Linux system and the learning of Hadoop cluster in cloud computing advantage, we build Hadoop cluster on the Linux platform.(2)Through the analysis of the shortage of HDFS in small file storage, compared to the advantages and disadvantages of the existing Hadoop archive and sequence file, I proposed optimization algorithms based on the merge file and designed small files processing module. Aiming at the uncertainty of the frequency of access to agricultural information resources, on the basis of optimization for small file, this paper proposed a kind of control algorithm based on a copy of the access frequency dynamic management FDMDR which Add file access frequency constraint basing on DMDR algorithm, copy resources dynamic management.(3)Based on the optimized Hadoop cluster, we construct agricultural information resource. Using struts2, spring and hibernate framework we develop the right management module in the agricultural information resources management software, file management module, interface management module and statistical analysis module. Among these, the permission management realizes the operations of increase, modification, deleting and so on and the users are divided into administrators, regional administrators and ordinary users. Document management realizes such file functions of uploading, downloading, deleting and browsing and provides the Web Service interface and HDFS interface management; Statistical analysis module uses the classification statistics, regional statistics and statistical methods for statistical analysis. The platform achieves effective management of agricultural information resources.Facing the actual needs of large number of small files in the management of agricultural resources, the development of agricultural information resource management system was completed in this paper based on the Hadoop platform building, parameter optimization, the algorithm improved. Research results show that the system can achieve file, images and video, The results shows that the system achieved the combination of the loose organization and unified management agricultural information resource, include the file, pictures, video and so on, this system has the characteristics of function stability, strong extensibility.