Research On Big Data Storage Technology For The Portrait Of Agricultural Enterprises

The emergence of the portrait system is a sign that the Internet development model has changed from rapid expansion to deep cultivation.The enterprise portrait system with agricultural enterprises as the research subject can play a guiding role in the development of agricultural enterprises.Data storage work,as an indispensable part of a system,also plays a pivotal role in the portrait system for agricultural enterprises(hereinafter referred to as the “agricultural enterprise portrait system”).However,in the field of data storage of agricultural enterprise portrait system,there are currently two problems to be solved: 1)There is no standardized data storage solution in this field,which can shield the underlying heterogeneous data source and shield the intermediate operations of heterogeneous data sources.Finally,the solution can be implemented in the same type of system in the field.2)There is no ripe data cache processing solution in this field,which can take into account the improvement of system access performance,data cache hit rate,and security risks of cache usage.The research in this paper is driven by the above two problems.The main work can be summarized as follows:(1)Propose a general data storage model for agricultural enterprise portrait systems.The storage model consists of general operator expression,analytic mapper,implementation set of operators and heterogeneous storage tools.The definition of general operator expression is based on the principle of high abstraction,and can adapt to various business scenarios of the agricultural enterprise portrait system,and has the characteristics of stability and compatibility.The implementation set of operators completes the storage from heterogeneous data sources to heterogeneous storage tools.The analytic mapper is responsible for parsing the operation operator expression and mapping the contents of the expression to the corresponding operator implementation.All work together to enable the storage model to shield the underlying complex storage work,shield the upper layer of complex business scenarios,reduce system development and maintenance costs,and improve system efficiency.While solving the data storage problem of the system,it also provides a solution for the data storage work of the same type of system in the field.(2)Propose a data caching solution for the agricultural enterprise portrait system.The solution consists of establishment of a three-tier cache architecture,improvement of the cache replacement strategy,selection of the cache internal storage structure and prevention of cache security risks.Multi-level redundant cache architecture,which can intercept more requests before they reach the database.The handling of security risks can effectively avoid cache failures such as cache invalidation,cache penetration,and cache avalanche.Based on this,an improved C-GDSF(Characteristic Greedy Dual Size Frequency)cache replacement strategy is proposed.Based on the cubic exponential smoothing algorithm,the characteristic heat of the data is predicted,and the final data value is calculated by combining the dimensions of cache value,data size and access frequency,which effectively improves the cache hit rate.Finally,the internal storage structure of cache is improved to improve the efficiency of cache access and update.The whole set of caching solution not only improves the efficiency and security of the cache itself,but also ensures the smooth operation of the storage system.At a deeper level,it also provides a solution for data caching of similar systems in the same field.(3)Based on the “Research On Key Technologies Of Big Data-driven Portrait Of Agricultural Enterprises” proposed by the Chinese Academy of Agricultural Sciences,the general data storage model was first implemented,and relevant experiments were carried out on this basis.The experimental results show that after applying the storage model,the repeated code rate of storage service is reduced by 28.1%,and the average reuse times of the operator is up to 2.6 times.The system response speed is basically the same as when the storage model is not used,and there is no significant drop due to the use of the storage model.The caching solution is then applied to the storage model and the relevant experiments are designed again.The experimental results show that after the system uses the cache,the response time of the request response is greatly reduced,and the system performance is greatly improved.Using the C-GDSF cache replacement strategy can achieve a cache hit rate of 83.1%.Under the same conditions,it is 3.4% higher than the volatile-lru strategy,4.2% higher than the volatile-lfu strategy,and 6.7% higher than the volatile-random strategy.Optimization of the internal storage structure of the cache reduces the time complexity of the cache usage.