Research And Visualization Application Of Crop Data Storage System Based On Hadoop

As a basic strategic resource,big data is a very important development research direction.The 19 th National Congress of my country also pointed out the development of big data in agriculture and rural areas to accelerate the research of key technologies of big data.The technology of big data includes data search,collection and preprocessing,data storage and management,as well as data analysis knowledge mining or client applications,etc.,especially in the storage application link,the use of big data technology to store data can effectively and quickly Analyzing and using the massive data obtained,this article mainly studies big data technology,builds a storage system based on Hadoop technology and visualizes the collected crop data.This article has three main contents.One is to study the big data technology ecosystem with the Hadoop platform as the core,including database types and deployment methods,and data visualization technologies.The second is to carry out in-depth research on data compression and system load balancing for the purpose of improving system IO operation performance and data transmission processing speed,reducing network load and memory space after researching big data storage systems,and using collected data Experimental tests on crop data show that the Huffman compression algorithm has the best compression effect and the effectiveness of load balancing for system optimization.The third is to design and implement a big data storage system based on the Hadoop technology ecosystem.The system design is based on the direction of data,starting with data collection and sending,Nginx load balancing and forwarding multiple Tomcat application services,two-layer Flume load balancing to collect data logs,Kakfa message queue cluster to transmit data,and then a big data distributed server cluster based on Hadoop.,Obtain the source data transmitted by Kafka,on the one hand,write it into the HBase database with HDFS as the underlying storage,on the other hand,use the Spark machine learning library to analyze and process the collected crop data,and write the results into the My SQL database.Finally,use the B/S software model to develop crop data visualization applications,use Java to develop back-end application services(Springboot,Mybatis framework),and develop front-end web applications based on Java Script language(Vue,Echarts framework).