Application Of Distributed Technology In Visualization System Of Fermentation

The main task of this paper is to study a variety of open source distributed technologies to form a parallel framework and apply it to the monitoring system of the fermentation room visualization system to deal with the massive data brought by project scale expansion and real-time online measurement hardware system,and to satisfy the fermentation room.The performance requirements of the visualization system for largescale iterative calculations of real-time computing and machine learning.Through the research of the most advanced distributed theory and the operating principle of open source projects,and based on this,we have optimized and expanded and designed a distributed system solution that is parallel,real-time,and consistent.The details are as follows:(1)Analyze the visualization system of the fermentation room in the traditional grain and oil industry.Explain the key points encountered in the process of implementation and several key technical difficulties that need to be solved in the design of the software system.That is,it can handle the pressure of massive data,and it can be stable and rapidly large-scale.Iterative calculations,real-time calculations,and data skew issues.Based on these requirements,we analyze the traditional distributed technology MapReduce and come to the conclusion that the traditional distributed technology can not meet the design requirements,and summarize the reasons;(2)Aiming at the design requirements of the visualization system of the fermentation room and the defects of the traditional distributed technology,a parallel distributed framework solution is proposed,which includes the parallelization of resource scheduling,parallel computing,parallelization of storage,parallelization of resource scheduling.By applying the Yarn resource scheduler based on the DRF algorithm,the distributed framework can accommodate a wider variety of open source technologies,making it possible to introduce external frameworks to resolve system internal defects;computing parallelization solves traditional distributed technologies by applying Spark’s RDD technology.It is difficult to carry out large-scale iterative problems;concurrent optimization of both storage and computation simultaneously mitigates the risk of data skew.(3)The application of Storm streaming computing technology in a parallel distributed framework enables the distributed framework to meet the real-time rapid calculation requirements of the continuous process link of the fermentation room visualization system.Based on this,the Off Line,NearLine,and Online are completed.Integrated Parallel Distributed Framework Design.And for the consistency risk of the parallelized distributed framework itself,Zookeeper is used as a distributed cluster coordination system to manage the cluster.(4)Based on the parallelized distributed framework and neural network principle,the artificial neural network algorithm is implemented and compared from the two aspects of real-time computing and offline computing,as a supplement to the distributed machine learning MLlib library,making More complete,you can face the various algorithmic requirements of the fermentation room visualization.(5)Set up a three-node distributed simulation test platform,build and debug the entire parallelized distributed framework,and test the performance of the system from off-line and real-time calculations to prove the parallel distributed system.In terms of memory usage and data processing speed,it can meet the design requirements of the fermentation room visualization system.