Design And Research Of Big Data Machine Software Middleware

Big Data Machine Software Middleware is one of the important research directions in the field of big data and cloud computing.The unique product form of Big Data Machine solves the problem of continuous expansion of infrastructure in the era of big data.The design and research of Big Data Machine Software Middleware can help meet the storage and computing requirements of massive data in the upper-layer applications,help the system to fully utilize the hardware performance,improve the system resource utilization,reduce the overall cost,and promote the whole big The sustainable development of the data-integrated ecosystem.The research on Big Data Machine Software Middleware mainly includes the following two points:An adaptive container resource scheduling algorithm based on queuing theory and particle swarm optimization is proposed.in the traditional data center or the current Big Data Machine environment,a growing number of applications generate a large number of tasks,and the ever-increasing task requests and the limited resources in the system environment form a sharp contradiction.Effectively improving the resource utilization of the Big Data Machine is the main means to solve this contradiction.This paper proposes an adaptive container resource scheduling algorithm based on container technology.The algorithm builds the mathematics of task average waiting time and system container resource utilization based on queuing theory.We uses a faster particle algorithm called PSO that with faster convergence to solving the model.The solution results can help us accurately adjust the number of containers and maximize resource conservation.Experiments show that the algorithm significantly improves resource utilization under the premise of satisfying the average waiting time of the task,and the average task latency of the algorithm is shorter than that of the mainstream Kubernetes based on incremental scheduling and full-quantity scheduling.Comprehensive resource utilization is higher.The implementation of the Big Data Machine Software Middleware system architecture is given.The implementation integrates the mainstream software middleware technology,and the entire software middleware uses the adaptive container resource scheduling algorithm,so that the entire Big Data Machine Software Middleware meets the task average waiting time.The ability to flexibly expand and improve the system's comprehensive resource utilization.In addition to meeting the daily computing and storage requirements of Big Data Machine,combined with the current research focus,the distributed storage module based on multi-level cache significantly improves the data retrieval efficiency,and improves HBase by designing HBase file index and large file retrieval module.Non-primary key indexing and large file access performance.