Research On The Key Technologies Of Big Data Container Cluster Based On IaaS

The advent of Docker container technology has revolutionized cloud computing virtualization and the evolution of enterprise IT infrastructure.Compared with traditional VM,Docker reduces the performance overhead of Hypersior layers and also drastically improves virtualization performance,which provides the foundation for the performance optimization of big data computing clusters deployed in the cloud.In a production environment,Docker usually runs in a cluster mode.The management architecture of the open source container,which is based on Kubernetes,Mesos and Swarm,provides a new idea for the construction of a new Platform as a Service(PaaS)platform.At the same time,as the excellent resource isolation provided by the IaaS layer,the containerized PaaS platform built on the IaaS layer will also bring better multi-tenancy and business development capabilities.In this thesis,based on the open-source cloud computing framework of OpenStack and Kubernetes,the overall architecture of large data container cluster based on IaaS is designed,the basic process and the related configuration of OpenStack service component deployment is summarized and also the deployment of IaaS service layer based on OpenStack is implemented.Based on the above,the basic architecture of a containerized PaaS platform based on Docker and microservices is analyzed and the deployment of the Kubernetes platform is completed.Based on Kubernetes and according to the characteristics of containerization and microservices,the main parts of the platform are implemented,which include the underlying network model,the application of agile deployment,the application management,the service discovery and the platform resource monitoring.The operation and basic function of the container-based PaaS platform based on IaaS has been initially realized by systematic testing.Based on the cloud platform built by IaaS and PaaS,this thesis focuses on the big data cluster containerization and the agile deployment method.A microservice partitioning method of Hadoop cluster is proposed,and the mirroring of Hadoop microservices is implemented by Docker mirroring technology.On this basis,a quick deployment template file for containerized platform is designed and the rapid deployment of Hadoop is achieved.The repetitive tasks in existing large data cluster deployments is simplified and the complexity of deployment is reduced.In view of the stability of the containerized platform,based on the core services of the containerized platform,a solution resolving the containerized platform stability based on service monitoring is proposed.In addition,about the resource collaboration dis patch between IaaS and PaaS,a strategy based on containerized platform resource monitoring feedback is proposed,which provides a new idea for the deep integration of laaS and container platform.