Research And Implementation Of Image Distribution And Resource Scheduling System In Large-scale Container Cloud Platform

With the development of computer technology,container technology represented by Docker has been widely used due to its flexibility and practicality.And the container orchestration platform represented by Kubernetes can help companies manage large-scale container clusters.As a result,the application and development of Kubernetes has become increasingly popular.The Docker containers is based on images,and images need to be transmitted to the worker node.In the present Docker image distribution framework,images are centrally stored in the server called image registry,and the container engine will download images from image registry as a client.When the scale of the container cluster increases,the image registry easily becomes the performance bottleneck of the cluster,which affects cluster stability.And image distribution in large-scale clusters may involve cross-region or even cross-border transmission.The C/S model is difficult to provide a higher transmission rate.Kubernetes uses the basic orchestration unit Pod to indirectly manage the container.After the Kubernetes resource scheduler binds the Pod to the work node,the Pod can be started for execution.The default scheduling strategy of Kubernetes is relatively simple,and only supports scheduling methods such as separation,label,and affinity.In large-scale container cloud platforms such as machine learning,big data processing and other Pod-intensive and frequent communication scenarios,centralized scheduling of Pods to the fewest nodes can reduce the storage and communication performance loss between Pods,but the Kubernetes default scheduling strategy does not able to achieve this scheduling goals.In response to the above problems,this thesis improves the Docker image distribution framework and designs the Kubernetes resource scheduler.The main contents of the thesis are as follows:1.Design a Docker image distribution framework based on the BitTorrent protocol.The existing Docker image distribution framework uses the C/S model,and all distribution requests are responded to by the unique image registry of the cluster.As a result,the image registry will become an important factor affecting the distribution of cluster images.In large-scale cluster scenarios,problems such as cluster instability and slow network transmission are likely to occur.This thesis designs and implements a Docker image distribution framework,including the components and image P2P distribution strategies in the framework.This framework supports the image P2P transmission mode based on the BitTorrent protocol to avoid the image registry from becoming a performance bottleneck and make full use of worker node bandwidth.Experiments prove that this framework improves the average image pull speed and reduces the load of the image registry.2.Propose a load prediction method for Docker image registry based on Holt-Winters model.Docker image registry has the characteristics of small and many files.Using P2P transmission as the default transmission mode is prone to too many seed files and low communication efficiency.This thesis proposes a method for predicting the load of a Docker image registry and designing a decision strategy for the image distribution mode.In this method,the image registry uses the single-point mode to transmit images by default.When an image distribution request is received,the image registry will train the Holt-Winters model and predict the recent load based on this model and determine whether to enable the P2P transmission mode.Experiments prove that this method effectively reduces the load of the image registry and at the same time reduces the number of seed files in the cluster.3.Design a Kubernetes advanced resource scheduler based on metric aggregation.In a heavy-load,high-interaction scenario,centralized scheduling of the same group of Pods to the fewest nodes can improve network communication efficiency,but the Kubernetes default scheduler does not support this compact scheduling strategy.At the same time,the evaluation of Pod priority in the scheduling process only comes from resource quota metric.The evaluation of Pod priority in large-scale container cloud platforms needs to combine more dimensional metrics.This thesis designs a Kubernetes resource scheduler that supports centralized scheduling and designs a Pod priority evaluation strategy based on metric aggregation to apply to the scheduler.The scheduler supports clustering and scheduling of the same group of Pods to the same node,and timely expulsion of non-same group of Pods,and at the same time aggregating multi-dimensional metrics such as resource quota,migration cost,and substitutability to generate priority for evaluating Pods.Experiments prove that this scheduler can effectively centralize and evict Pods during the scheduling process,which improves scheduling flexibility and resource utilization.This thesis builds a container cloud platform using the above-mentioned Docker image distribution framework and Kubernetes advanced scheduler.Through functional verification and performance testing,it is proved that the framework and method proposed in this thesis improve the efficiency of image distribution,reduce the load pressure of image warehouse,improve scheduling flexibility,and have the feasibility and practicability.