| High-performance Computing(HPC)is moving to the new height: Exscale.With continuous development,HPC has expanded from traditional scientific computing to emerging fields such as artificial intelligence and big data analysis.The user groups are more complex and the needs of users are increasingly diversified.Multi-user usage scenarios bring many challenges to HPC.For the problems of user privacy,isolation and environment customization in multi-user scenarios,it is inevitable to expose the privacy of users when a limited number of login nodes provide services for a large number of users at the same time.In addition,the preset standard environment is difficult to meet diverse user needs,and users with different needs are prone to different environment configuration contradictions.For the deployment overhead of jobs and their execution environment,the traditional manual deployment method is extremely inefficient,and it is easy to cause chaos in the execution environment in multi-user scenarios.Although containerization method brings convenience to the deployment of jobs and their execution environment,it is difficult to maintain container images independently by users in HPC systems which are highly customized in both software and hardware.At the same time,containerization is an additional burden for users.As the scale of HPC systems increases,the problem of large-scale application deployment and startup delays become more and more serious.At present,there are some accelerated technologies for containerized deployment methods,but none of them solve the fundamental problem of container image maintenance.Secondly,there is a lot of file redundancy in the bulky container image,and containerized method is not friendly to support large-scale parallel applications.In order to solve the above problems,this paper studies the virtualization technology in HPC multi-user scenario at the system software level.The specific work of this paper is as follows:1)The private execution environment based on lightweight virtualization,solved the problem of isolation,privacy,customized environment in the multi-user scenarios of two levels: node and cluster,broke through the bottleneck of time and space cost and maintenance overhead of existing virtualization methods in HPC,and realized the privatized and highly customizable user development environment,as well as the new mode of execution environment jointly maintained by users and administrators.The performance cost is less than 1%,and the space cost is less than 100 KB.It is lighter than mainstream container technologies such as Docker and Singularity.2)The self-deployed execution environment,realized the transparent deployment to the user’s private execution environment,and solved the problems of manual deployment and environment configuration in traditional HPC mode,which greatly reduced the workload of users and supported different jobs of different users to be loaded on the same node without interfering with each other.3)The application deployment and startup acceleration technology for large-scale HPC systems implemented efficient topology-aware file transfer for large-scale system,which improved deployment efficiency while effectively reduced network load.At the same time,the split-step transport and launch-in-advance mechanism are designed to reduce the startup latency of application.Compared to container-based application deployment,the speed is increased by 12 x,and the network load is reduced by 85%. |
