Design And Implementation Of Spark PaaS Platform Under Cloud Platform

With the rapid development of cloud computing,the improvement of computer hardware and software performance,and technological innovation,PaaS platforms have ushered in vigorous development in recent years.Various mainstream development frameworks and computing frameworks have begun to migrate to the cloud,improving developer productivity.At present,PaaS platforms provide mainly frameworks or services based on single nodes,such as program development software and data analysis tools.For the distributed framework,due to the difficulty of cluster deployment and management in the cloud,there are not many applications on the PaaS platform.The Spark distributed computing framework,as one of the most popular big data processing frameworks at present,is increasingly demanded by developers.Because manually deploying a set of Spark clusters requires the integration of a large number of computing resources and takes a lot of time,there are also many problems in the subsequent management of cluster resources.Therefore,this thesis aims at Spark cluster services and builds a Spark PaaS platform with cluster automated deployment and management functions based on ansible automated deployment and management tools,which facilitates users to upload data,submit and run Spark tasks.This thesis first analyzes related technologies and determines the feasibility of building a Spark PaaS platform.Then researched and analyzed some mainstream automated deployment tools to provide a theoretical basis for the selection of ansible tools.Combined with the actual needs,the system architecture of this platform is proposed and divided into multiple functional layers.On this basis,communication module,database module,basic function module,cluster management module,large file transfer module,HDFS operation area module,remote desktop service module,user management module,log management module and other modules are designed.After that,the detailed design and implementation of each functional module were carried out.For basic functions and cluster management modules,through the ansible service,functions such as small file transfer,remote command execution,cluster deployment and deletion,and cluster monitoring are implemented.For the large file transfer module,the functions of fast uploading of fragments and resume resume of breakpoints are realized.For Remote Desktop Services,an automatic login function is implemented.For the HDFS operating area module,the function of interacting with HDFS through a web page is implemented.Finally,the Spark PaaS platform was built and deployed,and functional tests and performance tests were performed on each module of the platform,proving that the Spark PaaS platform can meet the needs of users and has a good experience.