Design And Implementation Of Cloud-native Big Data Computing Engine Management Platform

With the rapid development of digital transformation,we have entered the era of data explosion.Faced with such a massive amount of data,traditional big data platforms have difficulty in meeting the computational efficiency and resource utilization needs.Cloud-native technology,with its advantages of elastic scalability,automated management,cross-platform compatibility,security,and reliability,can effectively complement the shortcomings of traditional big data platforms.Therefore,cloud-native technology plays an important role in the next generation of big data technology.Of course,cloud-native applications in the field of big data are just getting started,and there are many problems and challenges at present,such as:1)inconsistent cluster management,the deployment and management of big data clusters still need to be implemented through the management pages or interfaces provided by each component,which cannot achieve the unified management of various clusters on a unified platform;2)inconsistent operation and development,the separation of operation and development tasks increases communication costs and reduces the efficiency of big data processing;3)low conversion efficiency from technical architecture to actual scenarios,making it difficult to quickly build a big data analysis platform and achieve rapid conversion of enterprise data value.To solve the above problems,this paper focuses on the research and analysis of cloud-native technology,big data computing engine technology,and containerized service-related technology,and completes the design and implementation of a cloud-native big data computing engine management platform.The main research contents include:1)proposing and implementing a cloud-native deployment method for big data computing engines,which solves the problem of inconsistent management methods for big data computing engines by constructing a control plane to uniformly manage cluster nodes;2)proposing and implementing a containerized construction method for cloud-native big data platforms,which decomposes the complex construction process of big data platforms into operations on individual resources,solving the problem of complex architecture and difficult construction of cloud-native big data platforms;3)designing and implementing a cloud-native one-stop big data application management and development platform,developing and managing big data applications on one platform,realizing rapid deployment and online operation of big data applications,and meeting the delivery and operation needs of big data applications in cloud-native scenarios.Finally,this paper realizes a cloud-native big data computing engine management and development platform,which provides a one-stop solution from image management,container application deployment to cluster management,architecture management,application development,and ultimately to big data job workflows.The platform has been applied in the national key R&D program project "Research and Development of Science and Technology Consulting Technology and Service Platform Based on Big Data",verifying the effectiveness and feasibility of the platform and solution proposed in this paper.