The Design And Implementation Of Educational Data Mining Platform Based On Docker

With the rapid development of Internet information technology,more than one billion people surf the Internet every day,and these users will produce a large amount of information and data.According to scientific research statistics,the annual data generated worldwide are calculated by ZB.There are always some valuable rules or patterns behind these data.We can use mathematical analysis,machine learning or deep learning to dig out the valuable things hidden behind the data.But the surge in data volume also brings new problems and challenges:How to store massive data.How to use limited computing resources for large-scale machine learning model training.How to apply the meaningful machine learning model we have trained to more directions.At the same time,in recent years,the main purpose of cloud computing technology is to schedule and allocate limited computing resourcesrationally.The goal of cloud computing is to allocate resources such as CPU or memory to containers according to the expected or actual usage of tasks.With the efforts of large companies and open source communities,various cloud computing technologies have emerged.Containers and virtualization technology,as the basis of cloud computing,have also developed rapidly in this period.Docker,as a lightweight container technology,has already defeated the traditional VMWare virtual machine to a certain extent by virtue of faster start-up speed,less resource occupation,simple deployment and rich mirroring.Docker is one of the best ways to distribute and reuse programs because it contains both code and the environment in which the code runs and even contains data.In the background above,we designs and implements a data mining platform on the basis of Docker container technology on this paper.There are two main purposes of this platform: To provide a better environment for algorithm developers,so that developers can focus on the data and algorithm itself,and can package the trained algorithms and models as services to provide more users through the platform.Provide a platform for algorithm users to use the model for complex analysis of the corresponding data without concern for the preparation of the algorithm and the training of the model.Although the demand is clear,there are mainly the following difficulties: How to maximize resource utilization and system throughput in the case of limited resources.How to provide the trained algorithm and model as a service to users.In order to improve the efficiency,how to design the system can be deployed in a distributed environment,while providing high availability and efficient services.How to improve the universality of the platform.The paper focuses on the analysis and overcome the above difficulties.The platform mentioned in this paper mainly serves the educational administration data generated by the ntms of Jilin University,but ultimately it is intended to provide a general data mining platform.