Design And Implementation Of Big Data Analysis Platform Based On Microservice

In today's society,science and technology are developing rapidly,the application of "Internet+" in all walks of life is getting larger and larger,and the data generated by applications in all walks of life is growing in a blowout style.The data generated by many industry applications has far exceeded the data processing capabilities of traditional enterprises.However,for many companies,how to use these massive amounts of data is the key to future success.Therefore,companies urgently need to use big data analysis technology to mine the laws behind these data,and then use these laws and analysis to help them reduce costs,improve efficiency,and make more informed decisions.As the business develops,the system needs to be continuously expanded Obviously,the architecture of a single application cannot meet the needs of business development.Based on Spark+R's dual computing engine and Hadoop cluster,this paper builds a micro-service-based big data analysis platform,which can not only meet the needs of providing one-stop analysis for enterprises,but also make the system easy to expand and usable.Requirements for rapid developmentThe work of this article mainly includes the following aspects:First,the goals and problems to be solved of the big data analysis platform are described.For the problems that need to be solved,functional requirements analysis for data import,data analysis,data visualization,and non-functional requirements analysis and microservice architecture requirements are carried out.analysis.Then,the system was designed according to the requirements analysis,and the design of the system's logical architecture,technical architecture,and functional architecture,as well as the design of the main process and deployment architecture of the system were completed.According to the needs,the system is divided into user management,data source management,data pull management,data file management,task message management,analysis task management,model management,algorithm service management,algorithm introduction management,visual management,cluster monitoring,micro service monitoring Twelve functional modules;and introduced the detailed design of the system from the aspects of class structure design,database design,and key technologies.After that,according to the analysis of microservice architecture needs,the Spring Cloud microservice framework was used to complete the design of microservice technologies such as service division,service governance,service communication,service gateway,and load balancing.The technology implements all the services of the system,including user services,data services,analysis services,visualization services,and task services;and focuses on the realization of micro-services such as data management,data analysis,data visualization,and service monitoring.Finally,the system's functional test,performance test,high availability test and system scalability test analysis are completed,and the system availability,high performance and easy scalability are demonstrated.The big data analysis platform developed in this paper provides storage,processing,analysis,and visualization services.It supports the entire process of big data analysis from data aggregation,data processing,to analysis and visualization,which can help companies achieve big data modeling and analysis.Tap the value behind large-scale data.At present,the platform has been applied to data analysis of machine tools,power,coal and other industries.