Design And Implementation Of Big Data Continuous Analysis Platform

With the continuous development of computer and network technology,the world has entered the era of big data.The effective use of big data will greatly promote social development and scientific and technological progress.Data analysis came into being.However,data analysis has encountered some bottlenecks.The first is that data analysis requires a lot of programming experience.Secondly,in the iterative analysis process,there is a lot of repetitive work,which will greatly reduce the efficiency of data analysis.This paper designs and implements a big data continuous analysis platform.Compared with the commonly used Python and R language data analysis,the platform encapsulates the algorithm and constructs the data analysis process by dragging and dropping components,so that users do not need to care about the internal implementation details of the algorithm.Lowered the threshold for data analysis.At the same time,a customized K-Means clustering component for telecommunications fraud is proposed.Use the continuous process optimization algorithm based on the shortest path to extract potential structures from the existing process set and provide suggestions for the layout of the new process.In order to realize the continuous analysis platform of big data,this paper first introduces the development status and related technologies of relevant data analysis platforms at home and abroad,then analyzes the main functional requirements and non-functional requirements of the system,and then puts forward the key points that need to be solved in order to realize the platform Technology:In order to achieve visual process orchestration and process reuse,the system proposes data binding based on event monitoring mechanism,process description language based on XML format and process analysis engine;for continuous optimization of process,continuous optimization algorithm based on shortest path is proposed.After that,the overall architecture and outline design of the continuous analysis platform for big data are introduced.Based on the overall design,the design of the key modules is introduced in detail.Finally,the continuous analysis platform for big data is tested.In the last part,it summarizes the existing deficiencies of the platform and the future improvement direction for these deficiencies.