Design And Implementation Of Visual Data Platform Based On MapReduce

In the process of rapid development of Internet,more and more enterprises analyse and evaluate the data using BigData.Data analysts use MapReduce computing framework to summary and analysis,most data analysts even know the most popular Hadoop computing framework,but not proficient at using the Hadoop command to operate the computing tasks.For the use of MapReduce often requires very specialized cloud computing engineers to complete,but the learning process of Hadoop is extremely complex and time-consuming,which makes the data analysts using MapReduce difficult.On the other hand,the artificial MapReduce operation always result in mission failure or error by various subjective factors.Although there are some products designed for the MapReduce workflow engine,the products are using command line operation,lacking visual operation environment.Aiming at the above problems,this paper designed a visual platform combined with the visual transformation of XML algorithm,this paper mainly includes three aspects:1.In the defmition of workflow graphical way,this paper provide a visual interface to deploy MapReduce.When Data analysts face big data processing,they can deploy and define MapReduce jobs in visualization tools,do not need to care about the background process.In the operation of the user visual analysis,this paper puts forward the concept of the user's workspace,including the application of file sharing and workflow chart management.After the task is referred,this paper design and realize the visualization of the process state query module by using the feedback mechanism of Oozie.This design is convenient for MapReduce workflow job visual management.2.This paper illustrates the basic concept of the workflow engine and MapReduce framework,and introduces the visualization and XML document conversion algorithm Based on above,this paper design and describes XDWE basic operation process and related technology.Ac cording to the visualisation operation,this paper puts forward the working process to XML document conversion algorithm.At the same time,this paper puts forward XML document to working process conversion algorithm for workflow management.Aiming at the analysis of the MapReduce workflow,this paper designes the XDWE module to deploy the workflow.3.This paper analyses the business process and application requirements of the visual data processing platform,and for data modeling of the system data and process respectively.Based on above,this paper establish the overall application framework system.This platform designs multiple functional modules,coding design,database design and 10 design.This designment meets the platform requirements.This platform is realised and pass system tests.This paper discusses related theories of visual operation and workflow engine.Combined with the enterprise's actual use background,this paper designs a visual processing system meets the actual demand of enterprises,and implement the engineering system using related technology.The application of the system would make data analyst use transparently.Only need to submit the form of a flowchart,the analyst can get the result information.The platform test shows that the platform runs nore slowly than using MapReduce comands alone,but the thesis solves the data analyst using MapReduce for data analysis needs,they also can combine multiple MapReduce tasks and run very well.