Design And Implementation Of Exploratory Visual Data Analysis System

With the constant development of science and technology,the data related to our daily life is increasing at an amazing speed,and it is of great importance in our daily life.How to effectively analyze and process a large amount of data has become a hot area of research.By mapping the data into graphs,symbols and colors,data visualization technology,which is a necessary tool and method to big data analysis,shows the inner connotative information in a direct way,and it provides us the reference thoughts of further analyzing and processing the data.Visualization methods can be roughly divided into two categories:exploratory visualization and explanatory visualization.The aim of the former is to help us discover the data's inner connection in massive disordered data with the help of visualization graphs at the beginning stage of processing data.The latter transmits the known information or knowledge to the public in a visible way at the visual presentation stage.Faced with massive raw and original data,it is necessary to provide a platform which can be used for data transformation and exploratory visual analysis.Combined with background mentioned above and faced with common user,this thesis designs and builds a commonly used exploratory visual data analysis system which has functions of data access,data management,data transformation and data visualization.The main work of this thesis is as follow:(1)Introduce the process of data visualization,visual encoding and Spark,to provide the theoretical basis for the design of the system.(2)Based on the analysis of demands,the thesis refines the main functions of the system and divides the system into five modules,including project management,data set service,data set transformation,data visualization and user operation.(3)The system conducts architecture design based on B/S pattern,using Play! Framework2 as the Web service framework,and adopts the thought of MVC to conduct layered architecture.In terms of the demands of processing massive data,the system brings in Spark as the engine of data transformation and data inquiry,using HDFS to store files of datasets.(4)On this basis,the thesis gives the detailed implementation process of each module.Using activity diagram describes the workflow of modules function.Using sequence diagram describes operation logic an object relationships.(5)The thesis conducts a functional test and analyzes the performance of key functions.As for the deficiency that is heatmap's drawing ability in the system is not good under big data sets,the thesis offers a prioritization scheme combined with GeoHash,verifies and analyzes this scheme.The prioritization scheme can reduce the pressure of the front-end heatmap rendering in a certain extent,and has good drawing effect.