Data Visualization Performance Optimization Based On Incremental Calculation

With the development of big data technology,the explosive growth of data brought invaluable potential value to researchers,companies and society.In order to obtain valuable information from the mass data,we can display and analze data in an effective way by using data visualization techniques.Data visualization is a voluminous system engineering and its calculation process is very complicated.For large-scale datasets used in data visualization,the dataset will change incrementally as new data continues to be added over time,which will invalidate the previous display result.If you add additional data to the visualization results,or filter or drill,etc.,you need to find all the data from the database and calculate it again.This inevitably leads to low data re-use rate,loading data excessively at one time,loading ineffectively,high costs(frequent I/O overhead and network overhead).How to shift the focus of data visualization from cumbersome operations and long-term display waits to the results of data analysis has a very important research value.Incremental technology-based loading and calculations only need to load and calculate new data each time,and then merge historical data.Therefore,this paper proposed data visualization based on incremental computing,which combined incremental computing technology and data visualization,and not only can fully use historical data,but also speed up data visualization-process.This paper analyzes the existing data visualization system,and addresses the problem that existing data visualization systems cannot reuse historical data,and new data needs to be reacquired from the database.We designed and implemented a data visualization system based on incremental computing-IDVS.The system designs incremental optimization rules according to the intersection and difference set of time intervals to obtain incremental and non-incremental parts.Moreover,the construction rules of the cache intermediate result Key are designed to cache the historical intermediate result set based on the incremental optimization rules.In order to load and calculate newly added data,an incremental calculation algorithm SCIC is designed and implemented so that the system only needs to load and calculate new data each time,and then merges historical data to perform calculations,which improves the performance of the system.The experimental results show that IDVS can effectively use the historical intermediate result set and only calculate and load new data,which improves the data visualization efficiency.