Design And Implementation Of GriDoc Realtime Computation Interface

Information technology produces new data rapidly,which brings us into an era of big data.Big data provides a new perspective for people's awareness and decision making,so people pay more attention to data storage and processing.Big data is heterogeneous,distributed,mass and has low value density.Traditional relational database is powerless in face of big data.People began to seek more flexible and scalable data models.As a result,NoSQL gets rapid development.To handle large data storage and access,Prof.Qi Deyu proposed GriDoc model,which clearly defines the composition and storage properties of data.It can better describe specific application scenarios,and is more suitable for knowledge management oriented large data access.Batch processing and real-time computation are two mainstream big data processing methods.Hadoop is a classic batch framework,which can deal with massive data well,but is not real-time.On the contrary,real-time computation is more suitable for the scene which emphasizes response time,and the response time is less than one second.Storm is a mainstream real-time computing framework.This paper studies existing data models,GriDoc model,and compares their characteristics.GriDoc can well represent massive datas,but is weak in data processing.This paper achieves real-time computation interface to enhance Gri Doc data processing capability.With this interface,GriDoc becomes a perfect big data solution which can support data acquisition,processing and output.Firstly,we present processor data model to represent and control data processing.Use processor to represent customized processing program,abstract storage resource into container to support computation,and provide processing result to the external by view.Based on this model,we design real-time computation interface to provide data processing function and realize an architecture with the help of Storm.It supports users to customize data processing programs and connect them to real-time computing architecture in form of plug-ins.When users need to process data,they can apply processor to data with a bit of configuration.Data processing will become easy.In addition,GriDoc can provide processing results with interface which means we can obtain integration message withoutaccessing raw data.Then,this paper uses the real-time computation interface to realize a statistical generation module.On the one hand,we verify the usability of real-time computation interface.On the other hand,it also provides a real-time and configuration oriented statistical analysis module.Users can observe the statistical results by visual interface,and get the statistical data through interface provided by GriDoc client data access driver.In the end,we develop and test the improved GriDoc system,analyze its advantages and disadvantages,and describe future research directions.