Research For Intermediate Result’s Management And Data Access Technology In Column-oriented Database

As the development of the information society, the way of people’s communication arechanging from face to face to talk on QQ, weixin and other communication tools; That makesthe Internet data become bigger and bigger, comes to PB unit, even to several PB data, itmake most of company hard to handle with it; What’s more, company always analysis thosedata, and it also product more new data, so it is hard to avoid the enterprise’s database crash.Another thing, with the many different type of data, for example, the web page informationdata, the audio data, the movie data and many other kind of data, and this is one big problemto deal with internet data. So, how to store and deal with those big data becomes the wholeinternet’s hot topic, and the database, ones as a kind of the best media to store data, becomethe focus of every big enterprise. Many different version of different database comes, such asfile database, MonetDB, NoSQL and so on.This paper research on the project of <<Heterogeneous column database query system>>,the main principle of this project is decompose the SQL sentence to primitive dependencygraph, and then use the high parallelism GPU to speed up the performance of computeprimitive, use the CPU to combine the result compute by GPU to become the final result;thiskind of database is use to deal with big data, and with limited physical memory size, but thedata need to handle much bigger than the memory size, and the number of query is big, so itmakes the data swap between disk and memory become often; for the reason that the speed ofread/write from disk is much slower than read/write form memory; so this page propose onemethod to reduce the exchange between disk and memory, and speed up the queryperformance; this method is in physical level, change the memory allocation and recoverymechanisms, make the memory’s allocate and recovery more fast; in software level, analysiseach query’s relation, get the common part, so it can reduce the compute pressure, speed upthe query performance, and then make the database performance much better.Form the test result, we can tell, effective method can obvious improve the disk’sread/write speed; and with the memory manage technology, it make memory’s allocate andfree far more fast, and memory utilization. And after use the intermediate results managetechnology, the adduction of intermediate results raise obvious, reduce the system’s compute pressure, finally make the database performance better.