Adaptive Processing Of Ad-hoc Queries On Data Streams

For continuous query processing over Data Streams, query plan is mainly used as the underlying execution model, many adaptive methods assume user queries are almost predefined which make the global query plan optimization possible. While not all analysis requirements can be confirmed and predicted in early time, ad-hoc query is still needed in some practical issues. In the stock market monitoring area, analysts could submit some ad-hoc queries over high speed stock ticker streams in time, while observing the dynamic status of the market, to process the newly arrived stock tickers and validate their prediction against the analysis results. Adaptive ad-hoc query processing over data streams must be capable of transform the query semantics into the running query plan in time with an efficient online plan updating and migration, and also is scalable to continuous arriving queries without disruption of current running plan. Some existing methods wasting much time on the state in a large sliding time window, which leads to a severe response delay。Considering of solving the adaptive problem during ad-hoc query processing, this paper proposes a multi-tier query plan index to collect distribution statistics among all the internal operators’ output data streams。Directly reusing of existing operators is the vital issue for the incremental updating of current query plan, the plan index returns matched operators with their support for an ad-hoc query. Based on operator set returned by the plan index, we can select an best plan updating for current query according to a plan cost model we proposed, the selecting algorithm output a best plan updating and materialize the involved operators into current plan to complete the incremental plan updating. As shown in the experiments we conducted, the adaptive method performs better than existing methods on memory consumption and response delay.