The Design And Implementation Of Streaming System In Kylin Framework

With the arrival of big data age,a variety of big data framework continues to develop,memory computing and streaming computing framework has been more and more popular.The traditional OLAP engine,using the batch processing model,has been unable to meet the user’s performance requirements.Therefore,how to add the streaming processing or memory computing capability to the traditional OLAP engine,reducing the query response latency,has become a hot topic in the big data field.As an open source OLAP engine,Apache Kylin leverages the distributed computing framework Hadoop,and adopts the way of pre-process,which provides response in sub-second for most query.However,Kylin is actually a batch framework.The Kylin Streaming system,as a sub-module of Apache Kylin,adds the ability of streaming processing to Kylin systems,effectively reduces the latency of the pre-process.The Kylin Streaming system uses the streaming processing model,and Kafka as the streaming data platform.Data is aggregated in memory before cube building.The results of the aggregation can respond to the user’s query,which greatly reduces the query response delay.And then,the Streaming Cubing Engine module builds data cube based on the aggregation results,which as well meets the massive data query requests.With the Kylin Streaming system,Apache Kylin has become an OLAP engine with both off-line processing and real-time processing to meet the needs of users in different use case.