An Efficient Distributed Stream Processing System With RDMA-assisted One-to-many Data Partitioning

To process a large volume of real-time data streams,existing distributed stream processing systems(DSPSs)follow the principle of data parallelism and leverage different stream partitioning strategies.The one-to-many data partitioning strategies(e.g.,all grouping in Apache Storm)play a vital role in various big data applications.With one-tomany data partitioning,an upstream processing instance sends a generated tuple to a potentially large number of downstream processing instances.Existing DSPSs implement the one-to-many data partitioning based on instance-oriented communication,where an upstream instance transmits a tuple to different downstream ones separately.However,multiple downstream instances typically run on the same physical machine to fully exploit multi-core computation resources.Accordingly,a DSPS sends the tuple containing the same data item to a machine multiple times,raising significantly unnecessary costs for serialization and communications.Our experiments show that such a mechanism indeed leads to serious performance bottleneck due to CPU overload with high parallelism levels.To address the problem,we design and implement Whale,an efficient distributed stream processing system assisted by RDMA(Remote Direct Memory Access).Two factors contribute to the efficiency of this design.First,we propose a novel RDMA-assisted stream multicast scheme to alleviate the CPU workloads of an upstream instance in distributed stream processing.By exploring a self-adjusting non-blocking tree structure with a lightweighted switching mechanism,Whale copes well with highly dynamic data streams.Second,we re-design the communication mechanism in existing DSPSs by replacing the instance-oriented communication with a worker-oriented communication scheme.By leveraging a batch-and-dispatch communication style,an upstream instance can avoid separately contacting multiple downstream instances hosted on the same worker.Thus,it greatly saves the costs of unnecessary communications and serialization.We implement Whale on top of Apache Storm and conduct comprehensive experiments to evaluate this design using large-scale datasets from real-world applications.The results show that Whale achieves 39 X improvement of system throughput and reduces the processing latency by 96%,compared to existing designs.