The Design And Implementation Of Compatibility Module Of Storm On Gearpump Stream Engine

With the development of big data stream processing technology, more and more big data streaming engines have sprung up. Big data stream processing can be used to process the dynamic data in real time and feedback the results in time, and obtain the value of the data in its effective time. So it has become an important technology in many scenarios.Storm is a stream processing platform which is widely used in the industry, but it also has some limitations. Intel’s new open-source streaming engine Gearpump has solved some limitations of Storm and provided a more excellent streaming engine for users. In order to make users experience the excellent features of Gearpump with lowest cost, we design and develop the Storm compatibility module over Gearpump streaming engine to make the Storm application run directly on the Gearpump platform.In order to implement the Storm compatibility module over Gearpump stream processing engine, this paper firstly introduces related background and current situation of the development about stream processing platform, and does research on some popular stream processing platform like Storm, Samza and Spark. Then it introduces the key technology of this project, and does in-depth research and analysis of Storm and Gearpump, which are the two stream processing platform closely related to this project. And it gives the requirement of build a Storm compatibility module over Gearpump stream processing engine. Based on above work, it analyzes the requirements of the project, focusing on the functional requirements and use case description of submission of application and topology transformation, and the three sub modules, which are sub module of GearpumpNimbus, sub module of GearpumpStormClient and sub module of topology transformation, are detailed designed. Finally, implementation details of each sub module are given.This paper implements the Storm compatibility module over Gearpump, and Storm application can run successfully on Gearpump with the ensurance of high performance, so that Storm users can learn more about the excellent features of Gearpump streaming engine.