Study And Implementation Of Elastic Stream Computing In The Cloud

In recent years, as the development of Internet, more and more data are produced,and the producing rate is also much faster than before. To deal with this situation, more and more stream computing systems are created, which accept realtime data input,and after quick processing, output almost realtime results. These systems are widely used in the areas where realtime processing is quite worthy, such as ?nance, internet and so on. However, although stream computing systems can process huge data in realtime, another essential character of big data is ignored, that is big variety which contains much meaning, and one of it is the variety in rate. Variety in rate can cause the uncertainty of workload in system, it can even lead to the crash of system if the rate changes too much.Given this situation, we think that make the stream computing system elastic may be a good solution to the variety of rate. Nevertheless, most traditional elastic computing approaches are proposed for the cloud providers, in their assumptions, systems running is the cloud are all well distributed which can be scaled very easily. This assumption may be true in the area of web servers, but in some cases, it’s not entirely correct, especially in the area of distributed systems, many facts should be taken into consideration once they are scaled. There are also some researches about elastic stream computing, but most of them just focus on the workload of software system, without any attention to the status of cluster. Besides, they do not involve the whole process of elastic stream computing, just part of them, such as an algorithm for scaling or a framework supporting scaling.To deal with these problems, we propose a cloud based approach which can adjust the stream computing system according to the workload, and we also implement an elastic system RubberStream basing on this approach and Storm. It can ?nd the exceptions of components’ workload and make some changes quickly. In addition, by combining the status of software and hardware, it can scale the cluster automatically with adaptive algorithms. Besides, RubberStream supports both horizontal and vertical scale, so that more choices are available when scaling. Furthermore, we implement a task scheduler for RubberStream which can keep balance among heterogeneous machines.In the end of this thesis, we evaluate RubberStream’s function and performance in a real commercial cloud, Aliyun. The results show that RubberStream can adjust the parallelism and scale the cluster correctly at right time. In addition, it can save much cost with just a little performance loss compared with Storm deployed in a big static cluster.