The Design And Application Of Distributed Real-Time Flow Computing Framework Based On Akka

With the rapid development of information technology, the data from various sources of information are also growing rapidly, so it is very important for us to deal with these massive data. But for different types of data need to adopt different processing strategy, such as Hadoop framework that based on MapReduce model is suitable for the batch processing of mass static data, and for real-time processing of dynamic data stream, there is not good strategy with Hadoop framework. The computing framework designed in this thesis has better performance in terms of real-time processing. The real-time processing framework adopt stream calculation method to design. Different from the general stream calculation framework, the framework supports process scheduling of the data processing nodes, in this way, it can help to realize a real-time processing of dynamic data. The framework mainly is divided into the design, implementation, the practical application to elaborate.(1) The design of this framework is based on Akka, Akka is a library written in Scala, used to simplify develop the Actor model application that has fault tolerance and high scalability with Java or Scala Firstly, according to the characteristics of the Actor model, this thesis designs the front-end processing node (Frontend), the control node (Master), and the working node (Backend) of the framework. Secondly, this thesis mainly expounds three aspects:distributed computing, stream computing and process scheduling by the characteristics of these nodes. Finally, it elaborates the realization principle and the realization process of the framework by the explaination of the structure model and the processing flow of the whole framework.(2) Implementation of the framework is based on a large number of studies about the design and realization of stream computing (such as storm, spark, etc.) It is introduced by dividing it into four modules: front node module, scheduling module, working node module, resource management module, and the function of each module, the realization principle and the realization process are described in detail.(3) When it comes to the application of the framework. Firstly,this thesis introduces its application scene, such as real-time calculation, calculation of batches, real-time query and so on. Secondly, it elaborates how to use this computing framework by the state of two examples:"Collision Problem", "Resource Instant Search". Finally, it explains the practical effect of the framework by the analysis of the experiment process and results.The thesis provides developers that has the need of real-time computing of large scale data, and gives some design methods of stream computing basing on response.