Design And Implementation Of The Incremental Iterative Task Oriented Problem Set Programming Model

In recent years, big data computing technology developed rapidly, becoming the hot spot in the industry and academia. In order to handle large data sets, many research institutions and enterprises delved into this field and have put forward a lot of programming models for big data computing. The way to processing big data sets can be divided into two kinds: batch mode and flow processing mode. For these two modes, there are many programming model implementation systems. The current mainstream systems include Hadoop, Spark, Storm, S4, etc. These systems are particularly effective in handling certain type of tasks. However, it will be difficult when handling the incremental iterative tasks.When dealing with incremental iterative tasks, the application may generate new tasks, and these tasks should be computed in the next round. This process continues until all tasks are hadled and the application specified stoping conditions are reached. The current programming models are not designed to deal with this type of tasks. So it will reduce systemance performance or make the programming model more complicated when dealing with the incremental iterative tasks with existing programming models. So the problem set programming model is designed in this paper to solve the incremental iterative tasks.In this programming model, each task to deal with is called problem, the component to handle problems is called problem resolver. The problem set is design to save and manage the unsolved problems. The iteration controller is designed to controll the job execution. Problem resolver abtains problmes from problem set and solves them. The new-generated probems will be added to the problem set after fitered by iteration controller. The job will finish when the problem set is empty and the iteration controller’s stop condition is reached.The programming model is implemented as Proset system in this paper. Proset is developed in Java language and runs under Linux environment. Since each component may crash at run time, the backup and recovery mechanisms are designed to solve this problem. This improves the usability of the system. To suport multiple jobs running at the same time, resource scheduling is designed based on the dynamic priority strategy in this paper. This improves the utilization of system resources. The Proset system has been used to implement the distributed crawler system, and it runs well.