Design And Implement Of A Light Distributed Computing Engine Based On Memory

In recent years, with the developing of the network technology, the number of network users is growing. While surfing on the Internet, the users may product multiple behavior data, including web pages, shopping online, video, etc. The quantity of data which is producted by billions of people is enomous. How to store and maintain these data is a challange to us, but these ocean data can bring us a new era of opportunity. Using the data to data mining, giving the users personal reconmmending and real-time pushing become a profitable business model.For geting the exact user classification information, data need to be iterative computed. Therefore, how to support the high-speed iterative computation is a problem for us.A lighting distributed computation engine base on memory has been designed and implemented in this paper. It introducts the whole structure, and implement details. The system make suse of distributed computing, distributes the data to different machines, and alleviate the load, increase the computing speed. For supporting iterative computation, system store sthe data in memory rather than disk, and decrease the time of disk I/O. Before the whole computing process, system creates the computation scheduler. The scheduler saves the relationship between the data. When data lost for machines going wrong, it can be revoverd according to the scheduler. Meanwhile, the system providesthe users program interface. The process of data distributed and recover is transparent to users. System makes both computing control nod and computing excute node include a resource control mechanism. System also introduce the load balancing, realtime proteining every node ruuning conditions. System distribute task based on running conditional.In implementing, the communicating model uses the Epoll drive to support asynchronous event. System can process effectively when comuniting multiple network I/O. Moreover, the thread pool model can process file I/O, database I/O. It can process slow disk I/O without processing fast netstwork I/O.At last, system is evaluated systematively. The result of evaluationg shows the engine can support some distributed computing algorithem and provide the fault-tolerant. The performance also exceeds other traditional distributed computing engine. In addition, System has some enhances compared with popular distributed computing engine when running multiple different business task.