Design Of Distributed Processing System Based On Windows Operating System

With the rapid development of IT and communication industry, the global data is explosively increasing, and the times of big data have come into being. Simultaneously, the mainstream technique of processing big data—distributed system—has become one of the most popular research techniques in recent years. Under this background, in this thesis the distributed cooperative processing system which is used to complete data processing is designed based on the hardware platform of generic PC cluster using the Windows operating system.Distinguishing by the operation of computing server, two design patterns of distributed processing system are illustrated in this thesis. One is “active mode” distributed processing system which is based on the management of database. And another is “passive mode” distributed processing system which is based on client/server(C/S) architecture. These two kinds of distributed processing system are realized by the theories of directed acyclic graph(DAG), distributed file system, database, etc.Utilizing database as management core, “active mode” distributed processing system inserts compute task/modules into database as log. Subsequently, the computing servers will compete to get data processing module and complete it. Four strategies of compute module scheduling are proposed in this thesis. Meanwhile, the simple weight priority scheduling strategy and the compute task priority scheduling strategy are applied in “active mode” distributed processing system. According to the actual effects of efficiency, the simple weight priority scheduling strategy performs better in global module processing, and the compute task priority scheduling strategy has better results in data processing tasks which have high timing requirement.“Passive mode” distributed processing system is designed by C/S architecture. In this system, monitoring server play a regulatory role for compute task/modules and online computing server. In accordance with the simple weight priority scheduling strategy, monitoring server assigns compute task/modules to computing servers by checking the status of computing servers. Afterwards, computing servers will receive the instructions and complete corresponding data processing. Three technique aspects for implementing “passive mode” system—communication module, data structure and algorithms, and database—are analyzed in this thesis, and it is proved that “passive mode” system has stably nice effects on efficiency when changing the number or scale of compute tasks.Theoretically, these two modes of distributed processing system both have certain scalability. When the number of computing servers is less than 50, the acceleration effect of data processing is obvious. However, the acceleration trend becomes flat when the cluster contains more than 100 computing servers. Furthermore, these two modes of distributed processing system have been used in practical application, and both have certain efficiency promotion.In comparisons of these two modes of distributed processing system, “active mode” system have more advantages of efficiency and realization cost than “passive mode” system. But on the other hand, “active mode” system is inferior to its opponent in the areas of security, robustness and scalability.