Research On Mapreduce-oriented Data Center Networks With Recursively Hierarchical Structures In The Integrated Information Infrastructure

As the infrastructures and data processing mechanism, data center networks (DCN)andMapReducecomposethecorearchitectureofdatacentersandcloudcomputing. DCNand MapReduce also have emerged as the key technology supporting the integrated infor-mation infrastructure to process massive military information and data efficiently. TheyarethenecessaryconditionforachievingtheinformationsuperiorityinthefutureNetworkCentric Warfare. In recent years, emerging diverse services call for an improvement ofthe topological performances of DCN. In order to meet those requirements, several novelDCN structures have been proposed. One of these novel structures is recursively hier-archical structure, which has better self-organization, higher reliability, and better scala-bility than the others. It has important research value and extensive application prospectin both military and civil fields. These structures focus on optimizing some fundamentaltopological properties. However, they usually do not pay much attention to the require-ments of applications running on DCN, especially the adaptation to the data processingmechanism of MapReduce. This dissertation studies the matching of MapReduce anddata center networks with recursively hierarchical structures in the integrated informationinfrastructure. The main results and contributions of this dissertation are as follows.1）AmethodologyformeasuringthereliabilityofDCNwithrecursivelyhierarchicalstructures is researched.Considering the military requirement, a methodology for measuring the reliability ofrecursively hierarchical DCN is researched. The reliability criteria, including connectiv-ity, convergence, and sensitivity, are analyzed as measuring principles from the perspec-tive of topology design. Based on the formal description of recursively hierarchical DCN,concreteschemesformeasuringeachreliabilitycriterionareexplored. DCell,FiConnandBCube, which are main stream recursively hierarchical structures, are employed as casestoprovethatthemethodologyisfeasibleandeffective. TheexperimentalresultsshowthatFiConn possesses the best sensitivity, the worst connectivity and the worst convergence,and BCube is more reliable than the other two structures in terms of the three reliabilitycriteria.2）A methodology for MapReduce rationality verification is researched.Based on Object Petri Net (OPN), a systematic methodology to validate the ratio- nality of MapReduce is researched. The main functions of the rationality verification forMapReduce are constructed. The rationality criteria for each of the functions are exploredas verification principles, including logically executable procedure, no straggler, no Mapconflict, and reasonable execution time. Since OPN can well describe the inner relation-ship of a complex MapReduce procedure and simulate its running process automatically,OPN is leveraged to model a MapReduce procedure. Schemes for examining each ratio-nality criterion are proposed by analyzing the consequence of model execution, so as tovalidatetherationalityofaMapReduceprocedurewithoutrunningitonDCN.Theresultsfrom extensive case studies demonstrate the effectiveness of the proposed methodologyin verifying the rationality of MapReduce.3）A recursively hierarchical structure supporting MapReduce is designed.BasedonBCubeandFat-tree,aMapReducesupportedrecursivelyhierarchicalstruc-ture is presented, named Hyper-Fat-tree Network (HFN). HFN is recursively defined in asimilar way as BCube. To generate a high level HFN, a lower level HFN acts as a unit andmany such units are interconnected by means of a hypercube. The major difference fromBCube is that the lowest level HFN is a Fat-tree like redundant structure, which providesreliableservicesforMapReduce. Theinterconnectionrelationshipsamongswitches,mas-ter servers, and worker servers in the lowest level HFN are determined according to theprocedure of MapReduce. Consequently, HFN is well suited to the data processing mech-anism of MapReduce. HFN inherits the advantages of BCube as well as Fat-tree, andhence has high connectivity, low diameter and high reliability. HFN also scales well, asit accommodates much more servers than BCube, which meets the requirement of im-proving the scalability of DCN for the construction and development of the integratedinformation infrastructure.4）A methodology for organizing and maintaining data files on DCN with recur-sively hierarchical structures is researched.Based on Distributed Hash Table (DHT), a methodology for organizing and main-taining data files on recursively hierarchical DCN is researched. The functions of serversfor organizing and maintaining data files are studied according to their interconnectionrelationship in recursively hierarchical DCN. Utilizing the basic principles of DHT, theschemes for storing, reading, and maintaining data files on recursively hierarchical DCNare proposed. A specific DHT architecture and a corresponding routing scheme for orga- nizing and maintaining data files on recursively hierarchical DCN are presented. Fault-tolerant approaches, including fault-tolerant routing and approaches dealing with serverfailures, for organizing and maintaining data files on recursively hierarchical DCN areexplored. Experiments based on HFN and BCube are conducted to evaluate the meanroute length of data operation and the data operating success rate when considering nodefailures. It is demonstrated that the researched methodology can efficiently send variousdata operation information to corresponding servers. It is also proved that the method-ology is competent for organizing and maintaining data files on recursively hierarchicalDCN.5）A methodology for performing MapReduce on DCN with recursively hierarchi-cal structures is researched.Based on Distributed Hash Table (DHT), a methodology for performing MapRe-duce on recursively hierarchical DCN is researched. According to the basic data process-ing mechanism of MapReduce, the schemes for designating master servers and workerservers, assigning Map and Reduce tasks and delivering intermediate data on recursivelyhierarchical DCN are proposed. A specific DHT architecture and a corresponding rout-ing scheme for performing MapReduce on recursively hierarchical DCN are presented.Fault-tolerant approaches, including fault-tolerant routing and approaches dealing withserverfailures, for performingMapReduceonrecursivelyhierarchicalDCNareexplored.Experiments based on HFN and BCube are conducted to evaluate the network perfor-mances, including load balance, throughput and bandwidth, when running MapReduceby means of the researched methodology. Comprehensive analysis and simulations showthat the methodology can evenly distribute the workload and well support throughput-hungry MapReduce applications even considering node failures.