Distributed Computing Application Oroented Resource Scheduling Mechanisms In Optically Interconnected Data Center

With the rapid development of computer science,mobile Internet and cloud computing technology,the amount of data generated by Internet applications grows explosively.The large-scale Data Center(DC)becomes the fundamental infrastructure for big data processing and has caused widespread attention.Distributed computing applications introduce intensive transmission of large-scale data sets(from GB to TB),which in turn requires the high performance of intra-DC network.However,traditional electrical interconnection technology in DC cannot satisfy such requirements.The multi-granular all-optical switching technologies,with advantages of high bandwidth and low latency,are considered as key enablers of future intra-DC networks supporting high performance distributed computing applications.For optical intra-DC networks,the data transmission delay may still cause bottlenecks for distributed computing applications.This is because that conventional computing applications can only invoke computing resources and storage resources flexibly but merely regard the network as black boxes,while the network also lacks the capability of providing precise application-level flow optimization.In light of these,this thesis studies the synergistic scheduling mechanism of application and optical network.Specifically,the contributions of this thesis can be summarized as follow:Firstly,a novel network-aware resource allocation mechanism for distributed computing is proposed.Taking Hadoop and its resource management framework YARN(which is one of the most widely used distributed computing platforms)as example,we analysis the deficiencies of the existing resource allocation mechanisms and then propose an improvement approach by taking network resource status into consideration.On the basis of resource status perception,we describe a specific network-aware resource allocation algorithm,which allocates the resources for computing tasks with consideration of both tasks' data locations and network utilization.Simulation results show that,compared to original algorithm,the proposed network aware algorithm can significantly reduce the job completion time.Secondly,a novel application-aware optical network resource scheduling architecture is proposed and studied.The design principles of the architecture are introduced,which use the centralized control plane to realize the accurate mapping between application layer's requirements and underlying network's fine-grained resource scheduling.Meanwhile,the detailed design and implementation of the optical network controller and its communication interfaces are also presented,with a specific objective of supporting efficient Coflow scheduling.Finally,we evaluate the feasibility and effectiveness of the proposed architecture and mechanisms by proof-of-concept experiments on a system prototype.