Study On Optical Interconnection Computing System For Distributed Applications

With the rapid development of distributed computing,the network traffic in data centers has been dominated by the east-to-west traffic intra data center network,instead of the north-to-south traffic between the data center and the outside world.The development of distributed computing has brought a lot of challenges to the design of the data center network,and the most prominent problems are the complexity of task types in distributed computing,the burst of network traffic,and the diversity of network traffic demands.Due to the characteristics of low delay,high bandwidth,multiple switching dimensions,abundant network resource dimensions and flexible topology,optical switching technology has become a promising solution to solve the problem of the network on distributed computing.Although many achievements have been made in the existing research on data center network based on optical switching,the research on network characteristics of distributed applications is still lacking.On the one hand,previous studies on the underlying network topology of optical switching have not considered the network traffic space characteristics of distributed computing applications.On the other hand,the previous research on upper control and scheduling methods for distributed computing network traffic in optical switching data centers is often difficult to adapt to the high complexity and dynamics of data center networks.In addition,there is still much room for research on the collaborative design between distributed computing applications and optical switching network architecture.These problems restrict the further optimization of the distributed application performance in the optical interconnection data center.To handle the above problems,focusing on today's rapidly developing distributed applications,the goal of this thesis is to study the optical interconnection computing system,and to explore how to make good use of optical switching technology to solve the network pressure brought by distributed computing applications from different layers and perspectives of the underlying topology,the upper control,and application layer in the optical interconnection data center network.The main contribution of the thesis can be summarized as follows:First,from the perspective of the underlying optical topology layer of the network,this thesis studies how to effectively construct the optical topology in data centers to support the network traffic caused by the space characteristics of distributed computing.In this thesis,we propose OCBridge,a topology reconstruction mechanism,based on inter-regional traffic demand.By referring to the edge betweenness in graph theory to evaluate the value of optical links,OCBridge can assist the data center to reconstruct effective optical network topology.Based on this idea,OCBridge can significantly improve both the utilization rate of the optical link and the carrying capacity of data centers for the traffic generated by distributed computing applications.According to the simulation analysis in the hybrid optical-electrical data center architecture named Helios and the all-optical interconnection data center architecture named OpenScale,we prove that OCBridge is a general and effective method.According to experimental analysis,OCBridge can also accelerate network-intensive distributed computing applications.Second,from the perspective of the upper control layer of the network,this thesis studies how to apply artificial intelligence methods to handle the flow scheduling problem of the hybrid optical-electrical data center with high complexity and dynamic,and proposes Flow Splitter,a flow scheduling method based on deep reinforcement learning.The design of Flow Splitter in this paper avoids the effect of operation time of deep reinforcement learning on traffic scheduling,and improves the convergence of deep reinforcement learning.Based on learning and training,Flow Splitter can continually explore the network flow characteristics in the hybrid optical-electrical data center,and finally get the optimal flow scheduling strategy.Through simulation analysis,we found that Flow Splitter can effectively solve the defects of the existing network traffic scheduling methods,and greatly reduce the completion time of the latency-sensitive flows.Besides,we prove that with a little overall impact on the data center,our Flow Splitter is capable of meeting the needs of distributed computing applications.Third,this thesis explores how to co-design distributed computing applications with the underlying topology layer and the upper control layer of the optical interconnection data center network.This thesis proposes OEHadoop,a hybrid optical-electrical data center solution based on co-designing the network and application.OEHadoop tries to combine and optimize the three layers of the bottom topology layer,the algorithm in the upper control layer,and the application layer together.At the network physical topology layer,we add support for one-to-many flows in distributed computing applications.At the control layer,we add a resource scheduling algorithm to accelerate the network traffic.At the application layer,we make the design of a mechanism for the collaboration between the application layer and the network layer.Simulation analysis shows that OEHadoop can effectively accelerate the transmission speed of one-to-many traffic,which is common in distributed computing applications.OEHadoop can also accelerate distributed computing applications.Based on experiments,this thesis also verifies the feasibility of OEHadoop.