Joint Optimization Research Of Data Center Servers And Networks

As cloud computing core infrastructure and next-generation network technology platform for innovation, data centers around the world run large-scale distributed computing tasks have been built. The open source distributed computing platform hadoop, which regards distributed File System (Hadoop Distributed File System, HDFS) and MapReduce (Google MapReduce open source implementation) as the core, has been widely deployed in the data center relying on its advantages in high reliability, scalability, efficiency and fault tolerance. For data center network, new topologies and various bandwidth sharing mechanism also allow network throughput greatly improved.However, the computing resource optimization and optimization of network traffic have no direct correlation; because different optimization strategies have different traffic matrix, and different traffic matrix has differently effects on the network load, thus it impacts the performance of the calculation differently. In addition, as the time of hadoop development is relatively short, for the case of multi-job data distribution and handling is still not perfect.Therefore, based on the thought of joint optimization between network terminal and computing terminal and SDN (Software Defined Network), this article designed a scalable VM placement and network routing task joint optimization solution.In this scenario, first of all, the principle of HDFS file system and MapReduce computation processes were analyzed and introduced; and data storage Hadoop MapReduce computing platform for testing the performance and HDFS file system were built; then, according to the problems coming up from the information of testing statistics and summarization of related literature, the data center network topology Fat-Tree and the thoughts of Fat-tree applied in the Equal Cost Multipath (ECMP) protocol and SDN were introduced; in the following, the specific task assignment scheme and the joint scheme based on Fat-tree network topology were presented, which determines a variety of documents and divide rational allocation of HDFS location map/reduce tasks of different jobs;according to the characteristics of Fat-tree topology, new bandwidth allocation algorithm based on the ECMP protocol was designed. Finally, the joint optimization was verified by experiments, which turns out that our scheme based on the computing optimization can improve network throughput and improve the transmission network in different work streams.