Research On TCP Incast In Data Center Networks In Hadoop

Information age, cyber source increasing large, with the development of Internet, data requested by user becomes bigger. Since the cluster storage system has many advantages, so the cluster storage systems have been widely applied in data center. In a distributed storage environment, for example, a datacenter, the data blocks are usually striped stored as SRUs in many different servers. When too many users access concurrently, the number of responses servers increasing, the multiple senders transmit the data concurrently. When the amount of concurrent transmit data are larger than ethernet switch buffer, it will lead to Incast problem when in high bandwidth and low latency environment.This thesis researches Hadoop cluster structure, operation mechanism and the network collapse behavior in Hadoop cluster deeply. Connecting the Incast problem happening in Hadoop cluster working environment, we propose two solutions. The first one is by optimizing TCP timeout to inprove Incast phenomenon. We could change the size of m i nR T O to improve the network quality and the bandwidth utilization ratio. The second solution is through the way of staggereding and grouping the data to avoid Incast problem. Staggered the concurrently Transmission data to achieve the data’s serial transmission. Use the network simulation tool to simulate the two solutions. In the experiments, we compare the network transmission quality and bandwidth utilization. Simulation results show that the change number of m i nR T O could improve network transmit quality and transmit data staggered could avoid Incast problem effectively.The two solutions proposed in this thesis can both improve the network transmit quality and decrease the ratio of Incast in the cluster environment. They can also avoid network throughput reducing sharply.