MPI Collective Communication Optimization On TIANHE High-Speed Interconnect

MPI collective communications are widely used in modern parallel applications and are very important in scientific computation(Sometimes, collective communications take as high as 70% of the scientific computation time.) and coding parallel program, however, collective communications that based on software are fundamentally unscalable because of the increase of communication steps, computation data, and communication distance and so on. These increases result in a long collective communications work time, however, modern parallel applications are growing, which desperate for fast and effective collective communications.One approach to accelerating collective communications is the offload mode on interconnect network interface, which makes use of some special hardware to assist host dealing with some work including data computation and data movement. TIANHE high-speed interconnect network interface contain a trigger logic to offload data movement, which accelerating collective communications. In this paper, we focus on MPI collective communication optimization on TIANHE high-speed interconnect network. Our main contributions are as follows:1) We extend the ?-? model and use the extended ?-? model computing the cost results of software collective communications. These cost results is important to later analysis. The classical ?-? model is used to measure the collective communication cost in high level and can't get the cost result in a specific setting, the extended ?-? model, however, can work out it.2) We propose a new cost evaluation model that suit for the offload collective communications, and the results of the test show a good fit for the theoretical results. The offload evaluation model gives the rationale of the offload collective communication optimization, moreover, guide us to optimize offload collective communications.3) We optimize the offload barrier operation and offload broadcast operation. Barrier and broadcast operations are frequently used in collective communications, and can be used in other collective communications. We use k-ary tree and k-nomial tree to optimize barrier and broadcast operation, and prove that the offload collective communications are scalable.We test the offload collective communications on the TIANHE high-speed interconnect network and the results show that offload barrier performance is 2.17 times increased to point to point barrier, and the optimized offload barrier and broadcast gains 1.1 and 1.46 times faster to the normal offload barrier and broadcast. What's more, the cost model we established can be used to analyses the cost of offload collective communication, the test result shows that this analysis is of importance and fit to the test result.