Design And Implementation Of Fine-Grained Network Interface For Big Data Parallel Computing

The arrival of Big-Data has cause problems of data-intensive issues which have large scale of data with poor locality, causing irregular communications and abnormal memory access. Traditional high performance parallel computers are generally designed for computing-intensive problems, which suit for large-scale floating-point computations. However, these parallel computers may have difficult to deal with data-intensive problems. Especially, the traditional communication system in super computers suits well for the transmission of large blocks of data while the characteristic of poor locality in the size of Big-Data leads the processors to launch fine-grained memory access, which may cause the inefficiency of communication system.In order to solve the fine-grained communication problem of Big-Data parallel computing, this paper studies characteristics of Big-Data problems and researches the communication system architecture of parallel computers. The main work is as follows:First, we study the existing parallel computer architecture especially structures of the memory and communication system and analyze their shortcomings in the face of Big-Data problems. In order to solve the bottleneck of communication, we proposed a GNG communication system which including three components: network interface, memory access control and synchronization control.Second, in order to reduce the communication delay between nodes, we propose a low-latency hardware communication interface structure. By optimizing the structure of multiple aspects, a significant advantage of this interface is the reduction of communication latency between nodes. Based on high-speed InfiniBand interconnects, we implemented this communication interface on FPGA platform, making it compatible with standard InfiniBand subnet. In our test, this interface has an low latency of 0.398 ?s with a high link transmission rate of 40 Gbps.Third, in order to achieve efficient global memory access, we proposed to use a shared memory structure which has global unified address space. Under such a structure we use a unified model of CPU instruction to local and remote memory access requests. In our test, the remote read and write requests can be accomplished in 1.052?s and 0.474?s ?Fourth, we design a multi-node hardware synchronization scheme based on the above communication mechanism for the requirement of parallel algorithm. By optimizing the synchronization register and interface with the network, four nodes can complete a synchronization in 3.176 ?s under this mechanism.