Splintering commodity network protocols in high-performance computing

Systems research is about creating a performance environment for applications within the constraints of the underlying components. This dissertation is specifically about creating a performance environment for high-end applications within the constraints of commodity components (protocols, operating systems and hardware).; This research analyzes the bottlenecks of TCP/IP in today's clusters and finds that while the MPI implementations need to be improved, decreasing the overhead associated with TCP/IP will increase the performance of MPI. First, this dissertation outlines offloading IP fragmentation and reassembly onto a programmable NIC. This drastically reduces the overhead associated with the interrupt pressure of Ethernet frames and leads to a general method for offloading, splintering. Splintering means isolating a small functionality of a protocol, distributing this functionality, and constraining its implementation.; The splintering method is applied to TCP by splintering the demultiplexing of the connection and offloading the demultiplexing onto the NIC. This dissertation describes splintering and its relation to other offload methods including OS-bypass and zero-copy. In order for Splintered TCP to be a success, it must be constrained by the resource limitations of a commodity NIC.; This paper outlines the Extensible Message-oriented Offload model created to model offload techniques with respect to CPU speed. The model illustrates offload hiding. Splintered TCP performs offload hiding by allowing the application the flexibility of choosing when protocol headers are processed. The model also illustrates protocol bypass. Latency is lowered because most of the protocol processing is performed after the application receives the data.; Finally, the dissertation outlines the creation of Connection-less TCP which drastically decreases the memory usage of TCP connections, allowing them to be offloaded in a memory-constrained environment. Connection-less TCP allows connections and demultiplexing to be offloaded onto the NIC even when there are a large number of connections.