Design and evaluation of a multiple-protocol custom controller for hardware distributed shared memory systems

Minimizing the performance penalty induced by remote memory misses is a major design goal for all hardware distributed shared memory (DSM) systems. To improve performance, designers of commercial hardware DSM systems employ fast and expensive networks or add memory to cache remote data. However, these systems support a single protocol, which forces programmers to resort to data restructuring or page migration to eliminate remote misses when their impact on performance is too high. This dissertation describes a DSM controller that supports multiple protocols and allows the programmer or compiler to control the protocol to use on a per-object basis. Specifically, the DSM controller supports five protocols: (i) sequential consistent write-invalidate without replication in a remote access cache (RAC), (ii) sequential consistent write-invalidate with replication in a RAC, (iii) release consistent write-invalidate with replication in a RAC, (iv) sequential consistent migratory with replication in a RAC, and (v) release consistent update with replication in a RAC. We present the design of the DSM controller that supports these five protocols through a moderate number of extensions to a single protocol custom DSM controller. We also show that the update protocol can be effectively supported on a commodity component based system using a novel mechanism called release state table. We believe that the custom multiple protocol controller provides an interesting middle ground between a single protocol custom controller and a fully programmable controller. Our results indicate that by employing the appropriate protocol for all or some of the data structures in an application, application runtime can be reduced by up to 61% and network bandwidth consumption can be reduced by up to 91% compared to an architecture that supports only a write-invalidate protocol.