Research And Design Of Network Expansion With Multi-Root PCI Express Switch

I/O bus,connecting rich types of I/O devices to host computer,plays key role in modern computers.With continousely rising of I/O bandwidth,I/O bus has evolved from media-sharing bus,such ISA,PCI,etc,to switch-based PCI-Exprss.PCI-E switch is the core component in PCI-E specification,connects numerous I/O devices to host computer through end-to-end fabric,provides privately-owned datapath and bandwidth to each individual devices,and expands more I/O links through multiple switches connected in treelike topology.Along with rapid development of virtulization technology,thousands of cores and virtual machines lodge in one processor,and sharing access those I/O devices once had been privately owned by only one processor or core.This enforces PCI-E switch to support SR-IOV.Now,virtual mechines can be distributed among multiple processor or nodes,and even in the whole network,results in some new technology requirements for PCI-E Switch,such as Multi-Root PCI-E and PCI-E Fabric.So,it is very valuable to do some research on Multi-Root PCI-E and scalable PCI-E Fabric.In this paper,we introduce the evolution process of PCI and PCI-Express specification,and discuss some near-future trends of PCI-E developmenting such as function virtualization,multi-root supporting and shifting to system interconnection.We focus on some implementation strateges and technologies for multiroot PCI-E Switch and PCI-E Fabric in this paper.First,we designed a framework for probing and sensing PCI-E devices and functions in the fabric based on local internal proxy and global probe network.Second,based on this framework,we implement a mechanism to allocate devcies and functions to all nodes in the same PCI-E fabric.Third,we implement a Central Buffer and TILE based switching microarchitecture inside the PCI-E Switch,some design detail on the DAMQ mechanism for CB management are discussed.CB based architecture can provide high throughput and TILE architecture makes both front-end and back-end design more feasible.Finally,we analysis how to expand the PCI-E switch into a PCI-E based switching fabric,discuss some limitations on the scale of PCI-E fabric,we wish these discusses can provide some reference value to future PCI-E interconnection fabric design.