StarT-X: A PCI Arctic Network Interface Unit
The StarT-X Arctic Network Interface Unit (NIU) is a PCI card with
hardware support for user-level message passing in a cluster of
workstations interconnected by an Arctic
Switch Fabric.
This is one of the latest developments in a series of
StarT
parallel processing cluster projects. StarT-X NIU is a direct
follow-on to
StarT-Jr's FUNi
and carries over many of the same message-passing mechanisms.
However, StarT-X NIU achieves significantly better performance than
FUNi, in both bandwidth and latency, by handling the critical paths in
hardware rather than relying on embedded processing.
Salient features of StarT-X NIU are:
-
Three message-passing modes:
-
Physical I/O (PIO), memory-mapped register interface for message
passing
-
FUNi-style message-passing queues through the host DRAM
-
DMA transfer with automatic packetization of large data blocks
- Two priorities of messages
- Per packet selection of point-to-point network FIFO or
non-FIFO ordered delivery to utilize randomized uproute in the
Arctic Switch Fabric's Fat Tree network.
A more detailed description of the StarT-X project is available as slides
from a presentation given to Xolas users in April, 1997.
Hardware Status (Comments: This status section is (very) outdated. The activities described were completed succesfully in 1998.)
The first four StarT-X NIU's have been successfully tested with the
prototype Arctic Switch Fabric to interconnect a cluster of four Intel
Pentium PCs. Bandwidth at 70 MB/sec has been observed for DMA
transfers between two Intel Pentium PCs with 430FX chipsets.
User-to-user latency is less than 4 usec. StarT-X's performance is
only limited by the the performance of the PCI bus on the host system.
StarT-X NIU works with any host platform that supports the 32-bit
33MHz PCI bus standard. StarT-X NIU's are currently installed in
LCS's Xolas Cluster of 9
8-processors SUN Enterprise 5000 SMP's. We plan to produce up to 50
StarT-X NIU's to interconnect the other LCS clusters,
including the Pleiades Cluster (7x4 DEC Alpha's), and the pending
donation of 20+ Intel four-way SMP's.
Software Status (Comments: This status section is (very) outdated. The activities described were completed succesfully in 1998.)
Device drivers are available for Intel and SPARC platforms running
Linux and Solaris, respectively. The existing network management
software is functional, but extensions to its functionality and
user interface are still being added. JAM, an ultra
light-weight communication library, provides a mix of C function calls
and macros to reflect the hardware-supported message-passing
mechanisms with minimal overhead. JAM is ideal for building
higher-level libraries. Collaborative efforts to port
MPI
and distributed
Cilk
are currently
underway.
A StarT-X PCI NIU Card
Two StarT-X Cards in a SUN Enterprise PCI I/O Board
Four Arctic Test Boards (A 4-way switch)
LCS's Xolas Cluster of 9 SUN E5000 SMP's with 8 processors each
Arctic cables (gray ribbons about $100 each) exiting the back panel of a SUN E5000 with 4 StarT-X cards
James C. Hoe
(jhoe+www at ece_cmu_edu)
Keywords: StarT-X, StarT-Jr, FUNi, Arctic, network of workstations,
parallel processing, network interface, user-level, interprocessor
communication