Affiliations: Department of Computer Science, ETH Zurich, Zurich, Switzerland. E-mails: {robertge, bestam, htor}@inf.ethz.ch
Note: [] The author performed much of the implementation during an internship at UIUC/NCSA while the analysis and documentation was performed during a scientific visit at ETH Zurich. The author's primary e-mail: gerro@hrz.tu-chemnitz.de
Note: [] M. Besta is the recipient of the 2013 Google European Doctoral Fellowship in Parallel Computing.
Note: [] Corresponding author: Torsten Hoefler, Department of Computer Science, ETH Zurich, Universitätstr. 6, 8092 Zurich, Switzerland. E-mail: htor@inf.ethz.ch
Abstract: Modern interconnects offer remote direct memory access (RDMA) features. Yet, most applications rely on explicit message passing for communications albeit their unwanted overheads. The MPI-3.0 standard defines a programming interface for exploiting RDMA networks directly, however, it's scalability and practicability has to be demonstrated in practice. In this work, we develop scalable bufferless protocols that implement the MPI-3.0 specification. Our protocols support scaling to millions of cores with negligible memory consumption while providing highest performance and minimal overheads. To arm programmers, we provide a spectrum of performance models for all critical functions and demonstrate the usability of our library and models with several application studies with up to half a million processes. We show that our design is comparable to, or better than UPC and Fortran Coarrays in terms of latency, bandwidth and message rate. We also demonstrate application performance improvements with comparable programming complexity.
Keywords: Programming systems, programming models, Remote Memory Access (RMA), Remote Direct Memory Access (RDMA), MPI-3, one sided communication
