|
 |
cb411c |
From 6e0231074111ff1fda0b2de1765e1988fda2e9c9 Mon Sep 17 00:00:00 2001
|
|
|
cb411c |
From: Brendan Cunningham <bcunningham@cornelisnetworks.com>
|
|
|
cb411c |
Date: Fri, 7 May 2021 16:33:35 -0400
|
|
|
cb411c |
Subject: [PATCH] README: add section about OMPI 4.1.x, OFI BTL, and high-PPN
|
|
|
cb411c |
jobs.
|
|
|
cb411c |
|
|
|
cb411c |
Signed-off-by: Brendan Cunningham <bcunningham@cornelisnetworks.com>
|
|
|
cb411c |
---
|
|
|
cb411c |
README | 19 ++++++++++++++++++-
|
|
|
cb411c |
1 file changed, 18 insertions(+), 1 deletion(-)
|
|
|
cb411c |
|
|
|
cb411c |
diff --git a/README b/README
|
|
|
cb411c |
index 7990555..2961534 100644
|
|
|
cb411c |
--- a/README
|
|
|
cb411c |
+++ b/README
|
|
|
cb411c |
@@ -67,7 +67,6 @@ Contains the following sections:
|
|
|
cb411c |
- INSTALLING
|
|
|
cb411c |
* INSTALLING USING MAKEFILE
|
|
|
cb411c |
* INSTALLING USING EITHER YUM OR DNF
|
|
|
cb411c |
-- TESTING
|
|
|
cb411c |
- RELATED SOFTWARE TO PSM2
|
|
|
cb411c |
- SUPPORTING DOCUMENTATION
|
|
|
cb411c |
|
|
|
cb411c |
@@ -251,6 +250,24 @@ libraries available on them. Open MPI provides a standard configure, make and
|
|
|
cb411c |
make install mechanism which will detect and build the relevant PSM2 network
|
|
|
cb411c |
modules for Open MPI once the header and runtime files are detected.
|
|
|
cb411c |
|
|
|
cb411c |
+Open MPI 4.1.x, OFI BTL, and high PPN jobs
|
|
|
cb411c |
+----------------
|
|
|
cb411c |
+Open MPI added the OFI BTL for one-sided communication. On an OPA fabric, the
|
|
|
cb411c |
+OFI BTL may use the PSM2 OFI provider underneath. If PSM2 is in-use as both
|
|
|
cb411c |
+the MTL (directly or via OFI) and the BTL (via OFI), then each rank in the
|
|
|
cb411c |
+Open MPI job will require two PSM2 endpoints and PSM2 context-sharing will
|
|
|
cb411c |
+be disabled.
|
|
|
cb411c |
+
|
|
|
cb411c |
+In this case, total number of PSM2 ranks on a node can be no more than:
|
|
|
cb411c |
+ (num_hfi * num_user_contexts)/2
|
|
|
cb411c |
+Where num_user_contexts is typically equal to the number of physical CPU
|
|
|
cb411c |
+cores on that node.
|
|
|
cb411c |
+
|
|
|
cb411c |
+If your job does not require an inter-node BTL (e.g. OFI), then you can
|
|
|
cb411c |
+disable the OFI BTL in one of two ways:
|
|
|
cb411c |
+ 1. When building Open MPI, specify '--with-ofi=no' when you run 'configure'.
|
|
|
cb411c |
+ 2. When running your Open MPI job, add '-mca btl self,vader'.
|
|
|
cb411c |
+
|
|
|
cb411c |
MVAPICH2 support
|
|
|
cb411c |
----------------
|
|
|
cb411c |
MVAPICH2 supports PSM2 transport for optimized communication on HFI hardware.
|
|
|
cb411c |
--
|
|
|
cb411c |
2.32.0
|
|
|
cb411c |
|