From 6e0231074111ff1fda0b2de1765e1988fda2e9c9 Mon Sep 17 00:00:00 2001
From: Brendan Cunningham <bcunningham@cornelisnetworks.com>
Date: Fri, 7 May 2021 16:33:35 -0400
Subject: [PATCH] README: add section about OMPI 4.1.x, OFI BTL, and high-PPN
jobs.
Signed-off-by: Brendan Cunningham <bcunningham@cornelisnetworks.com>
---
README | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git a/README b/README
index 7990555..2961534 100644
--- a/README
+++ b/README
@@ -67,7 +67,6 @@ Contains the following sections:
- INSTALLING
* INSTALLING USING MAKEFILE
* INSTALLING USING EITHER YUM OR DNF
-- TESTING
- RELATED SOFTWARE TO PSM2
- SUPPORTING DOCUMENTATION
@@ -251,6 +250,24 @@ libraries available on them. Open MPI provides a standard configure, make and
make install mechanism which will detect and build the relevant PSM2 network
modules for Open MPI once the header and runtime files are detected.
+Open MPI 4.1.x, OFI BTL, and high PPN jobs
+----------------
+Open MPI added the OFI BTL for one-sided communication. On an OPA fabric, the
+OFI BTL may use the PSM2 OFI provider underneath. If PSM2 is in-use as both
+the MTL (directly or via OFI) and the BTL (via OFI), then each rank in the
+Open MPI job will require two PSM2 endpoints and PSM2 context-sharing will
+be disabled.
+
+In this case, total number of PSM2 ranks on a node can be no more than:
+ (num_hfi * num_user_contexts)/2
+Where num_user_contexts is typically equal to the number of physical CPU
+cores on that node.
+
+If your job does not require an inter-node BTL (e.g. OFI), then you can
+disable the OFI BTL in one of two ways:
+ 1. When building Open MPI, specify '--with-ofi=no' when you run 'configure'.
+ 2. When running your Open MPI job, add '-mca btl self,vader'.
+
MVAPICH2 support
----------------
MVAPICH2 supports PSM2 transport for optimized communication on HFI hardware.
--
2.32.0