error handling on mcclComm::init

Summary:
Arguably, the capture scope is very wide. The reason is that some of the errors should be crashing quickly, while others should be retried.

I found a really small number of cases, where `std::runtime_error` is used in place of something more suitable, such as an `invalid_argument`. We will follow up to clean these up, so that non-recoverable items would fail fast and recoverable ones are handled by Mccl. But these findings are harmless for initial enablement.

Reviewed By: dboyda

Differential Revision: D85824307

fbshipit-source-id: 1c5b4e647ccb3ee7f2eaad927071dd8f40411b89

Author: arttianezhu

comms/ctran/mapper/tests/CtranMapperTcpdmUT.cc

@@ -35,6 +35,8 @@ class CtranMapperTcpdmTest : public ::testing::Test {
       commRAII.reset();
     } catch (const std::runtime_error& e) {
       GTEST_SKIP() << "TCPDM backend not enabled. Skip test";
+    } catch (const ctran::utils::Exception& e) {
+      GTEST_SKIP() << "TCPDM backend not enabled. Skip test";
     }
   }
   void TearDown() override {

comms/ctran/mapper/tests/CtranMapperUT.cc

@@ -112,7 +112,15 @@ TEST(CtranMapperUT, EnableBackendWithMCCLBackendOverride) {
 TEST(CtranMapperUT, EnableBackendThroughCVARsWithTCPandIB) {
   setenv("NCCL_CTRAN_BACKENDS", "nvl, ib, socket, tcpdm", 1);
   ncclCvarInit();
-  EXPECT_THROW(createDummyCtranComm(), std::runtime_error);
+  std::optional<std::exception> ex;
+  try {
+    createDummyCtranComm();
+  } catch (const std::runtime_error& e) {
+    ex = e;
+  } catch (const ctran::utils::Exception& e) {
+    ex = e;
+  }
+  ASSERT_TRUE(ex.has_value());
 }
 
 TEST(CtranMapperUT, BackendEnum) {