ARTDAQSupervisor Error Messages if artdaq-daqinterface crashes

[corrodis]

We'll need to improve the error handling of ARTDAQSupervisor if there is an issue in artdaq-daqintercace.

Below I post what OTSDAQ is showing and what the console is showing. From these error messages, for me as a user, its not clear what the problem is at all. I did change something in the artdaq setup but if that causes issues we'll need to point the user to the issue and not just flood them in error messages.

OTSDAQ message

11/24/2025 3:42:53pm :
|At client: /WebPath/html/StateMachine.html:1118
State Machine Error:

|GatewaySupervisor: otsdaq/otsdaq/GatewaySupervisor/GatewaySupervisor.cc:4481
Failure performing transition from H-Halted to C-Configured.

Exception:
|GatewaySupervisor: otsdaq/otsdaq/GatewaySupervisor/GatewaySupervisor.cc:6183
Received error message from Supervisor instance = 'ARTDAQSupervisor' [LID=301] in Context 'ARTDAQ' [URL=http://mu2e-trk-01.fnal.gov:3088].

 Error Message =
|ARTDAQSupervisor:ARTDAQSupervisor        <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623
Error was caught in configuring thread:
|ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049
Error was caught while configuring:
|ARTDAQSupervisor-ArtdaqSupervisorCrv        <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947
Error calling recover transition!!!!


|FSM-ARTDAQSupervisor:ARTDAQSupervisor        <> : otsdaq/otsdaq/FiniteStateMachine/RunControlStateMachine.cc:696
Run Control Message Handling Failed with command 'Configure':
|ARTDAQSupervisor:ARTDAQSupervisor        <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623
Error was caught in configuring thread:
|ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049
Error was caught while configuring:
|ARTDAQSupervisor-ArtdaqSupervisorCrv        <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947
Error calling recover transition!!!!


 
|ARTDAQSupervisor:ARTDAQSupervisor        <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623
Error was caught in configuring thread:
|ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049
Error was caught while configuring:
|ARTDAQSupervisor-ArtdaqSupervisorCrv        <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947
Error calling recover transition!!!!



...
11/24/2025 3:42:53pm (Warning):
|At client: /WebPath/html/StateMachine.html:1063
State was changed unexpectedly from Configured to Failed. (Likely there was a error transitioning a Supervisor)

Console log:

Error (24Nov25.15:42:51) ARTDAQSupervisor.cc: ARTDAQSupervisor-ArtdaqSupervisorCrv    <> |ARTDAQSupervisor-ArtdaqSupervisorCrv   <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |    Error calling recover transition!!!!
Error (24Nov25.15:42:51) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |         Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv  <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |         Error calling recover transition!!!!
Error (24Nov25.15:42:52) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:42:52) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:42:52) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor:ARTDAQSupervisor       <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |        Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |         Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv  <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |         Error calling recover transition!!!!
Error (24Nov25.15:42:52) FSM: FSM-ARTDAQSupervisor:ARTDAQSupervisor   <> |FSM-ARTDAQSupervisor:ARTDAQSupervisor        <> : otsdaq/otsdaq/FiniteStateMachine/RunControlStateMachine.cc:696 |   Run Control Message Handling Failed with command 'Configure': |ARTDAQSupervisor:ARTDAQSupervisor         <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |      Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |   Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv        <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |   Error calling recover transition!!!!  |ARTDAQSupervisor:ARTDAQSupervisor      <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |     Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |        Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv       <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |        Error calling recover transition!!!!
Error (24Nov25.15:42:52) GatewaySupervisor: |GatewaySupervisor: otsdaq/otsdaq/GatewaySupervisor/GatewaySupervisor.cc:6150 |         Error! Gateway Supervisor can NOT Configure Supervisor instance = 'ARTDAQSupervisor' [LID=301] in Context 'ARTDAQ' [URL=http://mu2e-trk-01.fnal.gov:3088]. ConfigureFailed
Error (24Nov25.15:42:52) GatewaySupervisor: |GatewaySupervisor: otsdaq/otsdaq/GatewaySupervisor/GatewaySupervisor.cc:6183 |         Received error message from Supervisor instance = 'ARTDAQSupervisor' [LID=301] in Context 'ARTDAQ' [URL=http://mu2e-trk-01.fnal.gov:3088].  Error Message = |ARTDAQSupervisor:ARTDAQSupervisor    <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |       Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |      Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv         <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |      Error calling recover transition!!!! |FSM-ARTDAQSupervisor:ARTDAQSupervisor    <> : otsdaq/otsdaq/FiniteStateMachine/RunControlStateMachine.cc:696 |       Run Control Message Handling Failed with command 'Configure': |ARTDAQSupervisor:ARTDAQSupervisor     <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |  Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |       Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv    <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |       Error calling recover transition!!!!  |ARTDAQSupervisor:ARTDAQSupervisor  <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |         Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |    Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv   <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |    Error calling recover transition!!!!
Error (24Nov25.15:42:52) GatewaySupervisor: |GatewaySupervisor: otsdaq/otsdaq/GatewaySupervisor/GatewaySupervisor.cc:4481 |          Failure performing transition from H-Halted to C-Configured. Exception: |GatewaySupervisor: otsdaq/otsdaq/GatewaySupervisor/GatewaySupervisor.cc:6183 |  Received error message from Supervisor instance = 'ARTDAQSupervisor' [LID=301] in Context 'ARTDAQ' [URL=http://mu2e-trk-01.fnal.gov:3088].  Error Message = |ARTDAQSupervisor:ARTDAQSupervisor       <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |        Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |         Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv  <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |         Error calling recover transition!!!! |FSM-ARTDAQSupervisor:ARTDAQSupervisor     <> : otsdaq/otsdaq/FiniteStateMachine/RunControlStateMachine.cc:696 |  Run Control Message Handling Failed with command 'Configure': |ARTDAQSupervisor:ARTDAQSupervisor      <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |     Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |        Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv       <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |        Error calling recover transition!!!!  |ARTDAQSupervisor:ARTDAQSupervisor     <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |  Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |       Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv    <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |       Error calling recover transition!!!!
Error (24Nov25.15:42:52) FSM: FSM-GatewaySupervisor   <> Unexpected Failure state for GatewaySupervisor is Failed
Error (24Nov25.15:42:52) FSM: FSM-GatewaySupervisor   <> Error message was as follows: |GatewaySupervisor: otsdaq/otsdaq/GatewaySupervisor/GatewaySupervisor.cc:4481 |          Failure performing transition from H-Halted to C-Configured. Exception: |GatewaySupervisor: otsdaq/otsdaq/GatewaySupervisor/GatewaySupervisor.cc:6183 |  Received error message from Supervisor instance = 'ARTDAQSupervisor' [LID=301] in Context 'ARTDAQ' [URL=http://mu2e-trk-01.fnal.gov:3088].  Error Message = |ARTDAQSupervisor:ARTDAQSupervisor       <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |        Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |         Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv  <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |         Error calling recover transition!!!! |FSM-ARTDAQSupervisor:ARTDAQSupervisor     <> : otsdaq/otsdaq/FiniteStateMachine/RunControlStateMachine.cc:696 |  Run Control Message Handling Failed with command 'Configure': |ARTDAQSupervisor:ARTDAQSupervisor      <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |     Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |        Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv       <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |        Error calling recover transition!!!!  |ARTDAQSupervisor:ARTDAQSupervisor     <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:623 |  Error was caught in configuring thread: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1049 |       Error was caught while configuring: |ARTDAQSupervisor-ArtdaqSupervisorCrv    <> : otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:947 |       Error calling recover transition!!!!
Error (24Nov25.15:42:53) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:42:54) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:42:55) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:42:55) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:42:56) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:42:57) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:42:58) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:42:58) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:42:59) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:43:00) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:43:01) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:43:01) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:43:02) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:43:03) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:43:04) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:43:04) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:43:05) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:43:05) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:43:07) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:43:07) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:43:08) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]
Error (24Nov25.15:43:09) ARTDAQSupervisor.cc: ARTDAQSupervisor:ARTDAQSupervisor       <> |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1736 |  Error calling artdaq_process_info function:
Error (24Nov25.15:43:10) ARTDAQSupervisor.cc: |ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:1692 |         Attempt n 5. Error calling 'state' function from getDAQState_() - here was the value: |StringMacros.cc: otsdaq/otsdaq/Macros/StringMacros.cc:1457 |      ots::stackTrace: [1] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::getDAQState_()+0xab8 [0x7fe5fc585b48] [2] //home/mu2ecrv/ots_v8_00_00/local/install/lib/libARTDAQSupervisor.so : ots::ARTDAQSupervisor::daqinterfaceRunner_()+0x252 [0x7fe5fc591272] [3] /home/mu2e/spack_areas/art-suite-s132/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path/linux-almalinux9-x86_64_v3/gcc-11.5.0/gcc-13.1.0-byfvucekqqajdjtlhhxxnpcjgtuumjsh/lib64/libstdc++.so.6 : +0xe93e3 [0x7fe60436e3e3] [4] /lib64/libc.so.6 : +0x8a19a [0x7fe603e8a19a] [5] /lib64/libc.so.6 : +0x10f240 [0x7fe603f0f240]

[corrodis]

I found the information that I want to see in this case in the OTSDAQ log file of the ARTDAQSupervisor context:

my state is 'stopped'
setdaqcomps called with BoardReader
getting state for component name 'DAQInterface', my name is 'DAQInterface'
  File "/home/mu2ecrv/ots_v8_00_00/local/install/rc/control/component.py", line 101, in state
    traceback.print_stack()
my state is 'stopped'
|24-Nov-2025 15:42:45 CST:Info:ARTDAQSupervisor.cc: otsdaq/otsdaq/ARTDAQSupervisor/ARTDAQSupervisor.cc:924 |    ARTDAQSupervisor-ArtdaqSupervisorCrv    <> Calling do_boot

Mon Nov 24 15:42:46 CST 2025: BOOT transition underway

Subsystem 2: subsystem source(s): None, subsystem destination: None


Subsystem 3: subsystem source(s): ['3'], subsystem destination: 3

BoardReader          at mu2e-trk-01.fnal.gov:19100, part of subsystem 2, has rank 0
EventBuilderCrv      at mu2e-trk-01.fnal.gov:19101, part of subsystem 2, has rank 1
logger0              at mu2e-dl-01-data.fnal.gov:19102, part of subsystem 2, has rank 2
dispatcher0          at mu2e-crv-01.fnal.gov:10500, part of subsystem 3, has rank 3

On randomly selected node (mu2e-dl-01-data.fnal.gov), will confirm that the DAQ setup script
/home/mu2ecrv/ots_v8_00_00/spack/../.rte_crv_setup_ots.sh
doesn't return a nonzero value when sourced...
STDOUT:

STDERR:
Warning: untrusted X11 forwarding setup failed: xauth key data not generated

done (2.3 seconds).

Nonzero return value (255) resulted when trying to run the following on host mu2e-crv-01.fnal.gov:
mkdir -p -m 0775 /home/mu2ecrv/ots_v8_00_00/Data_crv/Logs
mkdir -p -m 0775 /home/mu2ecrv/ots_v8_00_00/Data_crv/Logs/pmt
mkdir -p -m 0775 /home/mu2ecrv/ots_v8_00_00/Data_crv/Logs/boardreader
mkdir -p -m 0775 /home/mu2ecrv/ots_v8_00_00/Data_crv/Logs/eventbuilder
mkdir -p -m 0775 /home/mu2ecrv/ots_v8_00_00/Data_crv/Logs/dispatcher
mkdir -p -m 0775 /home/mu2ecrv/ots_v8_00_00/Data_crv/Logs/datalogger
mkdir -p -m 0775 /home/mu2ecrv/ots_v8_00_00/Data_crv/Logs/routingmanager

STDOUT output:

STDERR output:
ssh: connect to host mu2e-crv-01.fnal.gov port 22: No route to host


Returned value of 255 suggests that the ssh call to mu2e-crv-01.fnal.gov

[corrodis]

I have another example here. In OTSDAQ I see this

The information I need and I think the user needs to know without digging into the log files can be found in the the artdaq context log file:

Sending start transition to artdaq processes...
Thu Dec 04 08:05:58.293037763: Sending transition to dispatcher0
Thu Dec 04 08:05:58.340021281: Sending transition to logger0
Thu Dec 04 08:06:04.391971805: Sending transition to EventBuilderCrv
Thu Dec 04 08:06:04.763923984: Sending transition to BoardReader
done (8.6 seconds).

BoardReader at mu2e-trk-01.fnal.gov:19100, after 1.1 seconds returned string was:
Success

EventBuilderCrv at mu2e-trk-01.fnal.gov:19101, after 0.4 seconds returned string was:
Success

logger0 at mu2e-dl-01-data.fnal.gov:19102, after 6.1 seconds returned string was:
Error starting logger0 for run number 120185.

dispatcher0 at mu2e-trk-01-data.fnal.gov:10500, after 0.0 seconds returned string was:
Success


Unexpected status message from process logger0 at
mu2e-dl-01-data.fnal.gov:19102: "Error starting logger0 for run number
120185."
See logfile mu2e-dl-01-data.fnal.gov:/home/mu2ecrv/ots_v8_00_00/Data_crv/Logs/logger0-mu2e-dl-01-19102/logger0-mu2e-dl-01-19102-20251204080544-3916725.log for details