Logging Error Handling

Simulation Development Workflow → Logging & Error Handling

Up to date for Platform 0.44.2

Written by Connor Jakubik

---

Logging

Space Teams sims span multiple threads per program, multiple programs per computer, and multiple computers connected to a sim. In order to provide logging that works between all of these, we have made our own log-to-file libraries and utility functions that show messages on-screen.

Log Severities

• Info/INFO: Normal information output.
• Warning/WARN: Something happened that is unusual and could indicate a problem.
• Error/ERROR: There is a problem that requires attention, and the sim accuracy could be compromised as a result.
• Fatal/FATAL: There is a problem that the sim cannot recover from, and the sim should close immediately.
• Chat - Special-use severity indicating this is for a chat message.
• Verbose/VERBOSE: (Not implemented yet in most of our logging tools.) Information that is too verbose to display normally but can be accessed through changing settings.

How Space Teams programs handle logging

On the Compute_Server or any Python System, console output as well as logging to a file are all handled by our custom-made Atomic Logger feature. This is a threadsafe logging framework that has configurable console and file outputs, does colored text output with ANSI escape sequences, and allows many separately-configured Logger objects to write to overlapping sets of output files / pipes with proper mutual-exclusion locking.

On-Screen Log Messages and On-Screen Alerts are typically broadcast across an entire sim, and show up in both the Compute_Server / Python System console / logs as well as on-screen on connected SpaceCRAFT clients.

In the SpaceCRAFT Unreal Engine client, most Pawns will have a UI that includes a scrollable on-screen log view and a place for on-screen alerts to show as well.

Log output directories

For each simulation, log files are generated with names according to this syntax: Sim_(SimConfig filename).log. Variants of this filename are used for each Python System, as well as a filtered log file for each C++ System.

• Compute_Server/Logs/*.log
• Compute_Server/Logs/*_PySys_*.log
• Compute_Server/Logs/Systems/

Sort files by most-recently-modified while browsing these folders to easily find the latest logs.

When a new log file is created that matches the name of an old log file, we move the old copy to the same filepath with -BACKUP-.log instead of .log.

---

Error Handling

Reporting an unexpected error

Please report unexpected errors that you run into during normal use by:

1. Search here (Issues) for previous reports of similar issues
   1. If you find an Issue that seems to match the error you encountered, please write a new comment on that issue describing what you were doing when the error happened and what you expected the program to do instead of the error.
   2. On that comment, also upload your log file(s) and copy the text from the Unreal Crash Reporter if applicable.
2. If you didn't find a similar issue, create a new Issue report here.

Unreal Engine Client (SpaceCRAFT.exe)

In the SpaceCRAFT Unreal Engine client, Fatal errors are typically handled by popping up a modal window with the error message and asking if the user wants to quit or return to the menu. Non-Fatal errors are typically indicated in the on-screen log message console. SpaceCRAFT.exe doesn't have logging-to-file enabled in the User Release at this time (v0.28.0); that only happens in development builds.

In the event of an unhandled exception, a GPU crash, or some other crash, the Unreal Crash Reporter comes up with a window that gives some information about the error that happened. There is an option to submit an error report here, but that does not get sent to SimDynamX; it gets sent to Epic Games. Instead, to report a bug to us, you will need to copy the error log output and include it in your report as mentioned above.

C++ System / Compute_Server

When an unhandled exception or other internal error occurs in a C++ System, unless the -noCatchSystemExceptions argument is passed directly to the run_Server executable, the exception is caught and printed by the Fatal log function. Operating system level crashes such as a Read Access Violation (segfault) are not typically handled as exceptions, and instead immediately terminate the program. We have implemented Structured Exception Handling (SEH) in most areas of the run_Server process and Python API on Windows, which takes those crashes and turns them into exceptions.

Attaching a debugger program such as Visual Studio to the run_Server.exe process before getting past the Sync state of the sim can help to diagnose a crash that happens without any logged message.

Python System

When an unhandled exception or other internal error happens in a Python System, the custom_exception_handler that is set up at the top of the Python System reroutes and prints the traceback and exception text to that Python System's log file. Additionally, the "normal" output of the Python System (print()) is rerouted to buffers in the main Compute_Server process (run_Server). In the event of a presumed crash on one of the Python Systems, this buffer is printed to the main log file. There are some edge cases where a Python System may exit immediately without logging, usually due to a complex error in async Python code created by the user.

How to use the Logger

C++:

// In a System:
logger.Info("My message");
logger.Fatal("Something bad happened, more details should be provided for debugging"); 
// This routes to both the main simulation log file and the System Instance specific log file.

// From a place where no logger exists yet
AtomicLogger::Logger logger;
// TODO example on setting logger options
logger.Info("My message"); // Same usage from here

Python:

# In a System:
st.logger_info("My message")
st.logger_fatal("Something bad happened, more details should be provided for debugging")
# This routes to only the PySys log file for this system.

How to use OnScreen functions

• OnScreenAlert: Send alert that is shown on all app screens, is deduplicated by topic name, and is erased after a few seconds. The deduplication works by the alert changing to show the freshest message per topic in the alert view.
• OnScreenLogMessage: Send message that is shown on all app screens, and stays in a log.

C++:

// During a sim:
SimGlobals::OnScreenLogMessage("My message", "TheCategory", st_global::Severity::Info);
SimGlobals::OnScreenLogMessage("Something bad happened", "TheCategory", st_global::Severity::Error);
SimGlobals::OnScreenAlert("My message", "TheTopic", st_global::Severity::Info);
SimGlobals::OnScreenAlert("Something bad happened", "TheTopic", st_global::Severity::Error);
// All OnScreen messages/alerts cause a log message to be saved on all programs which receive it.

Python:

# During a sim:
st.OnScreenLogMessage("My message", "TheCategory", st.Severity.Info)
st.OnScreenLogMessage("Something bad happened", "TheCategory", st.Severity.Error)
st.OnScreenAlert("My message", "TheTopic", st.Severity.Info)
st.OnScreenAlert("Something bad happened", "TheTopic", st.Severity.Error)