Enable building and running on macOS

[taalexander]

This PR adds macOS support for CUDA-Q, addressing platform-specific differences in linking, symbol visibility, shell compatibility, and library handling. The changes should enable building and running CUDA-Q on macOS with Apple Silicon (arm64) and Intel (x86_64) architectures but has currently only been tested on x86.

This is a large PR and full Python support also requires Python wheels and CI enablement. I have structured this PR such that it should not impact existing Linux builds. I recommend we treat this as phase one and merge after review/passing CI. We will then follow up with Python wheel and CI PRs to complete support.

PRs:

• Enable building and running on macOS #3711
• Python wheel support for macOS
• CI support for macOS

I've tried my best to summarize the contents of the PR below:

1. Build System (CMake)

Platform Detection & Configuration

• Added macOS sysroot detection via xcrun --show-sdk-path for C++ stdlib headers
• Configured platform-appropriate linker flags (--no-as-needed for Linux, alternatives for macOS)
• Added CUDAQ_LIBCXX_PATH and CUDAQ_SYSROOT_PATH configuration for cross-platform header discovery
• Updated rpath handling: @executable_path on macOS vs $ORIGIN on Linux
• Changed CMAKE_INSTALL_RPATH to use semicolon separators on macOS

Library Linking Changes

• Moved MLIR/LLVM dependencies from PUBLIC to PRIVATE in cudaq-mlir-runtime to reduce symbol visibility issues due to two-level namespace
• Updated plugin extension handling: .dylib on macOS vs .so on Linux
• Added platform-conditional linking for circular dependencies (--start-group not available on Apple ld)
• Force two-level namespace linking for cudaq-common to prevent symbol collisions with OpenSSL

New cudaq-utils Library

Created a new low-level utilities library which resolves a circular dependency where cudaq-operator needs complex_matrix functions but is built before libcudaq

---

2. Two-Level Namespace Workarounds

macOS uses two-level namespace linking by default, where symbols are bound to specific libraries. This causes issues with LLVM/MLIR's static initializer pattern (PassRegistry, TargetRegistry, cl::Options).

Workarounds Implemented

Workaround	Location	Purpose
flat_namespace linker flag	CMakeLists.txt	Global symbol visibility
force_load for LLVM CodeGen	cmake/BuildHelpers.cmake	Ensures static initializers run in correct library context
add_lib_loading_macos_workaround	cmake/BuildHelpers.cmake	Helper for Python extension targets to ensure proper library loading order
Symbol unexport list	lib/Support/Config/CMakeLists.txt	Hides LLVM/MLIR symbols from CUDAQTargetConfigUtil to prevent symbol collisions
Explicit InitializeNativeTarget	CUDAQuantumExtension.cpp	macOS-only target registration for Python extension to workaround issues where the targets were registered to the wrong registry copy
Execution manager override API	execution_manager.h	Allows explicit manager setting across library boundaries to manage behaviour with execution manager default symbol resolution

Future Removal Pathway

The targeted fixes (PRIVATE linking, force_load, unexport lists) should eventually allow removing the global flat_namespace flag. To test removal:

1. Comment out add_link_options("-Wl,-flat_namespace") in CMakeLists.txt
2. Rebuild and run full test suite
3. If symbol-not-found errors occur, identify which library needs additional force_load

However, there is an immediate blocker with cudaq-mlir-runtime and cudaq libraries both containing pass registries and both containing passes. To resolve this would likely require centralizing these registries would likely require larger library restructuring/registry development and I consider this work for a follow up PR.

---

3. Platform Portability Fixes

Type Size Differences

• unsigned long is 8 bytes on macOS/arm64 but 4 bytes on some Linux systems so we switch usages to std::uint64_t
• Updated CCTypes.cpp to use explicit size types where needed

Library Path Handling

• TargetConfig.cpp: Handle .dylib vs .so extensions
• fixup-linkage.cpp: Added handling for define weak linkage (macOS clang emits some functions with weak linkage)

Shell Compatibility (POSIX)

Replaced bash commands with POSIX-compatible alternatives:

• |& → 2>&1 | for stderr piping
• Fixed mktemp template usage (macOS requires XXXXXX suffix)
• Updated shebang and array handling in shell scripts

Standard Library Differences

• std::vector<bool> has different internal layout between libc++ and libstdc++
• Added explicit extern char **environ declaration in MQPUUtils.cpp (POSIX requires explicit declaration on macOS)

Other

• Updated Stim CMakeLists.txt to use platform-appropriate symbol hiding syntax

---

4. Third-Party Patches

Added patches in tpls/customizations/ for compatibility:

Patch	File	Purpose
LLVM idempotent option category	llvm/idempotent_option_category.diff	Makes cl::OptionCategory registration idempotent to handle multiple LLVM copies registering the same category (avoids assertion failures)
Pybind11 LTO flag fix	pybind11/pybind11Common.cmake.diff	Fixes pybind/pybind11#5098 - incorrect -flto= flag generation for Clang

Xtensor xio.hpp Workaround

• Removed #include <xtensor/xio.hpp> in molecule.cpp and replaced with manual printing
• Workaround for clang 17-18 template ambiguity with svector's rebind_container (LLVM #91504)

---

5. (Pre-existing) Bug Fixes

Note most of these would have been caught by static code analysis. The majority of the bugs were likely a result of more aggressive allocator on OSX.

File	Fix
RegToMem.cpp	Added proper WalkResult return after op->erase() to prevent iterator invalidation
LoopUnrollPatterns.inc	Fixed iterator handling in loop unrolling pattern
ResetBeforeReuse.cpp	Fixed stale pointer bug caused by canonicalization in Quantinuum pass
CombineMeasurements.cpp	Clarified success return for erased unused measurements

---

6. Python Bindings

• Added #ifdef __APPLE__ conditional for InitializeNativeTarget calls
• Updated CMake to use add_lib_loading_macos_workaround for Python extension targets
• Fixed Python virtual environment stdlib availability on macOS

---

7. Documentation & Developer Setup

• Updated Dev_Setup.md for developer environment setup
• Added requirements-dev.txt for Python development dependencies
• Updated Building.md with platform-specific notes
• Updated scripts/install_toolchain.sh and scripts/install_prerequisites.sh for macOS build instructions
• Updated scripts/build_cudaq.sh with improved macOS build support
• Changed OpenSSL build to use CMake on macOS to avoid pkg-config resolution issues with flat namespace

---

8. Test Updates

• Updated test RUN lines to use platform-appropriate flags (calling_convention.cpp, infinite_loop.cpp, kernel exec transform tests)
• Replaced .so with %cudaq_plugin_ext substitution for cross-platform tests
• Added DISCOVERY_TIMEOUT 120 to backend unit tests (likely just required for my slow machine)
• Split qvector_init_from_vector.cpp to separate large array test (qvector_init_large_array.cpp) which is skipped on macOS due to stack size
• Fixed cudaq-qpud linking to enforce two-level namespace for braket backend tests

Known Limitations

Stack Size

macOS has a smaller default stack size (8MB) compared to Linux. Some tests with large stack allocations (e.g., large array initializations) may fail. The qvector_init_large_array.cpp test is currently skipped on macOS for this reason. Future work may address this via ulimit -s or code refactoring.

flat_namespace

The flat_namespace linker flag can cause symbol collisions with system libraries. We should work toward removing this in a follow-up PR.

LLVM cl::OptionCategory Duplicate Registration (Requires LLVM Patch)

Both libcudaq and cudaq-mlir-runtime link LLVM and therefore each contain their own copy of LLVM's cl::OptionCategory static globals. When both libraries are loaded, LLVM's default behavior asserts on duplicate category names.

The idempotent_option_category.diff patch makes registration idempotent, allowing the same category to be registered multiple times without assertion failures. This is a workaround—the proper fix would be to restructure the libraries so only one contains LLVM command-line infrastructure, but that requires more significant refactoring.

---

Testing

All tests passing on x86 with ctest --output-on-failure

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[sacpis]

/ok to test 639dac7

Command Bot: Processing...

[sacpis]

/ok to test 5965f42

Command Bot: Processing...

[sacpis]

/ok to test d6f87f9

Command Bot: Processing...

[sacpis]

/ok to test 5b5f4b3

Command Bot: Processing...

[taalexander]

/ok to test f0b2fd3

[github-actions]

CUDA Quantum Docs Bot: A preview of the documentation can be found here.

[taalexander]

/ok to test 8241423

Command Bot: Processing...

[github-actions]

CUDA Quantum Docs Bot: A preview of the documentation can be found here.