Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
e64bb97
Improve test coverage of Python/C++ interface code
greenc-FNAL Dec 19, 2025
4e0957f
Initial plan
Copilot Jan 12, 2026
d3bd321
Add Variant helper and address review comments
Copilot Jan 12, 2026
26fa647
Fix code review comments
Copilot Jan 12, 2026
1abca07
Apply cmake-format fixes
github-actions[bot] Jan 12, 2026
ba1377a
Apply Python linting fixes
github-actions[bot] Jan 12, 2026
e644d7c
Initial plan
Copilot Jan 12, 2026
e2cb860
Fix ruff F722 and mypy errors in vectypes.py by using type aliases wi…
Copilot Jan 12, 2026
49739fe
Simplify metaclass implementation per code review feedback
Copilot Jan 12, 2026
7d71301
Fix CodeQL alert
greenc-FNAL Jan 12, 2026
adbeaea
Apply clang-format fixes
github-actions[bot] Jan 14, 2026
f7e4a81
Fix Python tests and enforce NumPy requirement
greenc-FNAL Jan 14, 2026
ce1c0ac
Apply cmake-format fixes
github-actions[bot] Jan 14, 2026
c6db4ac
More tests to fill gaps
greenc-FNAL Jan 14, 2026
1ce1292
Apply cmake-format fixes
github-actions[bot] Jan 14, 2026
52fa79b
Apply Python linting fixes
github-actions[bot] Jan 14, 2026
fe5971a
Address remaining `ruff` issues
greenc-FNAL Jan 14, 2026
12a20b5
Per Gemini 3 Pro, get GIL when updating ref count
greenc-FNAL Jan 14, 2026
93d4528
Attempt to address CI hangs in `py:badbool` and `py:raise` tests
greenc-FNAL Jan 14, 2026
ae6f729
More coverage improvement
greenc-FNAL Jan 14, 2026
36aa482
Apply Python linting fixes
github-actions[bot] Jan 14, 2026
c6914d7
Apply cmake-format fixes
github-actions[bot] Jan 14, 2026
b5a5c4b
Silence inapposite complaints; remove unused class
greenc-FNAL Jan 14, 2026
9d1ffc5
More hang protection
greenc-FNAL Jan 14, 2026
8112bc6
Extra diagnostics to debug hangs during testing
greenc-FNAL Jan 14, 2026
04ec9f8
More debug logging
greenc-FNAL Jan 15, 2026
b231116
Remove `failing_test_wrap.sh` as unnecessary
greenc-FNAL Jan 15, 2026
63c06bf
Replace unsafe macro call with safe equivalent
greenc-FNAL Jan 15, 2026
a14544d
Remove all diagnostics to see if problems return
greenc-FNAL Jan 15, 2026
30cda8e
Remove diagnostic deadends and other unneeded code
greenc-FNAL Jan 15, 2026
cb3017a
Apply clang-format fixes
github-actions[bot] Jan 15, 2026
f8346fd
Apply cmake-format fixes
github-actions[bot] Jan 15, 2026
d72842b
Armor-plate `WILL_FAIL` tests against false pass
greenc-FNAL Jan 15, 2026
0cf0b27
Remove possibly-problematic initialization check
greenc-FNAL Jan 15, 2026
cc5aa40
Apply cmake-format fixes
github-actions[bot] Jan 15, 2026
e8b0bf9
Further attempts to prevent stalls
greenc-FNAL Jan 15, 2026
e5d1508
Remove diagnostic invocations from coverage workflow
greenc-FNAL Jan 15, 2026
6461df8
Encourage `ctest --test-timeout` to limit impact of stalling tests
greenc-FNAL Jan 15, 2026
2839756
First pass at addressing review comments
greenc-FNAL Jan 15, 2026
25ad760
Restore array-bounds warning deactivation for GCC 15
greenc-FNAL Jan 15, 2026
039db91
Improve Python argument ordering stability
greenc-FNAL Jan 15, 2026
d26a3ce
Apply clang-format fixes
github-actions[bot] Jan 15, 2026
ef6ef47
Make sure types agree with what's in vectypes.py (#10)
knoepfel Jan 15, 2026
dad36de
Apply cmake-format fixes
github-actions[bot] Jan 15, 2026
1931812
Revert unwanted change per review
greenc-FNAL Jan 15, 2026
87a8983
Have CMake report module check results
greenc-FNAL Jan 15, 2026
5cc2d6d
Python AdjustAnnotations class improvements
greenc-FNAL Jan 15, 2026
004e6aa
Apply cmake-format fixes
github-actions[bot] Jan 15, 2026
630e126
Include Python files in coverage change detection
greenc-FNAL Jan 16, 2026
5f36485
Make sure non-test Python code is tested
greenc-FNAL Jan 16, 2026
af9c2a0
Apply Python linting fixes
github-actions[bot] Jan 16, 2026
0edd4d2
Apply cmake-format fixes
github-actions[bot] Jan 16, 2026
fa4b955
Address `ruff` issues
greenc-FNAL Jan 16, 2026
a4c4009
Resolve issues with Python testing and coverage
greenc-FNAL Jan 16, 2026
11d811e
Enable FORM by default in presets
greenc-FNAL Jan 16, 2026
6487281
Temporarily restore packaging workaround pending reconciliation
greenc-FNAL Jan 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,3 +137,59 @@ All Markdown files must strictly follow these markdownlint rules:
- **MD034**: No bare URLs (for example, use a markdown link like `[text](destination)` instead of a plain URL)
- **MD036**: Use # headings, not **Bold:** for titles
- **MD040**: Always specify code block language (for example, use '```bash', '```python', '```text', etc.)

## Development & Testing Workflows

### Build and Test

- **Environment**: Always source `setup-env.sh` before building or testing. This applies to all environments (Dev Container, local machine, HPC).
- **Configuration**:
- **Presets**: Prefer `CMakePresets.json` workflows (e.g., `cmake --preset default`).
- **Generator**: Prefer `Ninja` over `Makefiles` when available (`-G Ninja`).
- **Build**:
- **Parallelism**: Always use multiple cores. Ninja does this by default. For `make`, use `cmake --build build -j $(nproc)`.
- **Test**:
- **Parallelism**: Run tests in parallel using `ctest -j $(nproc)` or `ctest --parallel <N>`.
- **Selection**: Run specific tests with `ctest -R "regex"` (e.g., `ctest -R "py:*"`).
- **Debugging**: Use `ctest --output-on-failure` to see logs for failed tests.
- **Guard against known or suspected stalling tests**: Use `ctest --test-timeout` to set the per-test time limit (e.g. `90`) for 90s, _vs_ the default of 1500s.

### Python Integration

- **Naming**: Avoid naming Python test scripts `types.py` or other names that shadow standard library modules. This causes obscure import errors (e.g., `ModuleNotFoundError: No module named 'numpy'`).
- **PYTHONPATH**: Only include paths that contain user Python modules loaded by Phlex (for example, the source directory and any build output directory that houses generated modules). Do not append system/Spack/venv `site-packages`; `pymodule.cpp` handles CMAKE_PREFIX_PATH and virtual-environment path adjustments.
- **Test Structure**:
- **C++ Driver**: Provides data streams (e.g., `test/python/driver.cpp`).
- **Jsonnet Config**: Wires the graph (e.g., `test/python/pytypes.jsonnet`).
- **Python Script**: Implements algorithms (e.g., `test/python/test_types.py`).
- **Type Conversion**: `plugins/python/src/modulewrap.cpp` handles C++ ↔ Python conversion.
- **Mechanism**: Uses substring matching on type names (for example, `"float64]]"`). This is brittle.
- **Requirement**: Ensure converters exist for all types used in tests (e.g., `float`, `double`, `unsigned int`, and their vector equivalents).
- **Warning**: Exact type matches are required. `numpy.float32` != `float`.

### Coverage Analysis

- **Tooling**: The project uses LLVM source-based coverage.
- **Requirement**: The `phlex` binary must catch exceptions in `main` to ensure coverage data is flushed to disk even when tests fail/crash.
- **Generation**:
- **CMake Targets**: `coverage-xml`, `coverage-html` (if configured).
- **Manual**:
1. Run tests with `LLVM_PROFILE_FILE` set (e.g., `export LLVM_PROFILE_FILE="profraw/%m-%p.profraw"`).
2. Merge profiles: `llvm-profdata merge -sparse profraw/*.profraw -o coverage.profdata`.
3. Generate report: `llvm-cov show -instr-profile=coverage.profdata -format=html ...`

### Local GitHub Actions Testing (`act`)

- **Tool**: Use `act` to run GitHub Actions workflows locally.
- **Configuration**: Ensure `.actrc` exists in the workspace root with the following content to use a compatible runner image:
```text
-P ubuntu-latest=catthehacker/ubuntu:act-latest
```
- **Usage**:
- List jobs: `act -l`
- Run specific job: `act -j <job_name>` (e.g., `act -j python-check`)
- Run specific event: `act pull_request`
- **Troubleshooting**:
- **Docker Socket**: `act` requires access to the Docker socket. In dev containers, this may require specific mount configurations or permissions.
- **Artifacts**: `act` creates a `phlex-src` directory (or similar) for checkout. Ensure this is cleaned up or ignored by tools like `mypy`.

2 changes: 1 addition & 1 deletion .github/workflows/cmake-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ jobs:

echo "➡️ Running tests..."
echo "::group::Running ctest"
if ctest --progress --output-on-failure -j "$(nproc)"; then
if ctest --progress --output-on-failure --test-timeout 90 -j "$(nproc)"; then
echo "::endgroup::"
echo "✅ All tests passed."
else
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/coverage.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ jobs:
file-type: |
cpp
cmake
python

- name: Report detection outcome
run: |
Expand Down Expand Up @@ -161,7 +162,7 @@ jobs:
export LLVM_PROFILE_FILE="$PROFILE_ROOT/%m-%p.profraw"

echo "::group::Running ctest for coverage"
if ctest --progress --output-on-failure -j "$(nproc)"; then
if ctest --progress --output-on-failure --test-timeout 90 -j "$(nproc)"; then
echo "::endgroup::"
echo "✅ All tests passed."
else
Expand Down
26 changes: 15 additions & 11 deletions .gitignore
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't some of these changes suggest the build directory may be at /? Is this ever the case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It pegs the ignore expression to the top-level directory, otherwise it matches in subdirectories also.

Original file line number Diff line number Diff line change
@@ -1,23 +1,27 @@
# Build directories
build/
build-cov/
_build/
*.dir/
phlex-src
phlex-build/
CMakeCache.txt
/phlex-src/
/phlex-build/
/CMakeCache.txt
CMakeFiles/
_deps/
/_deps/
_codeql_detected_source_root

# CMake user-specific presets (not generated by Spack)
CMakeUserPresets.json
/CMakeUserPresets.json

# Coverage reports
coverage.xml
coverage.info
coverage-html/
.coverage-generated/
.coverage-artifacts/
/coverage.profdata
/coverage_*.txt
/coverage.xml
/coverage.info
/coverage-html/
/profraw/
/.coverage-generated/
/.coverage-artifacts/
*.gcda
*.gcno
*.gcov
Expand Down Expand Up @@ -45,4 +49,4 @@ __pycache__/
.DS_Store
# act (local workflow testing)
.act-artifacts/
.secrets
.secrets
25 changes: 16 additions & 9 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,11 @@ project(phlex VERSION 0.1.0 LANGUAGES CXX)
cet_cmake_env()
# ##############################################################################

# Set CI/test timeouts to a conservative value to avoid long stalls in CI.
# Use cache variables so generated CTest/Dart files pick this up when configured.
set(DART_TESTING_TIMEOUT 90 CACHE STRING "Timeout (s) for Dart/CTest runs")
set(CTEST_TEST_TIMEOUT 90 CACHE STRING "Per-test timeout (s) for CTest")

# Make tools available
FetchContent_MakeAvailable(Catch2 GSL mimicpp)

Expand All @@ -70,13 +75,13 @@ add_compile_options(
)

if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
if(
CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL "14.1"
AND CMAKE_COMPILER_VERSION VERSION_LESS "15"
)
# GCC 14.1 issues many false positives re. array-bounds and
# stringop-overflow
add_compile_options(-Wno-array-bounds -Wno-stringop-overflow)
if(CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL "14.1")
if(CMAKE_CXX_COMPILER_VERSION VERSION_LESS "15")
add_compile_options(-Wno-stringop-overflow)
endif()
if(CMAKE_CXX_COMPILER_VERSION VERSION_LESS "16")
add_compile_options(-Wno-array-bounds)
endif()
endif()
endif()

Expand Down Expand Up @@ -108,7 +113,8 @@ if(ENABLE_TSAN)
-g
-O1
# Ensure no optimizations interfere with TSan
"$<$<COMPILE_LANG_AND_ID:CXX,GNU>:-fno-omit-frame-pointer -fno-optimize-sibling-calls>"
"$<$<COMPILE_LANG_AND_ID:CXX,GNU>:-fno-omit-frame-pointer>"
"$<$<COMPILE_LANG_AND_ID:CXX,GNU>:-fno-optimize-sibling-calls>"
)
add_link_options(-fsanitize=thread)
else()
Expand All @@ -130,7 +136,8 @@ if(ENABLE_ASAN)
-g
-O1
# Ensure no optimizations interfere with ASan
"$<$<COMPILE_LANG_AND_ID:CXX,GNU>:-fno-omit-frame-pointer -fno-optimize-sibling-calls>"
"$<$<COMPILE_LANG_AND_ID:CXX,GNU>:-fno-omit-frame-pointer>"
"$<$<COMPILE_LANG_AND_ID:CXX,GNU>:-fno-optimize-sibling-calls>"
)
add_link_options(-fsanitize=address)
else()
Expand Down
1 change: 1 addition & 0 deletions CMakePresets.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"name": "default",
"hidden": false,
"cacheVariables": {
"PHLEX_USE_FORM": "ON",
"CMAKE_EXPORT_COMPILE_COMMANDS": "YES",
"CMAKE_CXX_STANDARD": "20",
"CMAKE_CXX_STANDARD_REQUIRED": "YES",
Expand Down
5 changes: 5 additions & 0 deletions plugins/python/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,8 @@ target_link_libraries(pymodule PRIVATE phlex::module Python::Python Python::NumP
target_compile_definitions(pymodule PRIVATE NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION)

install(TARGETS pymodule LIBRARY DESTINATION lib)

install(
DIRECTORY python/phlex
DESTINATION lib/python${Python_VERSION_MAJOR}.${Python_VERSION_MINOR}/site-packages
)
56 changes: 56 additions & 0 deletions plugins/python/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Phlex Python Plugin Architecture

This directory contains the C++ source code for the Phlex Python plugin, which enables Phlex to execute Python code as part of its computation graph.

## Architecture Overview

The integration is built on the **Python C API** (not `pybind11`) to maintain strict control over the interpreter lifecycle and memory management.

### 1. The "Type Bridge" (`modulewrap.cpp`)

The core of the integration is the type conversion layer in `src/modulewrap.cpp`. This layer is responsible for:
- Converting Phlex `Product` objects (C++) into Python objects (e.g., `PyObject*`, `numpy.ndarray`).
- Converting Python return values back into Phlex `Product` objects.

**Critical Implementation Detail:**
The type mapping relies on **string comparison** of type names.

- **Mechanism**: The C++ code checks whether `type_name()` contains `"float64]]"` to identify a 2D array of doubles.
- **Brittleness**: This is a fragile contract. If the type name changes (e.g., `numpy` changes its string representation) or if a user provides a slightly different type (e.g., `float` vs `np.float32`), the bridge may fail.
- **Extension**: When adding support for new types, you must explicitly add converters in `modulewrap.cpp` for both scalar and vector/array versions.

### 2. Hybrid Configuration

Phlex uses a hybrid configuration model involving three languages:

1. **Jsonnet** (`*.jsonnet`): Defines the computation graph structure. It specifies:
- The nodes in the graph.
- The Python module/class to load for specific nodes.
- Configuration parameters passed to the Python object.
2. **C++ Driver**: The executable that:
- Parses the Jsonnet configuration.
- Initializes the Phlex core.
- Loads the Python interpreter and the specified plugin.
3. **Python Code** (`*.py`): Implements the algorithmic logic.

### 3. Environment & Testing

Because the Python interpreter is embedded within the C++ application, the runtime environment is critical.

- **PYTHONPATH**: Must be set correctly to include:
- The build directory (for generated modules).
- The source directory (for user scripts).
- Do not append system/Spack `site-packages`; `pymodule.cpp` adjusts `sys.path` based on `CMAKE_PREFIX_PATH` and active virtual environments.
- **Naming Collisions**:
- **Warning**: Do not name test files `types.py`, `test.py`, `code.py`, or other names that shadow standard library modules.
- **Consequence**: Shadowing can cause obscure failures in internal libraries (e.g., `numpy` failing to import because it tries to import `types` from the standard library but gets your local file instead).

## Development Guidelines

1. **Adding New Types**:
- Update `src/modulewrap.cpp` to handle the new C++ type.
- Add a corresponding test case in `test/python/` to verify the round-trip conversion.
2. **Testing**:
- Use `ctest` to run tests.
- Tests are integration tests: they run the full C++ application which loads the Python script.
- Debugging: Use `ctest --output-on-failure` to see Python exceptions.
83 changes: 83 additions & 0 deletions plugins/python/python/phlex/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
"""Phlex Python Utilities.

Call helpers and type annotation tools for the Phlex framework.
"""

import copy
from typing import Any, Callable


class AdjustAnnotations:
"""Wrapper to associate custom annotations with a callable.

This class wraps a callable and provides custom ``__annotations__`` and
``__name__`` attributes, allowing the same underlying function or callable
object to be registered multiple times with different type annotations.

By default, the provided callable is kept by reference, but can be cloned
(e.g. for callable instances) if requested.

Phlex will recognize the "phlex_callable" data member, allowing an unwrap
and thus saving an indirection. To detect performance degradation, the
wrapper is not callable by default.

Attributes:
phlex_callable (Callable): The underlying callable (public).
__annotations__ (dict): Type information of arguments and return product.
__name__ (str): The name associated with this variant.

Examples:
>>> def add(i: Number, j: Number) -> Number:
... return i + j
...
>>> int_adder = AdjustAnnotations(add, {"i": int, "j": int, "return": int}, "iadd")
"""

def __init__(
self,
f: Callable,
annotations: dict[str, str | type | Any],
name: str,
clone: bool | str = False,
allow_call: bool = False,
):
"""Annotate the callable F.

Args:
f (Callable): Annotable function.
annotations (dict): Type information of arguments and return product.
name (str): Name to assign to this variant.
clone (bool|str): If True (or "deep"), creates a shallow (deep) copy
of the callable.
allow_call (bool): Allow this wrapper to forward to the callable.
"""
if clone == "deep":
self.phlex_callable = copy.deepcopy(f)
elif clone:
self.phlex_callable = copy.copy(f)
else:
self.phlex_callable = f
self.__annotations__ = annotations
self.__name__ = name
self._allow_call = allow_call

# Expose __code__ from the underlying callable if available, to aid
# introspection (e.g. by C++ modulewrap).
self.__code__ = getattr(self.phlex_callable, "__code__", None)
self.__defaults__ = getattr(self.phlex_callable, "__defaults__", None)
self.__kwdefaults__ = getattr(self.phlex_callable, "__kwdefaults__", None)

def __call__(self, *args, **kwargs):
"""Raises an error if called directly.

AdjustAnnotations instances should not be called directly. The framework should
extract ``phlex_callable`` instead and call that.

Raises:
AssertionError: To indicate incorrect usage, unless overridden.
"""
assert self._allow_call, (
f"AdjustAnnotations '{self.__name__}' was called directly. "
f"The framework should extract phlex_callable instead."
)
return self.phlex_callable(*args, **kwargs) # type: ignore
5 changes: 5 additions & 0 deletions plugins/python/src/lifelinewrap.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,14 @@ static int ll_clear(py_lifeline_t* pyobj)

static void ll_dealloc(py_lifeline_t* pyobj)
{
// This type participates in GC; untrack before clearing references so the
// collector does not traverse a partially torn-down object during dealloc.
PyObject_GC_UnTrack(pyobj);
Py_CLEAR(pyobj->m_view);
typedef std::shared_ptr<void> generic_shared_t;
pyobj->m_source.~generic_shared_t();
// Use tp_free to pair with tp_alloc for GC-tracked Python objects.
Py_TYPE(pyobj)->tp_free((PyObject*)pyobj);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks scary. I don't claim that it's wrong. But do we understand why these changes are required?

}

// clang-format off
Expand Down
Loading
Loading