Merge branch 'main' of github.com:isl-org/Open3D into feature/ndt-registration

Author: ssheorey

.dockerignore

@@ -8,3 +8,14 @@ docs/_out
 .python-version
 cmake-build-*
 .git
+
+# CI runner artifacts (wheel job downloads the lib image next to the checkout).
+# Not used by Dockerfile COPY; including them blew the build context to ~20GB.
+open3d-lib-sycl-shared.tar.zst
+open3d-lib-*.tar.zst
+open3d-lib-*.tar.gz
+open3d-devel-*.tar.xz
+open3d-devel-*.tar.gz
+open3d-ci-*.tar.xz
+open3d-ci-*.tar.gz
+open3d*.whl

.github/workflows/README.md

@@ -8,6 +8,49 @@ possible. For jobs that use docker, or auxiliary jobs (e.g. uploading artifacts
 to google cloud), use ubuntu-latest to avoid changing the version on GitHub
 actions updates. On macOS and Windows, using Ubuntu reduces CI cost.
 
+## CI test layout
+
+Open3D splits **C++** and **Python** testing so each configuration is exercised
+once, without re-running the full suite in every wheel matrix cell. Docker-based
+workflows use `docker/docker_test.sh`; see `docker/docker_test.sh` usage for tag
+names.
+
+### `docker_test.sh` phases
+
+Optional second argument (default `all`):
+
+| Phase | Runs |
+|-------|------|
+| `cpp` | C++ gtest only (`./bin/tests --gtest_shuffle`) |
+| `lib` | `cpp` + command-line tools + C++ linking example + `make uninstall` |
+| `python` | `pytest python/test` inside the CI image |
+| `all` | `lib` + `python` (full suite) |
+
+SYCL, CPU (`cpu-static`, `cpu-shared-ml`, …), and OpenBLAS docker tags support
+all phases. CUDA ML configs on GCE (`ubuntu-cuda.yml`) call `docker_test.sh`
+with one argument only → **`all`** on a GPU VM.
+
+### Where tests run
+
+| Area | C++ / integration | Python | Notes |
+|------|-------------------|--------|--------|
+| **ubuntu.yml** | `lib` per matrix leg | `python` in same image | All in Docker |
+| **ubuntu-sycl.yml** | `cpp` in **build-lib** (ON/OFF) | `python` in **build-wheel** after `docker_build.sh sycl-shared py*` | No duplicate bare `docker run` gtest |
+| **ubuntu-openblas.yml** | `lib` once on **build-lib-arm64** (reference image); amd64 `lib` in single job | `python` once per Python version in wheel job (`"${DOCKER_TAG}"`) | Wheel matrix does not re-run C++ |
+| **ubuntu-cuda.yml** | **`all`** on GCE per `CI_CONFIG` | same step | Only workflow with CUDA GPUs in tests |
+| **ubuntu-wheel.yml** | **Once** in **test-cpp** (gtest bundle from **build-lib**) | **test-wheel-cpu**: host `test_wheel` + `run_python_tests` per Python (CUDA + CPU wheel smoke); wheels built in Docker, tests on `ubuntu-22.04` runner | No GPU on GitHub |
+| **windows.yml** / **macos.yml** | gtest in **build-lib** | **test-wheel** on runner | Native builds, not docker |
+
+GitHub-hosted runners have **no CUDA GPU**. SYCL jobs rely on SYCL-CPU / skips;
+Windows **test-wheel** skips pytest on the `cuda` device matrix leg.
+
+### Downstream jobs when build-lib fails
+
+**build-wheel** (and related jobs such as **test-wheel**, **test-cpp** on
+ubuntu-wheel) use `if: ${{ always() && !cancelled() }}` with `needs: build-lib`
+so wheel builds and tests still run when a lib job failed (e.g. flaky gtest).
+Steps may fail if required artifacts were never uploaded.
+
 ## Documentation deployment
 
 ### Directory structure
@@ -83,7 +126,7 @@ Now `~/open3d-ci-sa-key.json` should have been created.
 
 ### CI Procedure
 
-The GCE CI workflow `.github/workflows/gce-ubuntu-docker.yml` performs these steps:
+The GCE CI workflow `.github/workflows/ubuntu-cuda.yml` performs these steps:
 
 -   Clone the repository
 -   Build docker image, starting with a an NVIDIA base devel image with CUDA and
@@ -92,10 +135,8 @@ The GCE CI workflow `.github/workflows/gce-ubuntu-docker.yml` performs these ste
 -   On Google Compute Engine (GCE), in parallel (up to GPU quota limit - currently
     4):
 -   Create a new VM instance with a custom OS image
--   Run docker image on GCE (Google Compute Engine) with environment variables
-    set for specific build config.
--   The docker image entrypoint is the `run-ci.sh` script: build, install, run
-    tests and uninstall.
+-   Build the docker image on the VM, then run `docker/docker_test.sh` for the
+    config (full `all` phase: C++ gtest, Python pytest, linking, uninstall).
 -   Delete the VM instance.
 
 A separate VM instance is created for each commit and build option. The VM

.github/workflows/macos.yml

@@ -143,6 +143,7 @@ jobs:
 
   build-wheel:
     needs: build-lib
+    if: ${{ always() && !cancelled() }}
     name: Build wheel
     permissions:
       contents: write  # upload
@@ -171,7 +172,7 @@ jobs:
     env:
       # TensorFlow v2.20 only supports macOS-14 (Apple Silicon) and python 3.9-3.13
       BUILD_TENSORFLOW_OPS: ${{ matrix.python_version >= '3.14' && 'OFF' || 'ON' }}
-      # PyTorch 2.10 only supports macOS-14 (Apple Silicon) and python 3.10-3.14
+      # PyTorch 2.13 only supports macOS-14 (Apple Silicon) and python 3.10-3.14
       BUILD_PYTORCH_OPS: 'ON'
       OPEN3D_ML_ROOT: ${{ github.workspace }}/Open3D-ML
     steps:
@@ -182,7 +183,9 @@ jobs:
         uses: actions/checkout@v4
         with:
           repository: isl-org/Open3D-ML
-          ref: main
+          # TODO(ss/sycl-mlops): switch back to main once the torch 2.13
+          # requirements-torch.txt bump on ss/xpu is merged upstream.
+          ref: ss/xpu
           path: ${{ env.OPEN3D_ML_ROOT }}
 
       - name: Download build artifact
@@ -264,6 +267,7 @@ jobs:
       contents: read
     runs-on: ${{ matrix.os }}
     needs: [build-wheel]
+    if: ${{ always() && !cancelled() }}
     strategy:
       fail-fast: false
       matrix:
@@ -284,7 +288,7 @@ jobs:
     env:
       # TensorFlow v2.20 only supports macOS-14 (Apple Silicon) and python 3.9-3.13
       BUILD_TENSORFLOW_OPS: ${{ matrix.python_version >= '3.14' && 'OFF' || 'ON' }}
-      # PyTorch 2.10 only supports macOS-14 (Apple Silicon) and python 3.10-3.14
+      # PyTorch 2.13 only supports macOS-14 (Apple Silicon) and python 3.10-3.14
       BUILD_PYTORCH_OPS: 'ON'
       OPEN3D_ML_ROOT: ${{ github.workspace }}/Open3D-ML
     steps:
@@ -295,7 +299,9 @@ jobs:
         uses: actions/checkout@v4
         with:
           repository: isl-org/Open3D-ML
-          ref: main
+          # TODO(ss/sycl-mlops): switch back to main once the torch 2.13
+          # requirements-torch.txt bump on ss/xpu is merged upstream.
+          ref: ss/xpu
           path: ${{ env.OPEN3D_ML_ROOT }}
 
       - name: Download wheels

.github/workflows/ubuntu-openblas.yml

@@ -31,8 +31,11 @@ jobs:
       - name: Docker build
         run: docker/docker_build.sh openblas-amd64-py312-dev
 
-      - name: Docker test
-        run: docker/docker_test.sh openblas-amd64-py312-dev
+      - name: Docker test (C++ and integration)
+        run: docker/docker_test.sh openblas-amd64-py312-dev lib
+
+      - name: Docker test (Python)
+        run: docker/docker_test.sh openblas-amd64-py312-dev python
 
   build-lib-arm64:
     name: Build Lib ARM64
@@ -68,6 +71,9 @@ jobs:
           path: open3d-lib-arm64.tar.gz
           retention-days: 1
 
+      - name: Docker test (C++ and integration)
+        run: docker/docker_test.sh openblas-arm64-py310-dev lib
+
   build-wheel-arm64:
     name: Build Wheel ARM64
     permissions:
@@ -76,6 +82,7 @@ jobs:
       attestations: write
     runs-on: ubuntu-24.04-arm
     needs: build-lib-arm64
+    if: ${{ always() && !cancelled() }}
     strategy:
       fail-fast: false
       matrix:
@@ -148,6 +155,5 @@ jobs:
         run: |
           .github/workflows/update_release.sh ${{ env.PIP_PKG_NAME }}
 
-      # Move after artifact upload, so failing / flaky C++ tests don't block the uploads.
-      - name: Docker test
-        run: docker/docker_test.sh "${DOCKER_TAG}"
+      - name: Docker test (Python)
+        run: docker/docker_test.sh "${DOCKER_TAG}" python

.github/workflows/ubuntu-sycl.yml

@@ -74,11 +74,16 @@ jobs:
           path: open3d-devel-*.tar.xz
           if-no-files-found: error
 
+      - name: Remove lib image tarball from workspace
+        # Multi-GB and no longer needed once uploaded; frees runner disk for the
+        # C++ tests. The devel / wheel / ccache tarballs stay: the main-branch
+        # release and ccache upload steps below still read them.
+        if: ${{ matrix.BUILD_SHARED_LIBS == 'ON' }}
+        run: rm -f open3d-lib-sycl-shared.tar.zst
+
       - name: Docker test (C++)
-        # Only the C++ phase runs here; Python tests run later in the
-        # build-wheel job against the actual per-Python-version wheel
-        # (see docker/docker_test.sh's cpp_test()/python_test() split),
-        # so we avoid testing Python twice.
+        # C++ only here; Python runs in build-wheel via docker_test.sh python
+        # inside the per-Python-version image (see docker/docker_test.sh).
         run: |
           if [ "${{ matrix.BUILD_SHARED_LIBS }}" = "ON" ]; then
             docker/docker_test.sh sycl-shared cpp
@@ -119,6 +124,8 @@ jobs:
       artifact-metadata: write
     runs-on: ubuntu-latest
     needs: build-lib
+    # Run even when build-lib failed (e.g. flaky C++ tests) so wheel jobs still execute.
+    if: ${{ always() && !cancelled() }}
     strategy:
       fail-fast: false
       matrix:
@@ -136,7 +143,6 @@ jobs:
             python_version: '3.13'
     env:
       PYTHON_VERSION: ${{ matrix.python_version }}
-      OPEN3D_ML_ROOT: ${{ github.workspace }}/open3d_ml
       # TensorFlow v2.20 only supports python 3.9-3.13
       BUILD_TENSORFLOW_OPS: ${{ matrix.python_version >= '3.14' && 'OFF' || 'ON' }}
     steps:
@@ -154,21 +160,19 @@ jobs:
           path: .
 
       - name: Load Lib Image
-        run: zstd -d --stdout open3d-lib-sycl-shared.tar.zst | docker load
+        run: |
+          zstd -d --stdout open3d-lib-sycl-shared.tar.zst | docker load
+          rm -f open3d-lib-sycl-shared.tar.zst
 
       - name: Docker build
         run: |
           export BASE_IMAGE=open3d-lib-sycl-shared:latest
-          
+
           # We execute docker buildscript with python version argument.
           docker/docker_build.sh sycl-shared py${{ matrix.python_version }}
 
-          # The loaded build-lib image contains its Python 3.12 wheel. Keep
-          # only the wheel produced for this matrix entry before later steps
-          # expand open3d*.whl.
-          PYTHON_ABI="cp${PYTHON_VERSION/./}"
-          find . -maxdepth 1 -name 'open3d*.whl' \
-            ! -name "*${PYTHON_ABI}-${PYTHON_ABI}-*.whl" -delete
+      - name: Docker test (Python)
+        run: docker/docker_test.sh sycl-shared python
 
       - name: Generate artifact attestation
         if: ${{ github.ref == 'refs/heads/main' }}
@@ -184,37 +188,6 @@ jobs:
           path: open3d*.whl
           if-no-files-found: error
 
-      - name: Checkout Open3D-ML source code
-        uses: actions/checkout@v4
-        with:
-          repository: isl-org/Open3D-ML
-          ref: main
-          path: ${{ env.OPEN3D_ML_ROOT }}
-
-      - name: Install runtime dependencies
-        # Open3D links against EGL for GPU-accelerated offscreen rendering
-        # (see cpp/open3d/visualization/visualizer/EGLOffscreenContext.*).
-        # This bare runner (unlike the docker_build.sh image) does not have
-        # it installed by default.
-        run: sudo apt-get update && sudo apt-get install --yes libegl1
-
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: ${{ matrix.python_version }}
-
-      - name: Test Python package
-        run: |
-          python -V
-          source util/ci_utils.sh
-          test_wheel open3d*.whl
-
-      - name: Run Python unit tests
-        run: |
-          source util/ci_utils.sh
-          echo "Running Open3D python tests..."
-          run_python_tests
-
       - name: Update devel release
         if: ${{ github.ref == 'refs/heads/main' }}
         env:

.github/workflows/ubuntu-wheel.yml

@@ -68,6 +68,7 @@ jobs:
     name: Test C++
     runs-on: ubuntu-22.04
     needs: [build-lib]
+    if: ${{ always() && !cancelled() }}
     steps:
       - name: Checkout source code
         uses: actions/checkout@v4
@@ -89,6 +90,7 @@ jobs:
 
   build-wheel:
     needs: build-lib
+    if: ${{ always() && !cancelled() }}
     permissions:
       contents: write  # Release upload
       id-token: write
@@ -209,6 +211,7 @@ jobs:
       contents: read
     runs-on: ubuntu-22.04
     needs: [build-wheel]
+    if: ${{ always() && !cancelled() }}
     strategy:
       fail-fast: false
       matrix:
@@ -240,7 +243,9 @@ jobs:
         uses: actions/checkout@v4
         with:
           repository: isl-org/Open3D-ML
-          ref: main
+          # TODO(ss/sycl-mlops): switch back to main once the torch 2.13
+          # requirements-torch.txt bump on ss/xpu is merged upstream.
+          ref: ss/xpu
           path: ${{ env.OPEN3D_ML_ROOT }}
       - name: Download wheels
         uses: actions/download-artifact@v4
@@ -276,8 +281,3 @@ jobs:
           source util/ci_utils.sh
           pi_tag=$(python -c "import sys; print(f'cp{sys.version_info.major}{sys.version_info.minor}')")
           test_wheel open3d_cpu*-"$pi_tag"-*.whl
-      - name: Run Python unit tests (CPU)
-        run: |
-          source util/ci_utils.sh
-          echo "Running Open3D python tests (CPU wheel)..."
-          run_python_tests

.github/workflows/ubuntu.yml

@@ -54,16 +54,27 @@ jobs:
           elif [ "${{ matrix.BUILD_SHARED_LIBS }}" = "ON"  ] && [ "${{ env.MLOPS }}" = "ON"  ] && [ "${{ env.DEVELOPER_BUILD }}" = "OFF" ]; then
             docker/docker_build.sh cpu-shared-ml-release
           fi
-      - name: Docker test
+      - name: Docker test (C++ and integration)
         run: |
           if [ "${{ matrix.BUILD_SHARED_LIBS }}" = "OFF" ] && [ "${{ env.MLOPS }}" = "OFF" ] && [ "${{ env.DEVELOPER_BUILD }}" = "ON" ]; then
-            docker/docker_test.sh cpu-static
+            docker/docker_test.sh cpu-static lib
           elif [ "${{ matrix.BUILD_SHARED_LIBS }}" = "OFF" ] && [ "${{ env.MLOPS }}" = "OFF" ] && [ "${{ env.DEVELOPER_BUILD }}" = "OFF" ]; then
-            docker/docker_test.sh cpu-static-release
+            docker/docker_test.sh cpu-static-release lib
           elif [ "${{ matrix.BUILD_SHARED_LIBS }}" = "ON"  ] && [ "${{ env.MLOPS }}" = "ON"  ] && [ "${{ env.DEVELOPER_BUILD }}" = "ON"  ]; then
-            docker/docker_test.sh cpu-shared-ml
+            docker/docker_test.sh cpu-shared-ml lib
           elif [ "${{ matrix.BUILD_SHARED_LIBS }}" = "ON"  ] && [ "${{ env.MLOPS }}" = "ON"  ] && [ "${{ env.DEVELOPER_BUILD }}" = "OFF" ]; then
-            docker/docker_test.sh cpu-shared-ml-release
+            docker/docker_test.sh cpu-shared-ml-release lib
+          fi
+      - name: Docker test (Python)
+        run: |
+          if [ "${{ matrix.BUILD_SHARED_LIBS }}" = "OFF" ] && [ "${{ env.MLOPS }}" = "OFF" ] && [ "${{ env.DEVELOPER_BUILD }}" = "ON" ]; then
+            docker/docker_test.sh cpu-static python
+          elif [ "${{ matrix.BUILD_SHARED_LIBS }}" = "OFF" ] && [ "${{ env.MLOPS }}" = "OFF" ] && [ "${{ env.DEVELOPER_BUILD }}" = "OFF" ]; then
+            docker/docker_test.sh cpu-static-release python
+          elif [ "${{ matrix.BUILD_SHARED_LIBS }}" = "ON"  ] && [ "${{ env.MLOPS }}" = "ON"  ] && [ "${{ env.DEVELOPER_BUILD }}" = "ON"  ]; then
+            docker/docker_test.sh cpu-shared-ml python
+          elif [ "${{ matrix.BUILD_SHARED_LIBS }}" = "ON"  ] && [ "${{ env.MLOPS }}" = "ON"  ] && [ "${{ env.DEVELOPER_BUILD }}" = "OFF" ]; then
+            docker/docker_test.sh cpu-shared-ml-release python
           fi
       - name: Generate package attestation
         if: ${{ github.ref == 'refs/heads/main' && matrix.BUILD_SHARED_LIBS == 'ON' }}
@@ -149,7 +160,9 @@ jobs:
           name: open3d-wheel-linux-x86_64
 
       - name: Checkout Open3D-ML
-        run: git clone --depth 1 https://github.com/isl-org/Open3D-ML.git -b main /tmp/Open3D-ML
+        # TODO(ss/sycl-mlops): switch back to main once the torch 2.13
+        # requirements-torch.txt bump on ss/xpu is merged upstream.
+        run: git clone --depth 1 https://github.com/isl-org/Open3D-ML.git -b ss/xpu /tmp/Open3D-ML
 
       - name: Install doc dependencies
         run: |