Merge branch 'main' into issue-6164-rsbag-post-processing

Author: aekanman

.dockerignore

@@ -8,3 +8,14 @@ docs/_out
 .python-version
 cmake-build-*
 .git
+
+# CI runner artifacts (wheel job downloads the lib image next to the checkout).
+# Not used by Dockerfile COPY; including them blew the build context to ~20GB.
+open3d-lib-sycl-shared.tar.zst
+open3d-lib-*.tar.zst
+open3d-lib-*.tar.gz
+open3d-devel-*.tar.xz
+open3d-devel-*.tar.gz
+open3d-ci-*.tar.xz
+open3d-ci-*.tar.gz
+open3d*.whl

.github/workflows/README.md

@@ -8,13 +8,59 @@ possible. For jobs that use docker, or auxiliary jobs (e.g. uploading artifacts
 to google cloud), use ubuntu-latest to avoid changing the version on GitHub
 actions updates. On macOS and Windows, using Ubuntu reduces CI cost.
 
+## CI test layout
+
+Open3D splits **C++** and **Python** testing so each configuration is exercised
+once, without re-running the full suite in every wheel matrix cell. Docker-based
+workflows use `docker/docker_test.sh`; see `docker/docker_test.sh` usage for tag
+names.
+
+### `docker_test.sh` phases
+
+Optional second argument (default `all`):
+
+| Phase | Runs |
+|-------|------|
+| `cpp` | C++ gtest only (`./bin/tests --gtest_shuffle`) |
+| `lib` | `cpp` + command-line tools + C++ linking example + `make uninstall` |
+| `python` | `pytest python/test` inside the CI image |
+| `all` | `lib` + `python` (full suite) |
+
+SYCL, CPU (`cpu-static`, `cpu-shared-ml`, …), and OpenBLAS docker tags support
+all phases. CUDA ML configs on GCE (`ubuntu-cuda.yml`) call `docker_test.sh`
+with one argument only → **`all`** on a GPU VM.
+
+### Where tests run
+
+| Area | C++ / integration | Python | Notes |
+|------|-------------------|--------|--------|
+| **ubuntu.yml** | `lib` per matrix leg | `python` in same image | All in Docker |
+| **ubuntu-sycl.yml** | `cpp` in **build-lib** (ON/OFF) | `python` in **build-wheel** after `docker_build.sh sycl-shared py*` | No duplicate bare `docker run` gtest |
+| **ubuntu-openblas.yml** | `lib` once on **build-lib-arm64** (reference image); amd64 `lib` in single job | `python` once per Python version in wheel job (`"${DOCKER_TAG}"`) | Wheel matrix does not re-run C++ |
+| **ubuntu-cuda.yml** | **`all`** on GCE per `CI_CONFIG` | same step | Only workflow with CUDA GPUs in tests |
+| **ubuntu-wheel.yml** | **Once** in **test-cpp** (gtest bundle from **build-lib**) | **test-wheel-cpu**: host `test_wheel` + `run_python_tests` per Python (CUDA + CPU wheel smoke); wheels built in Docker, tests on `ubuntu-22.04` runner | No GPU on GitHub |
+| **windows.yml** / **macos.yml** | gtest in **build-lib** | **test-wheel** on runner | Native builds, not docker |
+
+GitHub-hosted runners have **no CUDA GPU**. SYCL jobs rely on SYCL-CPU / skips;
+Windows **test-wheel** skips pytest on the `cuda` device matrix leg.
+
+### Downstream jobs when build-lib fails
+
+**build-wheel** (and related jobs such as **test-wheel**, **test-cpp** on
+ubuntu-wheel) use `if: ${{ always() && !cancelled() }}` with `needs: build-lib`
+so wheel builds and tests still run when a lib job failed (e.g. flaky gtest).
+Steps may fail if required artifacts were never uploaded.
+
 ## Documentation deployment
 
 ### Directory structure
 
--   `.github/workflows/documentation.yml`: Github Actions workflow file to
-    create and deploy documentation. Documentation is created for every branch
-    as a CI test, but deployed only for `main`.
+-   `.github/workflows/ubuntu.yml`: The `build-docs` job creates and deploys
+    documentation, using the Open3D Python wheel built by the `ubuntu` job in
+    the same workflow (EGL-based offscreen rendering means the standard wheel
+    supports headless notebook execution, so no separate headless build is
+    needed). Documentation is created for every branch as a CI test, but
+    deployed only for `main`.
 -   `util/ci_utils.sh:build_docs()`: Called by GitHub Actions to build documentation.
 -   `unpack_docs.sh`: Called by the documentation server to deploy the docs into
     the website.
@@ -80,7 +126,7 @@ Now `~/open3d-ci-sa-key.json` should have been created.
 
 ### CI Procedure
 
-The GCE CI workflow `.github/workflows/gce-ubuntu-docker.yml` performs these steps:
+The GCE CI workflow `.github/workflows/ubuntu-cuda.yml` performs these steps:
 
 -   Clone the repository
 -   Build docker image, starting with a an NVIDIA base devel image with CUDA and
@@ -89,10 +135,8 @@ The GCE CI workflow `.github/workflows/gce-ubuntu-docker.yml` performs these ste
 -   On Google Compute Engine (GCE), in parallel (up to GPU quota limit - currently
     4):
 -   Create a new VM instance with a custom OS image
--   Run docker image on GCE (Google Compute Engine) with environment variables
-    set for specific build config.
--   The docker image entrypoint is the `run-ci.sh` script: build, install, run
-    tests and uninstall.
+-   Build the docker image on the VM, then run `docker/docker_test.sh` for the
+    config (full `all` phase: C++ gtest, Python pytest, linking, uninstall).
 -   Delete the VM instance.
 
 A separate VM instance is created for each commit and build option. The VM

.github/workflows/documentation.yml

@@ -67,6 +67,7 @@ jobs:
       - name: Install dependencies
         run: |
           brew install ccache glslang spirv-cross gcc
+          # Fix gfortran not found issue
           ln -s $(brew --prefix gcc)/bin/gfortran-* /usr/local/bin/gfortran
 
           ccache -M 2G  # See .github/workflows/readme.md for ccache strategy.
@@ -102,7 +103,7 @@ jobs:
           ccache -s
 
       - name: Generate package attestation
-        if: ${{ env.BUILD_SHARED_LIBS == 'ON' }}
+        if: ${{ github.ref == 'refs/heads/main' && env.BUILD_SHARED_LIBS == 'ON' }}
         uses: actions/attest@v4
         with:
           subject-path: build/package/${{ env.DEVEL_PKG_NAME }}
@@ -116,7 +117,7 @@ jobs:
           if-no-files-found: error
 
       - name: Generate viewer attestation
-        if: ${{ env.BUILD_SHARED_LIBS == 'OFF' }}
+        if: ${{ github.ref == 'refs/heads/main' && env.BUILD_SHARED_LIBS == 'OFF' }}
         uses: actions/attest@v4
         with:
           subject-path: build/bin/open3d-*-app-macosx-10_15-${{ runner.arch }}.zip
@@ -142,6 +143,7 @@ jobs:
 
   build-wheel:
     needs: build-lib
+    if: ${{ always() && !cancelled() }}
     name: Build wheel
     permissions:
       contents: write  # upload
@@ -170,7 +172,7 @@ jobs:
     env:
       # TensorFlow v2.20 only supports macOS-14 (Apple Silicon) and python 3.9-3.13
       BUILD_TENSORFLOW_OPS: ${{ matrix.python_version >= '3.14' && 'OFF' || 'ON' }}
-      # PyTorch 2.10 only supports macOS-14 (Apple Silicon) and python 3.10-3.14
+      # PyTorch 2.13 only supports macOS-14 (Apple Silicon) and python 3.10-3.14
       BUILD_PYTORCH_OPS: 'ON'
       OPEN3D_ML_ROOT: ${{ github.workspace }}/Open3D-ML
     steps:
@@ -181,7 +183,9 @@ jobs:
         uses: actions/checkout@v4
         with:
           repository: isl-org/Open3D-ML
-          ref: main
+          # TODO(ss/sycl-mlops): switch back to main once the torch 2.13
+          # requirements-torch.txt bump on ss/xpu is merged upstream.
+          ref: ss/xpu
           path: ${{ env.OPEN3D_ML_ROOT }}
 
       - name: Download build artifact
@@ -237,6 +241,7 @@ jobs:
           echo "PIP_PKG_NAME=$PIP_PKG_NAME" >> $GITHUB_ENV
 
       - name: Generate wheel attestation
+        if: ${{ github.ref == 'refs/heads/main' }}
         uses: actions/attest@v4
         with:
           subject-path: build/lib/python_package/pip_package/${{ env.PIP_PKG_NAME }}
@@ -262,6 +267,7 @@ jobs:
       contents: read
     runs-on: ${{ matrix.os }}
     needs: [build-wheel]
+    if: ${{ always() && !cancelled() }}
     strategy:
       fail-fast: false
       matrix:
@@ -282,7 +288,7 @@ jobs:
     env:
       # TensorFlow v2.20 only supports macOS-14 (Apple Silicon) and python 3.9-3.13
       BUILD_TENSORFLOW_OPS: ${{ matrix.python_version >= '3.14' && 'OFF' || 'ON' }}
-      # PyTorch 2.10 only supports macOS-14 (Apple Silicon) and python 3.10-3.14
+      # PyTorch 2.13 only supports macOS-14 (Apple Silicon) and python 3.10-3.14
       BUILD_PYTORCH_OPS: 'ON'
       OPEN3D_ML_ROOT: ${{ github.workspace }}/Open3D-ML
     steps:
@@ -293,7 +299,9 @@ jobs:
         uses: actions/checkout@v4
         with:
           repository: isl-org/Open3D-ML
-          ref: main
+          # TODO(ss/sycl-mlops): switch back to main once the torch 2.13
+          # requirements-torch.txt bump on ss/xpu is merged upstream.
+          ref: ss/xpu
           path: ${{ env.OPEN3D_ML_ROOT }}
 
       - name: Download wheels

.github/workflows/macos.yml

@@ -154,7 +154,7 @@ jobs:
           fi
 
       - name: Generate package attestation
-        if: ${{ env.BUILD_PACKAGE == 'true' }}
+        if: ${{ github.ref == 'refs/heads/main' && env.BUILD_PACKAGE == 'true' }}
         uses: actions/attest@v4
         with:
           subject-path: ${{ github.workspace }}/open3d-devel-linux*.tar.xz

.github/workflows/ubuntu-cuda.yml

@@ -31,8 +31,11 @@ jobs:
       - name: Docker build
         run: docker/docker_build.sh openblas-amd64-py312-dev
 
-      - name: Docker test
-        run: docker/docker_test.sh openblas-amd64-py312-dev
+      - name: Docker test (C++ and integration)
+        run: docker/docker_test.sh openblas-amd64-py312-dev lib
+
+      - name: Docker test (Python)
+        run: docker/docker_test.sh openblas-amd64-py312-dev python
 
   build-lib-arm64:
     name: Build Lib ARM64
@@ -68,6 +71,9 @@ jobs:
           path: open3d-lib-arm64.tar.gz
           retention-days: 1
 
+      - name: Docker test (C++ and integration)
+        run: docker/docker_test.sh openblas-arm64-py310-dev lib
+
   build-wheel-arm64:
     name: Build Wheel ARM64
     permissions:
@@ -76,6 +82,7 @@ jobs:
       attestations: write
     runs-on: ubuntu-24.04-arm
     needs: build-lib-arm64
+    if: ${{ always() && !cancelled() }}
     strategy:
       fail-fast: false
       matrix:
@@ -128,10 +135,8 @@ jobs:
           PIP_PKG_NAME="$(basename ${GITHUB_WORKSPACE}/open3d-[0-9]*.whl)"
           echo "PIP_PKG_NAME=$PIP_PKG_NAME" >> $GITHUB_ENV
 
-      - name: Docker test
-        run: docker/docker_test.sh "${DOCKER_TAG}"
-
       - name: Generate wheel attestation
+        if: ${{ github.ref == 'refs/heads/main' }}
         uses: actions/attest@v4
         with:
           subject-path: ${{ github.workspace }}/${{ env.PIP_PKG_NAME }}
@@ -149,3 +154,6 @@ jobs:
           GH_TOKEN: ${{ github.token }}
         run: |
           .github/workflows/update_release.sh ${{ env.PIP_PKG_NAME }}
+
+      - name: Docker test (Python)
+        run: docker/docker_test.sh "${DOCKER_TAG}" python