Fix false positive triangle-triangle intersections

[gyuun]

Type

• Bug fix (non-breaking change which fixes an issue): Fixes TriangleMesh - is_self_intersecting return wrong result #5117
• New feature (non-breaking change which adds functionality). Resolves #
• Breaking change (fix or feature that would cause existing functionality to not work as expected) Resolves #

Motivation and Context

TriangleTriangle3d applies an absolute tolerance to plane-equation values computed with unnormalized triangle normals. For the triangles reported in #5117, this changes one of three same-sign values to zero, bypasses the same-side rejection, and produces a false positive.
The result also changes after a rigid rotation because the per-axis normalization changes the magnitude of these plane-equation values.

Checklist:

• I have run python util/check_style.py --apply to apply Open3D code style
  to my code.
• This PR changes Open3D behavior or adds new functionality.
  • Both C++ (Doxygen) and Python (Sphinx / Google style) documentation is
    updated accordingly.
  • I have added or updated C++ and / or Python unit tests OR included test
    results (e.g. screenshots or numbers) here.
• I will follow up and update the code if CI fails.
• For fork PRs, I have selected Allow edits from maintainers.

Description

This change adds a conservative half-space rejection before the existing triangle-triangle predicate.

Signed distances are computed using a unit triangle normal. If all three vertices of either triangle are unambiguously on the same side of the other triangle's supporting plane, the triangles are reported as separated. If a triangle does not define a usable plane, that plane-separation check is skipped.
This is an early rejection only: it returns false solely when either supporting plane establishes an unambiguous separation. If neither plane establishes a clear separation, the existing predicate is used unchanged.

An alternative would be to scale the epsilon used inside NoDivTriTriIsect by the corresponding normal magnitude, which is equivalent to applying the tolerance to unit-normal signed distances. That approach could eliminate the additional precheck, but it would require modifying vendored third-party code and could change near-coplanar classification throughout the existing predicate.

This PR therefore leaves the vendored implementation unchanged and adds only a conservative wrapper-level rejection for geometrically unambiguous separation.

No public API or third-party code is changed.

Tests added:

• Direct regression for the coordinates from TriangleMesh - is_self_intersecting return wrong result #5117.
• Rotation, translation, scale, vertex-order, and triangle-order coverage.
• Near-parallel intersecting-triangle regression.
• C++ TriangleMesh integration coverage.
• Python TriangleMesh API coverage.
• Near-degenerate intersecting-triangle regression

Test results:

• 7 focused C++ tests passed.
• 1 Python regression test passed.
• tests and python-package targets built successfully.

This is my first contribution to Open3D, so I would appreciate any feedback on the approach or test coverage. Thank you for reviewing!

[update-docs]

Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes.

[Copilot]

Pull request overview

Adds a conservative plane-separation precheck to reduce false-positive triangle intersections reported in #5117.

Changes:

• Adds unit-normal half-space rejection.
• Adds C++ and Python regression coverage.
• Documents the fix in the changelog.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
cpp/open3d/geometry/IntersectionTest.cpp	Adds the separation precheck.
cpp/tests/geometry/IntersectionTest.cpp	Tests transformations, ordering, and near-parallel intersections.
cpp/tests/geometry/TriangleMesh.cpp	Adds C++ mesh integration coverage.
python/test/geometry/test_trianglemesh.py	Adds Python API regression coverage.
CHANGELOG.md	Records the bug fix.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.