Huge seams in textured results

[chih-yu-chen]

Hello @nmoehrle and team,
I am using iPhone's mesh reconstruction from LiDAR, image capturing, and MVS-Texturing to reconstruct the textures in indoor scenes.

MVS-Texturing was built on Ubuntu 22.04 with changes from the oneTBB compatibility pull request. The dependencies can be seen in the cmake messages below.

-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting build type to 'RELWITHDEBINFO' as none was specified.
-- Found TBB: /usr/include (found version "/*
    Copyright (c) 2005-2021 Intel Corporation

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
*/

#include "../oneapi/tbb/version.h"
./*
    Copyright (c) 2005-2021 Intel Corporation

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
*/

#include "../oneapi/tbb/version.h"
")  
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") 
-- Found PNG: /usr/lib/x86_64-linux-gnu/libpng.so (found version "1.6.37") 
-- Found JPEG: /usr/lib/x86_64-linux-gnu/libjpeg.so (found version "80") 
-- Found TIFF: /usr/lib/x86_64-linux-gnu/libtiff.so (found version "4.3.0")  
-- Configuring done
CMake Warning (dev) at libs/tex/CMakeLists.txt:5 (add_library):
  Policy CMP0069 is not set: INTERPROCEDURAL_OPTIMIZATION is enforced when
  enabled.  Run "cmake --help-policy CMP0069" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  INTERPROCEDURAL_OPTIMIZATION property will be ignored for target 'tex'.
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at apps/texrecon/CMakeLists.txt:6 (add_executable):
  Policy CMP0069 is not set: INTERPROCEDURAL_OPTIMIZATION is enforced when
  enabled.  Run "cmake --help-policy CMP0069" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  INTERPROCEDURAL_OPTIMIZATION property will be ignored for target
  'texrecon'.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: ....../mvs-texturing/build

Here are the data and command to reproduce the issue: link

texrecon images_cams/ mesh.ply textured

The textured results always look like the images attached below.
There are huge seams that are most noticeable on the walls and floor.
One thing interesting is that the seams seem to be parallel to the coordinate axes. This is most noticeable for the horizontal seam and the seams on the floor, which also extend to the walls.
I have tried to use the area data term, skip global/local seam leveling, and fixate the exposure and white balance of the images I captured, but the seams are there everytime.

I even tried the regularization mentioned in issue #63 comment. Although the contrast of the two sides of the seams are smaller when global seam leveling is regularized by 1e-4, the seams still persist.

Has anyone encountered something similar or had some ideas how to fix it?
Any help would be greatly appreciated!

Thank you so much!

[thomas-hut]

@chih-yu-chen
Hi, I guess it would be helpful to increase the sampling rate in your case, as it seems that the raw data might be affected by varying light conditions. If you don’t mind, could you please share your raw data for review? Thank you!

[chih-yu-chen]

@thomas-graphopti

Thank you for the suggestion! The sampling rate is 1 frame per second now (captured using the ARCamera of iOS ARKit). I will try maybe 2 fps later.
And sure the raw data is here.

Thank you again!

[chih-yu-chen]

@thomas-graphopti
Hi Thomas,

I tried raising the frame rate to 2 fps but the textured results look basically the same.

I have also tried to fix the exposure and white balance of the ARCamera, or adjust the brightness of the images based on their exposure offset. The brightness of the textures changes, but the huge seams are still there.

Do you have other ideas why there are seams? Especially why are the seams parallel to the coordinate axes? (the negative z-axis points toward the camera-lookat of the first captured frame)

Thank you so much in advance!

[thomas-hut]

Hi @chih-yu-chen ,
I reviewed the data you provided, but it did not reveal significant insights. However, I would like to share some suggestions that might be helpful:

1. Adjust Image Quality: Start by setting the image compression rate to 100% to ensure the highest possible image quality.

2. Reduce Movement Speed: During the capturing process, try to move more slowly to maintain stability and improve data accuracy.

3. Enable Seam Leveling: Ensure that both global and local seam leveling are enabled in the MVS-Texturing phase for better results.

4. Evaluate ARKit Pose Estimation: It is possible that the ARKit pose estimation is not meeting your expectations. I recommend using tools such as COLMAP to correct and refine these poses.

5. Assess Model Alignment: Confirm that your raw model (e.g., mesh.ply) aligns accurately with the ARKit tracking coordinate system. Note that the LiDAR sensor in iPhones maintains a fixed distance from the main camera, but it is worth verifying that they are properly aligned.

These are my initial thoughts, and I hope they provide some assistance. Please feel free to share updates or reach out for further discussion.

[chih-yu-chen]

Hi @thomas-graphopti,

thank you for the suggestions!

I asked in the OpenMVS repo the other day (cdcseacave/openMVS#1179) and found out the cause was that Apple ARKit saves the reconstructed mesh into disconnected submeshes. Once the submeshes are connected, the textured results are perfect!

Thank you so much again for the help and suggestions!
I will close the issue now.