Feature request + pose estim algo questions

[sadanand1120]

Hi,

(1) the images captured currently are wide (1x) images. Can you add support for ultra-wide (0.5x) just like the native camera app in iDevices allow for? Atleast for the usb streaming option, this feature only affects RGB and the rest of the streaming pipeline I guess will remain same (unlike RGBD video where I some depth rescaling etc stuff might be needed)

(2) I am USB streaming raw data and dumping the rgb images + frame transforms. First, these metric poses -- how are they calculated exactly? Like algorithmically. And second, I was using the rgb + poses metadata to train on nerfstudio. However, the generated model had poor quality vs if I only use rgb and get colmap poses. Is that expected?

(3) as mentioned in (2) -- since colmap nerfstudio model was better I plan on using it. However, I do want real-world metric model. So my plan is to use the metric poses to calculate a best-fit affine colmap-to-real-world transform. Just wanted to ask if this sounds reasonable based on your knowledge on the metric poses that record3d outputs?

Thanks for the great app!

[marek-simonik]

Hi,

let me try to answer the questions:

1. I do not think Apple provides depth maps or camera tracking for the 0.5x lens, so in my opinion, this is a scenario, which is incompatible with what Record3D does (i.e. streaming RGB + Depth).
2. The poses are provided by Apple's ARKit. Their algorithms are not publicly known. COLMAP provides better poses because it performs global Bundle Adjustment, whereas ARKit does not adjust/refine the poses of past frames based on future observations. So what you observe is to be expected.
3. It sounds reasonable to me, but I have not tried this approach in practice, so I cannot attest how well will it work.

Thanks for using Record3D :)!