Questions regarding SAM3 backbone replacement and training requirements for C-RADIOv4

[JuiceCoffe]

Thank you for the impressive tech report on C-RADIOv4.
I have a few questions regarding the integration with SAM3 and the general training setup:

SAM3 Backbone Adaptation:
You mentioned that C-RADIOv4 can be used as a direct replacement for the SAM3 vision encoder. Aside from the multi-teacher distillation losses described in the paper, did you perform any additional training or fine-tuning specifically to adapt C-RADIOv4 to the SAM3 decoder? If so, could you provide more details on how this adaptation process was conducted?

Distillation:
What kind of dataset (and of what scale) is recommended or was used for the distillation process? Could you share the approximate computational resources required to perform the distillation of SAM3?

[mranzinger]

Hi, SAM3 was part of "stage 3" of the training process (from the radio v2.5 paper). We treated SAM3 like any other teacher, so it's just our typical student + adaptor setup, learning from their vision features just before the neck. Because our encoder is aligned with their, we get alignment with their decoder for free, although downstream finetuning would probably improve things further. It's perhaps a bit more dubious with SAM3 than SAM1, but we continue to use the mosaic adaptor for the low-res partition (also described in the radio v2.5 paper).

We distill from an internally curated dataset of 800M images, however, ImageNet-21k works extremely well as well. We use the internal dataset because it's composed of images that have commercially permissive licensing, allowing us to release these models with permissive licensing. Stage 3 was something like 4 days of 128 A100 gpus.