Can't convert AED models

[o-alexandre-felipe]

I noticed this pull request
#55

I was trying to convert a canary model to riva. But even with that I can't easily do the conversion.

Shouldn't we be able to use it like this?

huggingface-cli download --local-dir ./models/ nvidia/canary-1b
nemo2riva --out ./models/canary-1b.riva ./models/canary-1b.nemo 

To begin I noticed that in hugginface models

target: nemo.collections.asr.models.aed_multitask_models.EncDecMultiTaskModel

while the schema there was expecting nemo.collections.asr.models.EncDecMultiTaskModel

Should I get the models from other source? Should I use another branch? Or we need to write new validation rules?

Regards,
Alexandre

[anand-nv]

What version of nemo toolkit were you using to try this out?

[o-alexandre-felipe]

the last attempt was with

git+https://github.com/NVIDIA/NeMo@v2.20.0#nemo-toolkit[all]

[anand-nv]

You need to use a nemo version < 2.0

[o-alexandre-felipe]

Should

nemo2riva/requirements.txt

Line 4 in 85cb263

nemo_toolkit>=1.6.0

be updated to reflect this or is there a reason to keep the versions open?

I tried just tried
huggingface-cli download --local-dir ./models/ nvidia/canary-1b

and it comes with

target: nemo.collections.asr.models.aed_multitask_models.EncDecMultiTaskModel

What if one wants to deploy canary-1b-flash? As far as I know it requires a nemo-toolkit>=2

I also see that some whisper models are available at NGC in .riva format. This raises a wider question.
How are these models converted to .riva? Is there a separate project that I should be aware of?

Thank you.

[anand-nv]

canary-1b-flash requires nemo-toolkit>=2 and is not currently supported in riva (as of 25.02). #55 was for adding support via trt-llm decoder. https://github.com/anand-nv/TensorRT-LLM/tree/canary_support/examples/canary

[o-alexandre-felipe]

So should we keep this issue open or dismiss it?

[anand-nv]

Support would be available in 25.06 - #65

[ccyrene]

Hi @anand-nv, I’m truly impressed by your work on integrating Canary into the TensorRT-LLM framework 🙌

I have a question regarding converting custom models to TensorRT-LLM. If I have a custom model, how should I go about implementing the convert_checkpoint.py script for it? 🙏

I’d greatly appreciate any guidance or pointers you could share. Thank you so much!

[o-alexandre-felipe]

Thank you @anand-nv !