Faster Bicleaner AI

[ZJaume]

Benchmarks with one GPU on a interactive task (en-bs model and en-xx-large model)

configuration	speed (rows/s)
base fp32 batch32 block10k (previous config)	273
base fp32 batch128 block10k	320
base fp16 batch128 block10k	729
base fp16 batch512 block10k	746
base fp16 batch512 block50k	768
large fp32 batch32 block10k (previous config)	126
large fp16 batch512 block50k	276

We can squeeze a bit more speed with bigger batches but I guess we prefer to be on the safer side regarding possible OOM issues. I tested 1024 and 2048 batch sizes and does not crash, so 512 should be pretty safe.

[evgenyrp]

Great improvement!

Let's clarify the CUDA installation. Can we continue using the toolchain?

[evgenyrp]

I think it is a common optimization to prevent a lengthy installation of CUDA at runtime, right? @bhearsum Does it not work with the new version of Bicleaner? @ZJaume

[evgenyrp]

A different version of cuda/cudnn can be added in fetch and toolchain

[ZJaume]

I was not able to make the new Tensorflow version work with the toolchain that Taskcluster fetches. Tensorflow now it is using the CUDA pip packages like torch has been doing for some time. I'm not sure if I was not able to make it work because Tensorflow was not compatible with our toolchain version or just because it was not looking at the LD_LIBRARY_PATH. The instructions do not mention anything about optional cuda installation, so maybe the pip install tensorflow (without the cuda) just installs cpu version and cannot be linked to existing CUDA installation because it does not have symlinks.

[bhearsum]

If it does turn out that this causes a CUDA installation, and it uses a different CUDA than we currently have in toolchains, @evgenyrp is correct that we can add another fetch+toolchain task to grab that and use it here.

[evgenyrp]

@ZJaume AI tells me it might be possible, but I know nothing about this specific version of Tensorflow. I guess let's double-check that there's no way to install it the old way by adding another version to the toolchain. If not, let's just do the pip install. It will impact CI costs (not sure how much). In prod Bicleaner takes days anyway, so it doesn't make a difference.