GPU train, CPU test

[Alx-nder]

I am having difficulty deploying the model on a CPU-only device. I trained on GPU-enabled device then saved models as joblib. When I try to load and run on CPU device, i get the error message: "AssertionError: Torch not compiled with CUDA enabled". Is there something I could do to fix this?

[iivalchev]

Hi @Alx-nder did you just fit the model on GPU? If so there is no need for that. Skip the saving and just fit and do inference on the CPU device you would not get any performance penalty.

[Alx-nder]

Training on CPU will take ages. Ideally, I would love to train on GPU and then save models then do inference anywhere else (for example, think egde devices (kinda) that are not as powerful but need to make decisions given data).

The trained model, at inference, would look for CUDA resources, but I would map them to CPU cores instead.

Is this a reasonable thing to do?

[iivalchev]

What do you mean by training? If you just fit the model having a GPU or not is irrelevant it would simply create the preprocessing pipeline and preprocess the training data (depending on the fit_mode)

[Alx-nder]

I want to fit (or train) once, then call predict multiple times at different places. I think from what I see in the documentation, and from what you've said, I may have to call fit on each individual device. This is fine.
Thank you, iivalchev.