Not working in Python 3.12 / consider removing hdf5storage dependency

[smithara]

The current released version of hdf5storage is not working with Python 3.12+ (and the latest main on GitHub doesn't work with numpy 2), so can't use chaosmagpy on Python 3.12+. hdf5storage does not seem to be well maintained, so it's probably best to switch to an alternative (if there is one?).

[ancklo]

Thanks for making me aware of this issue.

I have experienced the same problems with the hdf5storage package and I often thought about moving away from it. The problem is that we use MAT-files to store the CHAOS coefficients, which are then loaded into chaosmagpy using hdf5storage.

I think the newer MAT-files can be loaded in like HDF5-files, so maybe the h5py package could be an alternative for at least loading in the CHAOS coefficients. However, there are also functions in chaosmagpy for writing MAT-files, which rely on the hdf5storage package. I would have to remove those completely. However, those functions are used in the actual production of the CHAOS files after the model estimation, which is done in Python.

I think as a step toward replacing the hdf5storage package, we could start using h5py for loading the CHAOS model files, which is also what users care about the most. On the model production side, it is probably best to write a script in MATLAB to produce the MAT-file from the outputs of our in-house Python modeling software.

[smithara]

(on re-reading I realise I used the unhelpful "not working" line! I should have written that it failed at import time because of a problem with hdf5storage needing distutils, though now I don't seem to find the problem in a fresh environment!)

Yeah it seems the situation in Python right now for MAT 7.3 files is not good. I was hoping scipy loadmat/savemat might be an option but there doesn't seem to be a plan to support HDF5 that MAT-7.3 uses.

Rewriting to use h5py might be the most reliable option, but I've found that https://github.com/joshayers/hdf5storage/tree/numpy2 is working on taking over hdf5storage. I'll keep an eye on it.

The packaging of chaosmagpy will also need some maintenance to work on python 3.13 so I think I should look at these together.

[ancklo]

It is good to hear that someone is working on an update of hdf5storage. However, I'm not sure how this will change the maintenance of the package.

So, as you say, using h5py might be more reliable. In addition, Mathworks may change the MAT-file storage conventions in the future but will probably keep using the HDF5 format.

[eureka-0]

Hi, I found that hdf5storage can work with python 3.12 after installing setuptools which provides distutils (removed from 3.12).

[ancklo]

Thank you for providing this fix, @eureka-0 . I will try to test if this works and then add setuputils as a dependency to chaosmagpy. Although it sounds like this should be better handled in hdf5storage.

[smithara]

hdf5storage has been taken over by someone else and a new 0.2 version is released which should fix this now! (https://pypi.org/project/hdf5storage/#history)

[eelcodoornbos]

Can I suggest a different solution, which is to not use .MAT files at all? There are alternative, more friendly and compatible formats for distributing datasets.

[ancklo]

Thanks for the comment. I agree that MAT-files should be eventually replaced by something else as the main way of distributing the model, and should only be available as an alternative if really needed.

Which formats could be considered? It must allow for structured datasets. I guess most people working with ESA data are able to read CDF? Although personally, I would prefer HDF5.