Iris 3.3.1 fails unhelpfully when it finds a bad cell_methods attribute

[alistairsellar]

🐛 Bug Report

Loading a file produced by the ESA CCI project, it fails with IndexError: list index out of range (full traceback in additional context section). To be fair on Iris, the file contains a dodgy cell_methods attribute (see also in additional context) and fails cfchecker as a result. (Deleting the cell_methods attribute with ncatted prevents the error.)

The alleged bug is the unhelpful error message (though the traceback gives a clue as to where the issue is).

How To Reproduce

Steps to reproduce the behaviour:

1. iris.load('/data/users/esmval/ESMValTool/temporary/obs/Tier2/ESACCI-SOILMOISTURE/OBS_ESACCI-SOILMOISTURE_sat_L3S-SSMV-COMBINED-v4.2_Lmon_sm_197901-201612.nc')

(Not saying which system that path is on, but AVD team may be able to guess ;o). The same file can be found on JASMIN at /gws/nopw/j04/esmeval/obsdata-v2/Tier2/ESACCI-SOILMOISTURE/OBS_ESACCI-SOILMOISTURE_sat_L3S-SSMV-COMBINED-v4.2_Lmon_sm_197901-201612.nc)

Expected behaviour

Either return a cube (potentially with a warning about the CF non-compliance), or provide an informative error message.

Iris 3.2.1 returned a cube with no warning, and no cell_methods attribute, which was acceptable behaviour for me at least. Note that both Iris 3.2.1 and 3.3.1 happily ignore bad cell_measures and standard_name attributes in the same file.

Screenshots

Environment

• OS & Version: RHEL7
• Iris Version: 3.3.1 (note that Iris 3.2.1.post0 loaded the file without complaint)

Additional context

Traceback and partial netCDF header in details...

Click to expand this section...

netCDF header for offending variable:

        float sm(time, lat, lon) ;
                sm:standard_name = "" ;
                sm:units = "m3 m-3" ;
                sm:cell_methods = "time" ;
                sm:cell_measures = "area" ;
                sm:long_name = "Volumetric Moisture in Upper Portion of Soil Column" ;
                sm:comment = "the volume of water in all phases in a thin surface soil layer." ;
                sm:_FillValue = 1.e+20f ;

Traceback

>>> import iris
>>> cube = iris.load('OBS_ESACCI-SOILMOISTURE_sat_L3S-SSMV-COMBINED-v4.2_Lmon_sm_197901-201612.nc')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/__init__.py", line 314, in load
    return _load_collection(uris, constraints, callback).merged().cubes()
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/__init__.py", line 280, in _load_collection
    result = _CubeFilterCollection.from_cubes(cubes, constraints)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/cube.py", line 104, in from_cubes
    for cube in cubes:
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/__init__.py", line 265, in _generate_cubes
    for cube in iris.io.load_files(part_names, callback, constraints):
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/io/__init__.py", line 208, in load_files
    for cube in handling_format_spec.handler(
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/netcdf.py", line 976, in load_cubes
    cube = _load_cube(engine, cf, cf_var, filename)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/netcdf.py", line 689, in _load_cube
    engine.activate()
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/_nc_load_rules/engine.py", line 97, in activate
    run_actions(self)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/_nc_load_rules/actions.py", line 521, in run_actions
    action_default(engine)  # This should run the default rules.
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/_nc_load_rules/actions.py", line 74, in inner
    rule_name = func(engine, *args, **kwargs)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/_nc_load_rules/actions.py", line 90, in action_default
    hh.build_cube_metadata(engine)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/_nc_load_rules/helpers.py", line 215, in build_cube_metadata
    cube.cell_methods = parse_cell_methods(nc_att_cell_methods)
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/netcdf.py", line 294, in parse_cell_methods
    for m in _split_cell_methods(nc_cell_methods):
  File ".........../default-2022_11_22/lib/python3.9/site-packages/iris/fileformats/netcdf.py", line 253, in _split_cell_methods
    method_indices.append((name_start_inds[-1], len(nc_cell_methods)))
IndexError: list index out of range

[acchamber]

This also was the case with #5237, although with some AVD help we did manage to find the issue

[rcomer]

I can't reproduce the failure with v3.3.1, but the file does not seem to have been modified since January 😕

In [1]: import iris

In [2]: iris.load('/data/users/esmval/ESMValTool/temporary/obs/Tier2/ESACCI-SOILMOISTURE/OBS_ESACCI-SOILMOISTURE_sat_L3S-SSMV-COMBINED-v4.2_Lmon_sm_197901-201
   ...: 612.nc')
Out[2]: [<iris 'Cube' of Volumetric Moisture in Upper Portion of Soil Column / (m3 m-3) (time: 456; latitude: 720; longitude: 1440)>]

In [3]: iris.__version__
Out[3]: '3.3.1'

[alistairsellar]

Thanks for investigating @rcomer. I think that the file has been updated, but as a mirror of larger tree, so the file timestamp probably reflects the update time in the source of the mirror.

I've taken a copy and hacked the problem back into the header. New bad file is: /data/users/hadtq/esmvaltool/OBS_ESACCI-SOILMOISTURE_sat_L3S-SSMV-COMBINED-v4.2_Lmon_sm_197901-201612.nc. This file reproduces the error, for me at least.

Note that the fix to the source file means that this is a less pressing issue for me at least. Though I'd still argue that Iris should fail more gracefully in such cases, as a nice-to-have.

[bjlittle]

May be related to issue #5119 and addressed by pull-request #5126.

[bjlittle]

Discussed at @SciTools/peloton, when/if #5126 is banked we can see whether it also addresses this issue.

[rcomer]

Have confirmed locally that #5126 would fix this.

[github-actions]

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

[trexfeathers]

We have some plans for Iris 3.12 to help with things like this