Only constrain using stash code or field name in the iris.load call#1718
Only constrain using stash code or field name in the iris.load call#1718SGallagherMet wants to merge 5 commits intoMetOffice:mainfrom
Conversation
…any recipes to configure.
…he stash code or field name during the initial call to iris.load.
…'t find any recipes to configure." as this should be part of a separate branch/PR. This reverts commit 80d9cc3.
… used on multiple models at once.
|
Given this is a performance change, I'd ideally like to do it without changing the API. Specifically that will involve doing it inside of The other aspect to this is that we currently apply a bunch of load-time callbacks to the loaded cubes, which crucially apply before they are filtered. Therefore we are already paying a significant price to look at all that metadata. We may need to split that processing into two as well, so we only do the variable name normalisation. That said, if you are already seeing a significant performance improvement with this change already, it is definitely worth investigation. Load speed has always been a bit slow in CSET, especially when loading multiple files. We previously had a pre-processor that munged each case into a single file before doing the diagnostics on it, which gave a significant speedup when running many tasks, but slowed down small numbers of tasks due to having to do a copy of the whole dataset. It also added significant complexity (and bugs) to the loading, so it was dropped. |
Iris has a built in 'shortcut' mechanism if the only constraint passed in to a an
iris.loadcall is a single stash code or a single field name. This PR changes the loading process of the recipes that only use a single parameter to first load the data, then apply other constraints using a filter step.For models with many parameters and frequent output this can significantly speed up the loading process as Iris can skip fields without fully loading their metadata to check the remaining constraints. A quick test on one of our UKV trial outputs suggested the run of the aggregation step was approximately halved.
Contribution checklist
Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.