enable composing input data

[juliasloan25]

Purpose

closes #41

This PR adds functionality to DataHandler, TimeVaryingInputs, and SpaceVaryingInputs to enable composing multiple input variables into one data variable. To do this, the user must specify a composing function, multiple variable names, and the file paths where they can be read from.

Most of the changes have been made at the DataHandler level. Each input variable has its own unique FileReader object, and each composed data variable has one Time/SpaceVaryingInput and one DataHandler. The composing function itself is applied in the regridded_snapshot function, just before regridding. The user will interact with this feature at the Time/SpaceVaryingInput level.

This feature is only available when using InterpolationsRegridder, not TempestRegridder.

Design decisions made include:

• If a pre-processing function is provided, it is applied to each input variable before they are composed.
• Variables are composed before regridding, to preserve higher resolution information
• We assume that all input variables have the same temporal and spatial dimensions. This is explicitly checked in the DataHandler constructor, and will raise an informative error message if it is not true.
• Multiple input variables can come from one file, or each from their own unique file. We don't currently support arbitrary numbers of input variables and files, since this would require more work to implement and is not an expected use case in the near term.

To-do

• DataHandler: add field to store Dict mapping input varnames to FileReaders
• DataHandler: add field to store list of input varnames
• DataHandler: add field to store composing function
• DH constructor: assert number of file paths supplied == number of varnames
• DH constructor: assert compose function is provided when multiple input vars provided
• DH regrid_snapshot: apply compose function to input vars, regrid result and return it
• SVI/TVI constructors: add fields for multiple file paths, multiple varnames, composing function (optional)
• update all docs
• add usage example to docs

Tests

• DH constructor: num file_paths != num varnames && num file_paths > 1 (should fail)
• DH constructor: multiple input vars, no compose func (should fail)
• DH constructor: 1 input var, compose func (should fail)
• DH constructor: TempestRegrid with multiple vars (should fail)
• DH constructor: 1 input var, no compose func (should pass)
• DH constructor: multiple input vars, compose func (should pass)
• SVI/TVI constructors with + without varnames

Content

---

• I have read and checked the items on the review checklist.

[Sbozzolo]

We should support passing only one file path with multiple variables: thats the most common use case

…

On Thu, Sep 5, 2024, 10:58 AM Julia Sloan ***@***.***> wrote: Purpose closes #41 <#41> To-do - DataHandler: add field to store Dict mapping input varnames to FileReaders (maintain one file_reader for composed data variable) - DataHandler: add field to store list of input varnames - DataHandler: add field to store composing function - DH constructor: assert number of file paths supplied == number of varnames - DH constructor: assert compose function is provided when multiple input vars provided - DH constructor: apply compose function to input vars, create data var - SVI/TVI constructors: add fields for multiple file paths, multiple varnames, composing function (optional) Tests - DH constructor: num file_paths != num varnames (should fail) - DH constructor: multiple input vars, no compose func (should fail) - DH constructor: 1 input var, compose func (should fail, or warn that using identity) - DH constructor: 1 input var, no compose func (should pass) - DH constructor: multiple input vars, compose func (should pass) - SVI constructors with + without varnames - TVI constructors with + without varnames Content ------------------------------ - I have read and checked the items on the review checklist. ------------------------------ You can view, comment on, or merge this pull request online at: #105 Commit Summary - 03ef58f <03ef58f> DH, SVI some changes File Changes (2 files <https://github.com/CliMA/ClimaUtilities.jl/pull/105/files>) - *M* ext/DataHandlingExt.jl <https://github.com/CliMA/ClimaUtilities.jl/pull/105/files#diff-95c9e30bd3fcf6388f3a6346ff337b9305fece08bbe9343199f832843b4bd2f2> (140) - *M* ext/SpaceVaryingInputsExt.jl <https://github.com/CliMA/ClimaUtilities.jl/pull/105/files#diff-c6f8c06af2248d2e1fd39f9a5690f588700eb35db050f5079976f960df9811cc> (9) Patch Links: - https://github.com/CliMA/ClimaUtilities.jl/pull/105.patch - https://github.com/CliMA/ClimaUtilities.jl/pull/105.diff — Reply to this email directly, view it on GitHub <#105>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACF6E7MU3RW2O2GU5QSCHXDZVCLTJAVCNFSM6AAAAABNXAKHXSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGUYDQNBQHE4TOMA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[juliasloan25]

We should support passing only one file path with multiple variables: thats
the most common use case

Okay, I'll change it to support that. Do we also want to support passing multiple file paths for multiple variables? I think this would become complicated quickly because there may not be a one-to-one mapping, i.e. in the case num file paths > 1 && num variables > 1 && num file paths != num variables, we would need more infrastructure to map variables to file paths

[Sbozzolo]

We can just do:

• 1 file, multiple variables,
• N files, N variables

We can add more if need arises.

[Sbozzolo]

Thank you!

I left some small comments and a couple of suggestions for cleaner code.