Make extracted raw data location configurable, with default to work#360
Make extracted raw data location configurable, with default to work#360
work#360Conversation
work
There was a problem hiding this comment.
I don't think this is doing what it's expected to.
When I set
RAW_DATA_DIR="<my_scratch_dir>/raw_data_test"
I can see that the directory is created, but I watched it throughout the CMEW run (repeatedly refreshing) and nothing appeared there. When I went back though the logs, I noticed these lines in the log (~/cylc-run/CMEW/run7/share/data/cdds/proc/GCModelDev/ESMVal/u-bv526/round-1/extract/log/cdds_extract_apm_2026-02-09T1202Z.log):
2026-02-09 12:02:23 cdds.extract.common.run_moo_cmd INFO: moo command: '['moo', 'select', '-i', '-d', '<my_home_dir>/cylc-run/CMEW/run7/share/data/cdds/proc/GCModelDev/ESMVal/u-bv526/round-1/extract/apm_1993-01-01T00:00:00Z_1993-12-01T00:00:00Z.dff', 'moose:/crum/u-bv526/apm.pp', '<my_home_dir>/cylc-run/CMEW/run7/share/data/cdds/cdds_data/GCModelDev/ESMVal/HadGEM3-GC31-LL/amip/r5i1p1f3/round-1/input/u-bv526/apm']'
So I think it is still extracting to the shared directory.
| MAX_PARALLEL_TASKS=4 | ||
| MODEL_ID="HadGEM3-GC5E-LL" | ||
| NUMBER_OF_YEARS=1 | ||
| RAW_DATA_DIR="" |
There was a problem hiding this comment.
Can you add some info to CMEW/meta/rose-meta.conf about the new variable?
e.g. it shouldn't be compulsory, the user might need help with when to specify it...
| MAX_PARALLEL_TASKS=4 | ||
| MODEL_ID="HadGEM3-GC5E-LL" | ||
| NUMBER_OF_YEARS=1 | ||
| RAW_DATA_DIR="" |
There was a problem hiding this comment.
Is it also worth adding some error checking?
e.g. when I accidentally set this to RAW_DATA_DIR="<my_scratch>/raw_data_test/" but failed to copy the initial "/", it created ~/cylc-run/CMEW/run5/work/1/standardise_model_data/<PATH_ABOVE>/proc.
Maybe we don't care.
Closes #156 .
PR creation checklist for the developer
<issue_number>above ☝️ been replaced with the issue number?mainbeen selected as the base branch?<issue_number>_<short_description_of_feature>?good first issuelabel) been added to the PR?Climate Model Evaluation Workflow (CMEW)project been added to the PR?Definition of Done for the developer
docdirectory) related to the change been updated appropriately, including the Quick Start section?PR creation checklist for the reviewer
<issue_number>above ☝️ been replaced with the issue number?mainbeen selected as the base branch?<issue_number>_<short_description_of_feature>?good first issuelabel) been added to the PR?Climate Model Evaluation Workflow (CMEW)project been added to the PR?Definition of Done for the reviewer
docdirectory) related to the change been updated appropriately, including the Quick Start section?