Skip to content

Modify METPlus apps to facilitate using at multiple sites #1906

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JorgeBornemann wants to merge 1 commit into
base: main
Choose a base branch
from
+44 −197
Open

Modify METPlus apps to facilitate using at multiple sites #1906

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 0 additions & 101 deletions src/CSET/cset_workflow/app/metplus_ascii2nc/file/ASCII2NC_python_embedding.conf
View file
Open in desktop
Copy link
Collaborator Author

@JorgeBornemann JorgeBornemann Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gone to restricted files repo

This file was deleted.

68 changes: 0 additions & 68 deletions src/CSET/cset_workflow/app/metplus_ascii2nc/file/python/read_ascii_point.py
View file
Open in desktop
Copy link
Collaborator Author

@JorgeBornemann JorgeBornemann Feb 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gone to restricted files repo

This file was deleted.

8 changes: 4 additions & 4 deletions src/CSET/cset_workflow/app/metplus_ascii2nc/opt/rose-app-niwa.conf
View file
Open in desktop
Copy link
Member

@jfrost-mo jfrost-mo Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With these files now under a site-specific directory, we don't need the site in the filename.

While we are at it, do we want to update any of these references to NIWA to ESNZ before it goes in? I'll leave it up-to-you.

Copy link
Collaborator Author

@JorgeBornemann JorgeBornemann Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I am still awaiting a bit of clarification on the site. maybe ESNZ will not be suitable, as I think our domain name will be NZIES (New Zealand Institute for Earth Sciences). So we can change names and remove the redundant info in a later ticket?

Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
[command]
default=app_env_wrapper run_metplus.py ASCII2NC_python_embedding.conf
default=app_env_wrapper run_metplus.py niwa/restricted_ASCII2NC_python_embedding_niwa.conf

[env]
CONDA_VENV_LOCATION = ${CONDA_METPLUS_VENV_LOCATION}
CONDA_VENV_LOCATION=${CONDA_METPLUS_VENV_LOCATION}
METPLUS_OBS_DIR=${METPLUS_OBS_DIR}
INPUT_READ_SCRIPT=restricted_read_ascii_point_niwa.py
TASK_TIME=20230705T00
TASK_ENDTIME=20230706T18
START_TIME=${TASK_START_TIME}
END_TIME=${TASK_END_TIME}
5 changes: 3 additions & 2 deletions src/CSET/cset_workflow/app/metplus_point_stat/opt/rose-app-niwa.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
[command]
default=app_env_wrapper run_metplus.py restricted_PointStat_cycling_niwa.conf restricted_PointStat_science_niwa.conf
default=app_env_wrapper run_metplus.py niwa/restricted_PointStat_cycling_niwa.conf niwa/restricted_PointStat_science_niwa.conf

[env]
CONDA_VENV_LOCATION = ${CONDA_METPLUS_VENV_LOCATION}
METPLUS_FCST_DIR = ${METPLUS_FCST_DIR}
TIME_START=20230705T00
START_TIME=${TASK_START_TIME}
FORECAST_LENTGH = ${FORECAST_LENGTH}
19 changes: 13 additions & 6 deletions src/CSET/cset_workflow/flow.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,12 +116,15 @@ final cycle point = {{CSET_TRIAL_END_DATE}}

[[METPLUS]]
[[[environment]]]
{% if METPLUS_GRID_STAT|default(False) %}
METPLUS_ANA_DIR = {{METPLUS_ANA_DIR}}
METPLUS_FCST_DIR = {{METPLUS_FCST_DIR}}
METPLUS_OBS_DIR = {{METPLUS_OBS_DIR}}
ROSE_APP_OPT_CONF_KEYS = {{METPLUS_OPT_CONFIG_KEYS}}
{% endif %}
METPLUS_ANA_DIR = {{METPLUS_ANA_DIR}}
METPLUS_FCST_DIR = {{METPLUS_FCST_DIR}}
METPLUS_OBS_DIR = {{METPLUS_OBS_DIR}}
ROSE_APP_OPT_CONF_KEYS = {{METPLUS_OPT_CONFIG_KEYS}}
Comment on lines +119 to +122
Copy link
Member

@jfrost-mo jfrost-mo Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these settings defined (even if to "") when METplus is not enabled in the rose-suite.conf? Or are they trigger ignored?

Copy link
Collaborator Author

@JorgeBornemann JorgeBornemann Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good point, they are used by point_stat and grid_stat. I don't know why they were coded only for grid_stat, and that is why I removed the check, but I can see it can break things outside NIWA.

Copy link
Collaborator Author

@JorgeBornemann JorgeBornemann Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put them in a includes/metplus_common.cylc file with the relevant logicals.

TASK_START_TIME = $(cylc cyclepoint --template CCYYMMDDThh)
# Using sed to parse only the numerical values of ANALYSIS_LENGTH, assumes is always in the format PThhH
# Done this way because isodatetime CLI doesn't seem to be able to parse durations
FORECAST_LENGTH = $(echo {{ ANALYSIS_LENGTH }} | sed 's/[^0-9]//g')
TASK_END_TIME = $(cylc cyclepoint --offset-hours ${FORECAST_LENGTH} --template CCYYMMDDThh)

# Noop tasks to ensure a complete/efficient workflow graph.
[[DUMMY_TASK]]
Expand Down Expand Up @@ -217,5 +220,9 @@ final cycle point = {{CSET_TRIAL_END_DATE}}
WEB_ADDR = {{WEB_ADDR}}
WEB_DIR = {{WEB_DIR}}

# Include files bring their own graph and runtime sections.
{% include 'includes/metplus_point_stat.cylc' %}
{% include 'includes/metplus_grid_stat.cylc' %}

# Site-specific details that add to or override the core workflow definition.
{% include 'site/' ~ SITE ~ '.cylc' %}
24 changes: 24 additions & 0 deletions src/CSET/cset_workflow/includes/metplus_point_stat.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{% if METPLUS_POINT_STAT|default(False) %}
[scheduling]
[[graph]]
{% if CSET_CYCLING_MODE == "case_study" %}
{% for date in CSET_CASE_DATES %}
R1/{{date}} = """
metplus_ascii2nc => metplus_point_stat
"""
{% endfor %}
{% elif CSET_CYCLING_MODE == "trial" %}
{{CSET_TRIAL_CYCLE_PERIOD}} = """
metplus_ascii2nc => metplus_point_stat
"""
{% endif %}

[runtime]
[[metplus_ascii2nc]]
# Runs METplus wrappers for ASCII to nc ingestion of obs.
inherit = METPLUS

[[metplus_point_stat]]
# Runs METplus wrappers for point stat calculations.
inherit = METPLUS
{% endif %}
16 changes: 0 additions & 16 deletions src/CSET/cset_workflow/includes/point_stat.cylc
View file
Open in desktop

This file was deleted.