🧊 Circumpolar Run Work Plan

---

Install cluster setup

git clone https://github.com/whrc/Circumpolar_TEM_aux_scripts.git
chmod +x cluster_install.sh
./cluster_install.sh

Instruction on Step-by-Step Tile Run

Initial Setup

1. Copy a Tile

Navigate to /mnt/exacloud/<yourname>_woodwellclimate_org/. Copy a tile into this folder.

gsutil -m  cp -r gs://regionalinputs/CIRCUMPOLAR/<tile_id> .

---

2. Process Climate Data

It could be worse to check the run-mask.nc file in the tile. If all zeros, it means that all gridcells are off. Run Hélène’s gap-filling script:

python ~/Circumpolar_TEM_aux_scripts/process_climate_data_gapfill.py /mnt/exacloud/<yourname>_woodwellclimate_org/<tile_id>

---

3. Analyze Input Data

Check for errors or anomalies (e.g., negative precipitation or NIRR values):

python ~/Circumpolar_TEM_aux_scripts/analyze_TEM_nc.py /mnt/exacloud/<yourname>_woodwellclimate_org/<tile_id>

---

3. Generate Climate Scenarios

Generate SSP scenario using:

python ~/Circumpolar_TEM_aux_scripts/generate_climate_scenarios.py /mnt/exacloud/<yourname>_woodwellclimate_org/<tile_id> /mnt/exacloud/<yourname>_woodwellclimate_org/<tile_id>_sc

---

4. Split Scenarios into Batches

From the directory above the scenario folder, run:

bp batch split -i /mnt/exacloud/<yourname>_woodwellclimate_org/<tile_id>_sc/<sspx_x_x_model_id> -b /mnt/exacloud/<yourname>_woodwellclimate_org/<tile_id>_sc/<sspx_x_x_model_id>_split --p 100 --e 2000 --s 200 --t 124 --n 76

This will create 100 batch folders in the ..._split directory.

---

5. Run Model Batches

• Run all batches at once:

  bp batch run -b /mnt/exacloud/<yourname>_woodwellclimate_org/<tile_id>_sc/<sspx_x_x_model_id>_split`

• Run a single batch: Navigate to the batch folder and run:

  sbatch slurm_runner.sh

---

6. Monitor Execution

Use squeue or sacct to monitor job progress:

squeue -u $USER
squeue --me
sacct -j <job_id>
python ~/Circumpolar_TEM_aux_scripts/check_runs.py /mnt/exacloud/<yourname>_woodwellclimate_org/<tile_id>_sc/<sspx_x_x_model_id>_split
python ~/Circumpolar_TEM_aux_scripts/check_tile_run_completion.py /mnt/exacloud/<yourname>_woodwellclimate_org/<tile_id>_sc/<sspx_x_x_model_id>_split

An auxiliary monitoring script will be added to this repo soon.

---

7. Merge Output

Once all batch jobs are complete:

bp batch merge -b /mnt/exacloud/<yourname>_woodwellclimate_org/<tile_id>_sc/<sspx_x_x_model_id>_split

This will create a folder all_merged folder inside the <sspx_x_x_model_id>_split folder

---

8. Plotting Results

Once all batches are merged, run the plotting script.

python ~/Circumpolar_TEM_aux_scripts/plot_nc_all_files.py /mnt/exacloud/<yourname>_woodwellclimate_org/<tile_id>_sc/<sspx_x_x_model_id>_split/all_merged/

This will create a PDF file containing plots for all merged files. DONE!

Automation all the the above steps. This script has three modes: base, full, and sc

python  ~/Circumpolar_TEM_aux_scripts/automation_script.py -h
python  ~/Circumpolar_TEM_aux_scripts/automation_script.py --mode base <tile_id>
#save to log_file
python  ~/Circumpolar_TEM_aux_scripts/automation_script.py --mode base <tile_id> > {log_file} 2>&1

If you need to run mutiple tiles, make a file in tiles/my_tiles.txt, save tile list into that file

python  ~/Circumpolar_TEM_aux_scripts/automation_script.py run_tiles.py tiles/my_tiles.txt

Run interactively inside the batch folder

/home/<yourname>_woodwellclimate_org/test/dvm-dos-tem/dvmdostem  -f config/config.js -l monitor  -p 100 -e 2000 -s 200 -t 124 -n 76

---

This document serves as a comprehensive work plan and operational guide for managing, processing, and automating large-scale Circumpolar simulations using the DVM-DOS-TEM model. Below is a summary with links to relevant sections:

🔗 Quick Navigation

• Circumpolar Overview and Empty Tiles
• Alaska Stage I Tiles Mapping
• Scenario Run Folder Structure
• Step-by-Step Tile Run Instructions
• Automated Runs Using Scripts
• Canada Stage II Tile Mapping
• To-Do and Automation Checklist

The map of the Circumpolar

Empty tiles

H10_V7	H11_V19	H11_V20	H12_V18	H12_V20	H12_V4
H12_V6	H13_V20	H14_V20	H3_V14	H9_V13

HG: All these tiles have land (as delineated in the example below - blue polygons). They were thus included in the original circumpolar map where a 1 mile buffer were applied to the boundary of the land (as delineated in the example below - pink polygons). However, some of this land (e.g. tiny islands of the Aleutians) were not covered by the vegetation map for instance.

Empty tile could also result from being located entirely on Greenland icefield (e.g. H10_V7)

One last case occuring in this list of empty tiles, is the lack of coverage from the vegetation map in the sourthern most regions (see example 3 below)

Stage I: Alaska Tiles (dynamic OLT) [olt-noncost]

Here we outline the workflow for processing tiles for the Alaska region. Some of the tiles overlap with Canada.

---

Initial Tile Workload Assignment

Elchin	Valeria	Doğukan
H10_V14	H11_V20	H8_V14
H10_V15	H12_V18	H8_V15
H10_V16	H12_V20	H8_V16
H10_V17	H13_V20	H8_V17
H10_V18	H14_V20	H8_V18
H10_V19	H5_V15	H9_V14
H11_V14	H5_V16	H9_V15
H11_V15	H6_V15	H9_V16
H11_V16	H6_V16	H9_V17
H11_V17	H7_V15	H9_V18
H11_V18	H7_V16	H9_V19
H11_V19

After Tile Analysis

Removed tiles with run-status = 0 for all grid cells.

Tile Workload Assignment

Elchin	Valeria	Doğukan
H10_V14		H8_V14
H10_V15		H8_V15
H10_V16		H8_V16
H10_V17		H8_V17
H10_V18		H8_V18
H10_V19	H5_V15	H9_V14
H11_V14	H5_V16	H9_V15
H11_V15	H6_V15	H9_V16
H11_V16	H6_V16	H9_V17
H11_V17	H7_V15	H9_V18
H11_V18	H7_V16	H9_V19

---

Scenario Run Folder Structure

• Region (Alaska)
  • tile_1_sc
  • tile_2_sc
  • ...
  • tile_n_sc
    • ssp1_2_6_access_cm2_split (base_scenario)
    • ssp1_2_6_mri_esm2_0_split
    • ssp2_4_5_access_cm2_split
    • ...
    • ssp5_8_5_mri_esm2_0_split
      • batches_[0...99]
      • all_merged (need to have scenario name in the attr, tile_id, exec time)
        • summary_plots.pdf

Next Scenario: ssp1_2_6_mri_esm2_0__ssp1_2_6

Once Stage I is complete, we can proceed to the next scenario.
For this stage, there is no need to re-run the -pr, -eq, or -sp phases.

HG: Transient doesn't need to be re-run either - unless there is some lingering issues with reasing in the restart files. For this first set of simulations, lets try without repeating historical runs. The appropriate set of flags to run DVM-DOS-TEM in this configuration is: ./dvmdostem -l fatal -f /path/to/config/config.js --no-output-cleanup --restart-run -p 0 -e 0 -s 0 -t 0 -n 76

Instead, we will use the script below to:

1. Copy the restart-tr.nc file into each batch directory.
2. Modify the Slurm job script in each batch to enable a restart run.

Before running this script, the new scenario must first be split into batches — see Step 4 for details.

python generate_next_scenario.py path_to_scenario/ssp1_2_6_access_cm2__ssp1_2_6_split path_to_next_scenario/..._split

After completing this step, repeat Steps 5 through 8 from Stage I.

The following command automates splitting and file copying steps for all scenarios. Keep this script in the folder above the scenario folder.

python orchestrate_scenarios.py --path-to-folder /mnt/exacloud/ejafarov_woodwellclimate_org/Alaska/ \
--tile-dir H10_V14_sc --new-scenario-script generate_next_scenario.py

---

Instructions for Running the Automation Script

Automation of the steps 1-8, next scenario, and the rest of the scenarios

Before running the automation script, please ensure that the folder structure is properly set up. Refer to Scenario Run Folder Structure for full details.

📁 Folder Structure

For each region (e.g., Alaska), the following folders must exist:

• input_tiles/ — contains the original input tiles
• tile#ids_sc/ — subfolders for each tile, e.g.:
  • H10_V14_sc/
  • H10_V15_sc/
  • H10_V16_sc/
• LOG/ — run_tiles.py will automatically create it if it doesn’t exist

⚠️ In the future, tile#ids_sc folders will be moved into corresponding case folders, such as:

• olt_const/
• olt_nonconst/
• olt_nonconst_fire/
• ...

🔤 Notation

• Base scenario: This refers to the one scenario (e.g., sp, eq, pr) that is run in full (including splitting, batch running, merging, and plotting).
• Currently, we support 8 scenarios. Any one of them can serve as the base scenario — we choose ssp1_2_6_access_cm2.

🚀 Running the Script

You can run the automation in different modes depending on your needs:

🔁 Full Mode — Run Base + All Others

python automation_script.py {tile#} --mode full

🧱 Base-Only Mode — Run Only the Base Scenario

python automation_script.py {tile#} --mode base

🔄 Scenario-Only Mode — Run the Remaining 7 Scenarios

python automation_script.py {tile#} --mode sc

Replace {tile#} with the tile ID, e.g. H11_V15.

🧵 Running Multiple Tiles in Background (with screen)

To run tiles without losing progress if your SSH connection is interrupted:

1. Start a new screen session:

   screen

2. Run your tile batch script:

   python run_tiles.py

3. Detach from the screen (keep it running in background):

   • Press Ctrl + A, then D

4. Reattach to your screen session later:

   screen -r

✍️ Editing the run_tiles.py Script

Open run_tiles.py and modify this list:

scs = ["H11_V15", "H11_V16", "H11_V17", ...]

Include the tile IDs you want to process. This script will:

• Run each tile in --mode sc
• Save output logs into the LOG/ folder (automatically created if missing)
• Log file names will follow the pattern:
  LOG/H11_V15.log, LOG/H11_V16.log, etc.

You can monitor progress by checking each corresponding log file:

tail -f LOG/H11_V15.log

---

Stage II: Canada Tiles [constant OLT]

Column 1	Column 2	Column 3
H1_V10	H6_V12	H4_V4
H9_V9	H3_V14 [empty]	H8_V9
H1_V7	H6_V6	H4_V5
H10_V10	H3_V2	H9_V10
H1_V8	H6_V7	H4_V6
H10_V11	H3_V3	H9_V11
H1_V9	H6_V8	H4_V7
H10_V8	H3_V4	H9_V12
H2_V10	H6_V9	H5_V11
H10_V9	H3_V5	H9_V14
H2_V11	H7_V10	H5_V12
H11_V8	H3_V6	H9_V13 [empty]
H2_V12	H7_V11	H5_V13
H11_V9	H3_V7	H5_V14
H2_V5	H7_V12	H5_V15
H4_V8	H3_V8	H5_V16
H2_V6	H7_V6	H5_V5
H5_V10	H3_V9	H6_V13
H2_V7	H7_V7	H6_V14
H5_V6	H4_V10	H6_V15
H2_V8	H7_V8	H6_V16
H5_V7	H4_V11	H7_V13
H2_V9	H7_V9	H7_V14
H5_V8	H4_V12	H7_V15
H3_V10	H8_V10	H7_V16
H5_V9	H4_V13	H8_V13
H3_V11	H8_V11	H8_V14
H6_V10	H4_V14	H8_V15
H3_V12	H8_V12	H8_V16
H6_V11	H4_V3
H3_V13	H8_V8

Notes

• Make sure you are working within your own namespace/folder on Exacloud to avoid conflicts.
• Follow naming conventions carefully to ensure smooth merging and reproducibility.

---

To Do

• Add monitoring script
• Finalize plotting script
• Automate renaming of gap-filled files

automation_script (Elchin/Doğukan)

• pull_tile
• run_gapfill_script
• generate_scenarios
• split_base_scenario
• run_base_scenarios
• wait_for_base_scenarios_to_finish
• merge_base_scenarios
• split_rest_scenarios
• modify_new_scenarios_copy_restart_file
• run_rest_scenarios
• merge_batches
• plot_all_merged_scenarios

stats

• in check_run stat on min, max exec time, and batch#

Post-analysis (Hélène)

• A list of the steps (functional types, benchmarking, etc.)
• Development of the global merge script

To update dvm-dos-tem version

git clone https://github.com/uaf-arctic-eco-modeling/dvm-dos-tem.git 
cd dvm-dos-tem
source /dependencies/setup-env.sh
make USEMPI=true

Extracting gridcell from the batch

bp extract_cell --input-path batch_#/input --output-path <path-to-output> -X x -Y y --p 100 --e 2000 --s 200 --t 123 --n 76

Resubmitting failed batches

python <path_to_folder>/resubmit_unfinished.py H7_V15_sc/ssp1_2_6_access_cm2_split/

HG: Note related to automation of scenario simulations. 1- scenario and GCM names should be stored in the metadata of the outputs 2- sets of outputs per scenario should be stored in different directories. The directory name should include the name of the scenario and GCM 3- path to restart files should be included in the config file, so there is no more need to copy the restart file in every scenario output directory...

Merging tile outputs [HG]

I wrote a python script that would loop through all the outputs tiles in a directory and merge the output tiles into single files per variable. This script needs: 1- all batch outputs per tile to be merged. 2- historical simulations to be stored in a specific directory (called tr preferably) 3- simulations to be done with dvm-dos-tem version 0.8.3, branch names new_outspec. Here is the path to the branch: https://github.com/uaf-arctic-eco-modeling/dvm-dos-tem/tree/new_outspec/

The script has a summary function that will summarize output variables is turned on.

The script is very crude at the moment, but it works. Give it a try and let me know what it works for you.

The script is stored in our input_production repository, in the branch named merge_tile_outputs. Here is the git path to the branch: https://github.com/uaf-arctic-eco-modeling/Input_production/tree/merge_tile_outputs The script is stored in ./utils/merge_out_synth.py