Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
name: Bug report
description: Report broken behavior, runtime errors, or incorrect outputs.
title: "[bug] "
labels:
- bug
body:
- type: markdown
attributes:
value: |
Before submitting:
- add labels for `type`, `area`, `priority`, and `provenance`
- if this is GEE-related, note whether Earth Engine auth and `GCP_PROJECT_ID` were set
- include exact CLI command or minimal repro

Recommended labels:
- type: `bug`
- area: e.g. `gee`, `fetch-data`, `historical-cli`, `nex-gddp`, `weather-station`, `hazards`
- priority: `priority:high|medium|low`
- provenance: `sammy`, `michel`, `codex`, or `user-feedback`
- type: textarea
id: summary
attributes:
label: Problem summary
description: What broke?
placeholder: Clear one-paragraph description of problem.
validations:
required: true
- type: textarea
id: repro
attributes:
label: Exact command or minimal repro
description: Paste exact CLI command, Python snippet, or steps.
render: bash
validations:
required: true
- type: textarea
id: observed
attributes:
label: Observed behavior
description: Include exact error text, traceback, or wrong output.
validations:
required: true
- type: textarea
id: expected
attributes:
label: Expected behavior
description: What should have happened instead?
validations:
required: true
- type: textarea
id: context
attributes:
label: Environment and context
description: OS, Python version, auth state, source, date range, cache status, etc.
placeholder: |
OS:
Python:
Source:
Date range:
Earth Engine auth:
GCP_PROJECT_ID set:
Cold-cache or warm-cache:
validations:
required: false
- type: textarea
id: labels
attributes:
label: Labels to apply
description: List labels you think fit if you cannot apply them yourself.
placeholder: bug, gee, fetch-data, priority:high, michel
validations:
required: false
5 changes: 5 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
blank_issues_enabled: true
contact_links:
- name: README setup and usage guide
url: https://github.com/CGIAR-Climate-Data-Hub/climate-toolkit#readme
about: Check installation, Earth Engine setup, and CLI usage before opening a new issue.
62 changes: 62 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
name: Feature or enhancement
description: Propose new capability, UX improvement, research task, or refactor slice.
title: "[enhancement] "
labels:
- enhancement
body:
- type: markdown
attributes:
value: |
Before submitting:
- add labels for `type`, `area`, `priority`, and `provenance`
- use `research` / `methodology` when issue depends on external evidence
- use `needs-docs` when user-facing behavior or CLI changes are expected

Recommended labels:
- type: `enhancement`, `research`, `methodology`, `performance`, `refactor`, `documentation`
- area: e.g. `gee`, `fetch-data`, `historical-cli`, `nex-gddp`, `weather-station`, `hazards`, `packaging`, `readme`
- priority: `priority:high|medium|low`
- provenance: `sammy`, `michel`, `codex`, or `user-feedback`
- type: textarea
id: summary
attributes:
label: Proposal summary
description: What should be added or changed?
validations:
required: true
- type: textarea
id: problem
attributes:
label: Problem or motivation
description: Why does this matter?
validations:
required: true
- type: textarea
id: scope
attributes:
label: Suggested scope
description: What should be in scope? What can wait?
validations:
required: false
- type: textarea
id: acceptance
attributes:
label: Acceptance ideas
description: How should we know this is done?
validations:
required: false
- type: textarea
id: references
attributes:
label: References or related issues
description: Literature, docs, issue links, screenshots, prior discussion.
validations:
required: false
- type: textarea
id: labels
attributes:
label: Labels to apply
description: List labels you think fit if you cannot apply them yourself.
placeholder: enhancement, ux, historical-cli, priority:medium, codex
validations:
required: false
71 changes: 38 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,44 @@ normal user workflows.

## Quick Start

### Earth Engine setup

Most historical gridded defaults and all current NEX-GDDP real-access paths use
Earth Engine-backed retrieval. Do this first.

Important:

- `YOUR_PROJECT_ID` below is placeholder text
- replace it with your real Google Cloud **Project ID**
- do not paste literal string `YOUR_PROJECT_ID`

```bash
python -c "import ee; ee.Authenticate()"
export GCP_PROJECT_ID=YOUR_PROJECT_ID
python -c "import ee; ee.Initialize(project='$GCP_PROJECT_ID'); print('EE init OK')"
```

Windows PowerShell:

```powershell
python -c "import ee; ee.Authenticate()"
$env:GCP_PROJECT_ID="YOUR_PROJECT_ID"
python -c "import ee; ee.Initialize(project='$env:GCP_PROJECT_ID'); print('EE init OK')"
```

Required in `.env.example`:

- `GCP_PROJECT_ID`
- optional `EARTHDATA_USERNAME` / `EARTHDATA_PASSWORD` for sources that still
use Earthdata-backed access

If you see:

- `Earth Engine project ID missing` -> set `GCP_PROJECT_ID`
- `Project 'projects/YOUR_PROJECT_ID' not found or deleted` -> placeholder was
not replaced with real project ID
- auth refresh / DNS errors -> refresh Earth Engine auth and check internet/DNS

### Installation

1. Clone repository
Expand Down Expand Up @@ -165,39 +203,6 @@ Reference note:

- `docs/distribution_workflow.md`

### Earth Engine setup

Most historical gridded defaults and all current NEX-GDDP real-access paths use
Earth Engine-backed retrieval. Before running those sources, authenticate Earth
Engine and set project ID.

```bash
python -c "import ee; ee.Authenticate()"
python -c "import ee; ee.Initialize(project='YOUR_PROJECT_ID')"
export GCP_PROJECT_ID=YOUR_PROJECT_ID
```

Windows PowerShell:

```powershell
python -c "import ee; ee.Authenticate()"
python -c "import ee; ee.Initialize(project='YOUR_PROJECT_ID')"
$env:GCP_PROJECT_ID="YOUR_PROJECT_ID"
```

Required in `.env.example`:

- `GCP_PROJECT_ID`
- optional `EARTHDATA_USERNAME` / `EARTHDATA_PASSWORD` for sources that still
use Earthdata-backed access

If you see:

- `Earth Engine project ID missing` -> set `GCP_PROJECT_ID`
- `Project 'projects/your-ee-project-id' not found or deleted` -> you left
placeholder value in environment or auth init command
- auth refresh / DNS errors -> refresh Earth Engine auth and check internet/DNS

### Recommended starting point

For historical daily climate workflows, start with:
Expand Down
9 changes: 7 additions & 2 deletions climate_tookit/fetch_data/source_data/sources/xee_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@
DEFAULT_EE_OPT_URL = "https://earthengine-highvolume.googleapis.com"
METERS_PER_DEGREE = 111_320.0
MISSING_EE_PROJECT_ID_PREFIX = "Earth Engine project ID is required."
EARTH_ENGINE_SETUP_URL = (
"https://github.com/CGIAR-Climate-Data-Hub/climate-toolkit#earth-engine-setup"
)


def import_xee_stack(required_for: str = "xee_common"):
Expand Down Expand Up @@ -59,7 +62,8 @@ def format_ee_setup_error(exc: Exception) -> str:
return (
"Earth Engine project ID missing. Set GCP_PROJECT_ID "
"(or GOOGLE_CLOUD_PROJECT / EE_PROJECT_ID) and retry. "
"Example: export GCP_PROJECT_ID=your-ee-project-id"
"Example: export GCP_PROJECT_ID=your-ee-project-id. "
f"Setup guide: {EARTH_ENGINE_SETUP_URL}"
)
lowered = message.lower()
if (
Expand All @@ -71,7 +75,8 @@ def format_ee_setup_error(exc: Exception) -> str:
return (
"Earth Engine auth refresh failed. Check internet/DNS access, then "
"refresh auth if needed with: "
".venv/bin/python -c \"import ee; ee.Authenticate(); ee.Initialize(project='YOUR_PROJECT_ID')\""
".venv/bin/python -c \"import ee; ee.Authenticate(); ee.Initialize(project='YOUR_PROJECT_ID')\". "
f"Setup guide: {EARTH_ENGINE_SETUP_URL}"
)
return message

Expand Down
50 changes: 50 additions & 0 deletions demo_issue_69.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
#!/usr/bin/env bash
# Demo for issue #69: NEX-GDDP batch "Earth Engine project ID missing".
# Shows the failure (no project id) then a cold-start live fetch (project id set,
# cache refreshed, fresh cache dir => no cached data used).
#
# Usage:
# ./demo_issue_69.sh <EE_PROJECT_ID>
# ./demo_issue_69.sh ee-peetmate
#
# Requires: .venv with earthengine-api, and EE auth already done
# (~/.config/earthengine/credentials). If not authed:
# .venv/bin/python -c "import ee; ee.Authenticate()"

set -u

PROJECT_ID="${1:-}"
PY=".venv/bin/python"
CACHE_DIR="/tmp/nexgddp_coldtest"

CMD_ARGS=(
-m climate_tookit.fetch_data.nex_gddp_batch
--variables precipitation,max_temperature,min_temperature
--from 2050-01-01 --to 2050-01-10
--lon 36.817 --lat -1.286
--model GFDL-ESM4 --scenario ssp245
--stage raw
)

echo "=============================================================="
echo " STEP 1: reproduce the bug — no project id set"
echo "=============================================================="
env -u GCP_PROJECT_ID -u GOOGLE_CLOUD_PROJECT -u EE_PROJECT_ID \
"$PY" "${CMD_ARGS[@]}"
echo
echo "(expected: 'Error: Earth Engine project ID missing.')"
echo

if [ -z "$PROJECT_ID" ]; then
echo "No EE project id passed. Re-run with: ./demo_issue_69.sh <EE_PROJECT_ID>"
exit 1
fi

echo "=============================================================="
echo " STEP 2: cold-start live fetch — project id set, cache wiped"
echo "=============================================================="
rm -rf "$CACHE_DIR" # cold start: no cached data
GCP_PROJECT_ID="$PROJECT_ID" \
"$PY" "${CMD_ARGS[@]}" --refresh-cache --cache-dir "$CACHE_DIR"
echo
echo "(expected: 'fetched' (not 'cache hit') + 10 rows of data)"
2 changes: 2 additions & 0 deletions tests/test_fetch_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -802,6 +802,7 @@ def test_main_prints_simple_message_when_ee_project_id_missing(self):
self.assertEqual(1, exit_code)
self.assertIn("Earth Engine project ID missing.", output)
self.assertIn("GCP_PROJECT_ID", output)
self.assertIn("earth-engine-setup", output)

def test_source_data_main_prints_simple_message_when_ee_project_id_missing(self):
argv = [
Expand Down Expand Up @@ -845,6 +846,7 @@ def download(self):
self.assertEqual(1, exit_code)
self.assertIn("Earth Engine project ID missing.", output)
self.assertIn("GCP_PROJECT_ID", output)
self.assertIn("earth-engine-setup", output)

def test_source_data_main_accepts_project_id_option(self):
argv = [
Expand Down
13 changes: 13 additions & 0 deletions tests/test_xee_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,19 @@ def test_format_ee_setup_error_simplifies_auth_refresh_network_failures(self):

self.assertIn("Earth Engine auth refresh failed.", message)
self.assertIn("ee.Authenticate()", message)
self.assertIn(xee_common.EARTH_ENGINE_SETUP_URL, message)

def test_format_ee_setup_error_adds_setup_guide_for_missing_project_id(self):
message = xee_common.format_ee_setup_error(
ValueError(
"Earth Engine project ID is required. Pass ee_project_id or set one of "
"GCP_PROJECT_ID, GOOGLE_CLOUD_PROJECT, or EE_PROJECT_ID."
)
)

self.assertIn("Earth Engine project ID missing.", message)
self.assertIn("GCP_PROJECT_ID", message)
self.assertIn(xee_common.EARTH_ENGINE_SETUP_URL, message)


if __name__ == "__main__":
Expand Down
Loading