Skip to content

Commit 719e145

Browse files
wietzesuijkerwietzesuijker
committed
docs: credential setup instructions with OVH Manager links
- workflows/README: explain secret purposes (event ingestion, storage, API auth) - workflows/README: add direct OVH Manager links for kubeconfig and S3 credentials - README: delegate setup to workflows/README - Separate operator usage (root README) from deployment setup (workflows/README)
1 parent b226d75 commit 719e145

File tree

2 files changed

+118
-158
lines changed

2 files changed

+118
-158
lines changed

‎README.md‎

Lines changed: 17 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# EOPF GeoZarr Data Pipeline
22

3-
**Kubernetes pipeline: Sentinel CPM Zarr → Cloud-Optimized GeoZarr + STAC Registration**
3+
**Kubernetes pipeline: Sentinel Zarr → Cloud-Optimized GeoZarr + STAC Registration**
44

55
Automated pipeline for converting Sentinel-1/2 Zarr datasets to cloud-optimized GeoZarr format with STAC catalog integration and interactive visualization.
66

@@ -56,33 +56,21 @@ Transforms Sentinel-1/2 satellite data into web-ready visualizations:
5656
- Sentinel-1 GRD (SAR backscatter)
5757

5858

59-
## Requirements & Setup
59+
## Setup
6060

61-
### Prerequisites
61+
**Prerequisites:**
62+
- Kubernetes cluster with [platform-deploy](https://github.com/EOPF-Explorer/platform-deploy) (Argo Workflows, RabbitMQ, STAC API, TiTiler)
63+
- Python 3.13+ with `uv`
64+
- `kubectl` configured
6265

63-
- **Kubernetes cluster** with [platform-deploy](https://github.com/EOPF-Explorer/platform-deploy) infrastructure
64-
- Argo Workflows (pipeline orchestration)
65-
- RabbitMQ (event-driven automation)
66-
- STAC API & TiTiler (catalog & visualization)
67-
- **Python 3.13+** with `uv` package manager
68-
- **S3 storage** credentials (OVH de region)
69-
- **Kubeconfig** in `.work/kubeconfig`
66+
**📖 Complete setup guide:** See [workflows/README.md](workflows/README.md) for:
67+
- kubectl configuration (OVH Manager kubeconfig download)
68+
- Required secrets (RabbitMQ, S3, STAC API)
69+
- Workflow deployment (`kubectl apply -k`)
7070

71-
Verify infrastructure:
71+
**Quick verification:**
7272
```bash
73-
export KUBECONFIG=$(pwd)/.work/kubeconfig
74-
kubectl get pods -n core -l app.kubernetes.io/name=argo-workflows
75-
kubectl get pods -n core -l app.kubernetes.io/name=rabbitmq
76-
```
77-
78-
### Deploy Workflows
79-
80-
```bash
81-
# Apply to staging
82-
kubectl apply -k workflows/overlays/staging
83-
84-
# Apply to production
85-
kubectl apply -k workflows/overlays/production
73+
kubectl get wf,sensor,eventsource -n devseed-staging
8674
```
8775

8876
---
@@ -214,40 +202,12 @@ Tests are planned for `tests/` directory (structure exists, test files to be add
214202

215203
## Configuration
216204

217-
### S3 Storage
218-
219-
```bash
220-
kubectl create secret generic geozarr-s3-credentials -n devseed-staging \
221-
--from-literal=AWS_ACCESS_KEY_ID="<your-key>" \
222-
--from-literal=AWS_SECRET_ACCESS_KEY="<your-secret>"
223-
```
224-
225-
| Setting | Value |
226-
|---------|-------|
227-
| **Endpoint** | `https://s3.de.io.cloud.ovh.net` |
228-
| **Bucket** | `esa-zarr-sentinel-explorer-fra` |
229-
| **Region** | `de` |
205+
**📖 Full configuration:** See [workflows/README.md](workflows/README.md) for secrets setup and parameters.
230206

231-
### RabbitMQ
232-
233-
Get password:
234-
```bash
235-
kubectl get secret rabbitmq-password -n core -o jsonpath='{.data.rabbitmq-password}' | base64 -d
236-
```
237-
238-
| Setting | Value |
239-
|---------|-------|
240-
| **URL** | `amqp://user:[email protected]:5672/` |
241-
| **Exchange** | `geozarr-staging` |
242-
| **Routing key** | `eopf.items.test` |
243-
244-
**Message format:**
245-
```json
246-
{
247-
"source_url": "https://stac.core.eopf.eodc.eu/collections/sentinel-2-l2a/items/...",
248-
"collection": "sentinel-2-l2a-dp-test"
249-
}
250-
```
207+
**Quick reference:**
208+
- S3: `s3.de.io.cloud.ovh.net` / `esa-zarr-sentinel-explorer-fra`
209+
- Staging collection: `sentinel-2-l2a-dp-test`
210+
- Production collection: `sentinel-2-l2a`
251211

252212
---
253213

‎workflows/README.md‎

Lines changed: 101 additions & 101 deletions
Original file line numberDiff line numberDiff line change
@@ -1,150 +1,150 @@
11
# Workflows
22

3-
Argo Workflows configuration using Kustomize for environment management.
3+
Event-driven Argo Workflows for Sentinel-2 GeoZarr conversion and STAC registration.
44

5-
## Purpose
5+
**Architecture**: RabbitMQ messages → Sensor → WorkflowTemplate (convert → register) → S3 + STAC API
66

7-
Event-driven pipeline orchestration for Sentinel-2 GeoZarr conversion and STAC registration. RabbitMQ messages trigger workflows that run a 2-step DAG: **convert → register**.
7+
---
88

9-
## Structure
9+
## Quick Setup
10+
11+
### 1. Configure kubectl
12+
13+
Download kubeconfig from [OVH Manager → Kubernetes](https://www.ovh.com/manager/#/public-cloud/pci/projects/bcc5927763514f499be7dff5af781d57/kubernetes/f5f25708-bd15-45b9-864e-602a769a5fcf/service) (**Access and Security** tab).
1014

15+
```bash
16+
mv ~/Downloads/kubeconfig-*.yml .work/kubeconfig
17+
export KUBECONFIG=$(pwd)/.work/kubeconfig
18+
kubectl get nodes # Verify: should list 3-5 nodes
1119
```
12-
workflows/
13-
├── base/ # Core resources (namespace-agnostic)
14-
│ ├── kustomization.yaml # References all resources
15-
│ ├── workflowtemplate.yaml # 2-step pipeline DAG
16-
│ ├── sensor.yaml # RabbitMQ → Workflow trigger
17-
│ ├── eventsource.yaml # RabbitMQ connection config
18-
│ └── rbac.yaml # ServiceAccount + permissions
19-
└── overlays/
20-
├── staging/
21-
│ └── kustomization.yaml # devseed-staging namespace patches
22-
└── production/
23-
└── kustomization.yaml # devseed namespace patches
20+
21+
### 2. Create Required Secrets
22+
23+
The pipeline needs 3 secrets for: **event ingestion** (RabbitMQ), **output storage** (S3), and **STAC registration** (API auth).
24+
25+
**RabbitMQ credentials** (receives workflow trigger events):
26+
```bash
27+
# Get password from cluster-managed secret
28+
RABBITMQ_PASS=$(kubectl get secret rabbitmq-password -n core -o jsonpath='{.data.rabbitmq-password}' | base64 -d)
29+
30+
kubectl create secret generic rabbitmq-credentials -n devseed-staging \
31+
--from-literal=username=user \
32+
--from-literal=password="$RABBITMQ_PASS"
2433
```
2534

26-
## Apply to Cluster
35+
**S3 credentials** (writes converted GeoZarr files):
36+
```bash
37+
# Get from OVH Manager → Users & Roles → OpenStack credentials
38+
# https://www.ovh.com/manager/\#/public-cloud/pci/projects/bcc5927763514f499be7dff5af781d57/users
39+
40+
kubectl create secret generic geozarr-s3-credentials -n devseed-staging \
41+
--from-literal=AWS_ACCESS_KEY_ID=<your-ovh-access-key> \
42+
--from-literal=AWS_SECRET_ACCESS_KEY=<your-ovh-secret-key>
43+
```
2744

28-
**Staging (devseed-staging):**
45+
**STAC API token** (registers items, optional if API is public):
2946
```bash
30-
kubectl apply -k workflows/overlays/staging
47+
kubectl create secret generic stac-api-token -n devseed-staging \
48+
--from-literal=token=<bearer-token>
3149
```
3250

33-
**Production (devseed):**
51+
### 3. Deploy Workflows
52+
3453
```bash
35-
kubectl apply -k workflows/overlays/production
54+
kubectl apply -k workflows/overlays/staging # Staging (devseed-staging)
55+
kubectl apply -k workflows/overlays/production # Production (devseed)
3656
```
3757

3858
**Verify deployment:**
3959
```bash
40-
# Check resources (expected output shows 1 of each)
4160
kubectl get workflowtemplate,sensor,eventsource,sa -n devseed-staging
61+
# Expected: 1 WorkflowTemplate, 1 Sensor, 1 EventSource, 1 ServiceAccount
62+
```
63+
64+
---
65+
66+
## Structure
67+
68+
```
69+
workflows/
70+
├── base/ # Core resources (namespace-agnostic)
71+
│ ├── workflowtemplate.yaml # 2-step DAG: convert → register
72+
│ ├── sensor.yaml # RabbitMQ trigger
73+
│ ├── eventsource.yaml # RabbitMQ connection
74+
│ ├── rbac.yaml # Permissions
75+
│ └── kustomization.yaml
76+
└── overlays/
77+
├── staging/ # devseed-staging namespace
78+
└── production/ # devseed namespace
79+
```
4280

43-
# Example output:
44-
# NAME AGE
45-
# workflowtemplate.argoproj.io/geozarr-pipeline 5m
46-
#
47-
# NAME AGE
48-
# sensor.argoproj.io/geozarr-sensor 5m
49-
#
50-
# NAME AGE
51-
# eventsource.argoproj.io/rabbitmq-geozarr 5m
52-
#
53-
# NAME SECRETS AGE
54-
# serviceaccount/operate-workflow-sa 0 5m
55-
56-
# Watch for workflows (should show Running/Succeeded/Failed)
81+
---
82+
83+
## Monitoring
84+
85+
**Watch workflows:**
86+
```bash
5787
kubectl get wf -n devseed-staging --watch
5888
```
59-
Example outputs:
89+
90+
**Example output:**
6091
```
6192
NAME STATUS AGE
6293
geozarr-79jmg Running 5m
6394
geozarr-95rgx Succeeded 9h
64-
geozarr-hpcvf Succeeded 10h
6595
geozarr-jflnj Failed 10h
6696
```
6797

68-
## Required Secrets
69-
70-
The pipeline requires these Kubernetes secrets in the target namespace:
71-
72-
### 1. `rabbitmq-credentials`
73-
RabbitMQ authentication for EventSource:
98+
---
7499

75-
```bash
76-
kubectl create secret generic rabbitmq-credentials \
77-
--from-literal=username=<rabbitmq-user> \
78-
--from-literal=password=<rabbitmq-password> \
79-
-n devseed-staging
80-
```
100+
## Configuration
81101

82-
### 2. `geozarr-s3-credentials`
83-
S3 credentials for GeoZarr output:
102+
### S3 Storage
84103

85-
```bash
86-
kubectl create secret generic geozarr-s3-credentials \
87-
--from-literal=AWS_ACCESS_KEY_ID=<access-key> \
88-
--from-literal=AWS_SECRET_ACCESS_KEY=<secret-key> \
89-
-n devseed-staging
90-
```
104+
- **Endpoint**: `https://s3.de.io.cloud.ovh.net` (OVH Frankfurt)
105+
- **Bucket**: `esa-zarr-sentinel-explorer-fra`
106+
- **Paths**: `tests-output/` (staging), `geozarr/` (production)
91107

92-
### 3. `stac-api-token` (optional)
93-
Bearer token for STAC API authentication (if required):
108+
### Workflow Parameters
94109

95-
```bash
96-
kubectl create secret generic stac-api-token \
97-
--from-literal=token=<bearer-token> \
98-
-n devseed-staging
99-
```
110+
Key parameters (see [../README.md](../README.md) for full reference):
100111

101-
## WorkflowTemplate Parameters
112+
- `source_url`: STAC item URL or Zarr URL
113+
- `register_collection`: Target STAC collection (default: `sentinel-2-l2a-dp-test`)
114+
- `s3_output_bucket`: Output bucket
115+
- `pipeline_image_version`: Docker image tag
102116

103-
See main [README.md](../README.md) for complete parameter reference.
117+
### Resource Tuning
104118

105-
| Parameter | Default | Description |
106-
|-----------|---------|-------------|
107-
| `source_url` | - | STAC item URL or direct Zarr URL |
108-
| `register_collection` | sentinel-2-l2a-dp-test | STAC collection ID |
109-
| `stac_api_url` | https://api... | STAC API endpoint |
110-
| `raster_api_url` | https://api... | TiTiler endpoint |
111-
| `s3_output_bucket` | esa-zarr... | S3 output bucket |
112-
| `pipeline_image_version` | fix-unit-tests | Docker image tag |
113-
114-
## Resource Configuration
115-
116-
To adjust CPU/memory limits, edit `workflows/base/workflowtemplate.yaml`:
119+
Edit `workflows/base/workflowtemplate.yaml`:
117120

118121
```yaml
119-
- name: convert-geozarr
120-
resources:
121-
requests:
122-
memory: 4Gi # Increase for larger datasets
123-
cpu: '1'
124-
limits:
125-
memory: 8Gi
126-
cpu: '2'
122+
resources:
123+
requests: { memory: 4Gi, cpu: '1' }
124+
limits: { memory: 8Gi, cpu: '2' } # Increase for larger datasets
127125
```
128126
127+
---
128+
129129
## Troubleshooting
130130
131-
**Kustomize build fails:**
131+
**Workflow not triggered:**
132132
```bash
133-
# Validate structure
134-
kubectl kustomize workflows/overlays/staging
133+
kubectl logs -n devseed-staging -l eventsource-name=rabbitmq # Check RabbitMQ connection
134+
kubectl get sensor -n devseed-staging geozarr-trigger -o yaml # Check sensor status
135+
```
135136

136-
# Check for duplicate resources
137-
find workflows -name "*.yaml" -not -path "*/base/*" -not -path "*/overlays/*"
137+
**Workflow fails:**
138+
```bash
139+
kubectl logs -n devseed-staging <workflow-pod-name> # View logs
140+
kubectl get secret -n devseed-staging # Verify secrets exist
138141
```
139142

140-
**Workflow not triggered:**
141-
- Check EventSource connection: `kubectl logs -n devseed-staging -l eventsource-name=rabbitmq`
142-
- Check Sensor status: `kubectl get sensor -n devseed-staging geozarr-trigger -o yaml`
143-
- Verify RabbitMQ port-forward or service access
143+
**Kustomize validation:**
144+
```bash
145+
kubectl kustomize workflows/overlays/staging # Validate YAML
146+
```
144147

145-
**Workflow fails:**
146-
- Check pod logs: `kubectl logs -n devseed-staging <workflow-pod-name>`
147-
- Verify secrets exist: `kubectl get secret -n devseed-staging geozarr-s3-credentials stac-api-token`
148-
- Check RBAC: `kubectl auth can-i create workflows --as=system:serviceaccount:devseed-staging:operate-workflow-sa`
148+
---
149149

150-
For full pipeline documentation, see [../README.md](../README.md).
150+
For complete documentation, see [../README.md](../README.md).

0 commit comments

Comments
 (0)