Merge pull request #819 from ypriverol/feat/move-annotated-projects-repo

Document annotated datasets move to sdrf-annotated-datasets

Author: ypriverol

.github/workflows/build-docs.yml

@@ -4,7 +4,6 @@ on:
   push:
     branches:
       - master
-      - dev
   workflow_dispatch:
 
 jobs:
@@ -31,46 +30,10 @@ jobs:
       - name: Install Python dependencies
         run: pip install pyyaml jinja2
 
-      - name: Set output directory based on branch
-        id: set-output-dir
-        run: |
-          if [ "${{ github.ref_name }}" == "master" ]; then
-            echo "output_dir=docs" >> $GITHUB_OUTPUT
-            echo "base_url=" >> $GITHUB_OUTPUT
-          else
-            echo "output_dir=docs/dev" >> $GITHUB_OUTPUT
-            echo "base_url=/dev" >> $GITHUB_OUTPUT
-          fi
-
       - name: Build documentation site
         run: |
           chmod +x scripts/build-docs.sh
-          if [ "${{ github.ref_name }}" == "dev" ]; then
-            ./scripts/build-docs.sh ${{ steps.set-output-dir.outputs.output_dir }} --dev
-          else
-            ./scripts/build-docs.sh ${{ steps.set-output-dir.outputs.output_dir }}
-          fi
-
-      - name: Checkout gh-pages branch for merging
-        if: github.ref_name == 'dev'
-        uses: actions/checkout@v4
-        with:
-          ref: gh-pages
-          path: gh-pages-existing
-
-      - name: Merge dev docs into existing gh-pages
-        if: github.ref_name == 'dev'
-        run: |
-          # Copy existing gh-pages content (master version)
-          mkdir -p final-docs
-          cp -r gh-pages-existing/* final-docs/ 2>/dev/null || true
-          # Remove old dev folder if exists
-          rm -rf final-docs/dev
-          # Copy new dev docs
-          cp -r docs/dev final-docs/dev
-          # Move final-docs to docs for deployment
-          rm -rf docs
-          mv final-docs docs
+          ./scripts/build-docs.sh docs
 
       - name: Add CNAME for custom domain
         run: echo 'sdrf.quantms.org' > docs/CNAME

.github/workflows/link-check.yml

@@ -2,7 +2,7 @@ name: Check Links
 
 on:
   push:
-    branches: [master, dev]
+    branches: [master]
     paths:
       - '**/*.md'
       - '**/*.adoc'

.github/workflows/validate-all.yml

@@ -1,24 +1,25 @@
-# This workflow will install Python dependencies and run the included validation script.
+# Validate the small set of SDRF examples kept in examples/ for quickstart and
+# specification illustration purposes. Annotated public datasets now live at
+# bigbio/sdrf-annotated-datasets and are validated by that repository's CI.
 
-name: Validate annotations
+name: Validate examples
 
 on:
   push:
     branches: [ master, dev ]
     paths:
-      - 'annotated-projects/**'
+      - 'examples/**'
   pull_request:
     branches: [ master, dev ]
     paths:
-      - 'annotated-projects/**'
+      - 'examples/**'
+  workflow_dispatch:
 
 jobs:
   sdrf_proteomics_validations:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
       - name: Set up Python 3.10
         uses: actions/setup-python@v5
         with:
@@ -28,30 +29,15 @@ jobs:
           python -m pip install --upgrade pip
           pip install wheel
           pip install git+https://github.com/bigbio/sdrf-pipelines
-      - name: Get a list of changed files
-        if: github.event_name != 'schedule'
-        id: getfiles
+      - name: Validate examples
         run: |
-          export OUTFILE=$HOME/filelist.txt
-          if [ "${{ github.event_name }}" == pull_request ]; then
-              echo 'Changed files:'
-              git diff-tree --no-commit-id --name-only -r origin/${{ github.event.pull_request.base.ref }} ${{ github.sha }} | tee "$OUTFILE"
-          else
-            # push
-            if [ "${{ github.event.before }}" == "0000000000000000000000000000000000000000" ]; then
-              parent="origin/master" # compare with master as a last resort
-            else
-              parent="${{ github.event.before }}"
+          shopt -s nullglob
+          failed=0
+          for f in examples/*/*.sdrf.tsv; do
+            echo "Validating $f"
+            if ! parse_sdrf validate-sdrf --sdrf_file "$f" --use_ols_cache_only; then
+              echo "::error file=$f::parse_sdrf validate-sdrf failed"
+              failed=1
             fi
-            git diff-tree --no-commit-id --name-only -r "$parent" ${{ github.sha }} | tee "$OUTFILE"
-          fi
-      - name: Run validation
-        run: |
-          if [ ${{ github.event_name }} == "schedule" ]; then
-            echo "Validating all projects (schedule)"
-            python validate.py -v
-          else
-            export FILELIST=$HOME/filelist.txt
-            cat "$FILELIST" | grep "*sdrf.tsv" | while read line; do echo "Changed file: $line"; parse_sdrf validate-sdrf --sdrf_file $line --skip_factor_validation --skip_experimental_design_validation --use_ols_cache_only; done
-          fi
-
+          done
+          exit "$failed"

DEVELOPMENT.md

@@ -36,7 +36,7 @@ proteomics-metadata-standard/
 │   ├── sdrf-terms.html        # Terms reference page
 │   ├── build-sdrf-index.py    # Builds dataset index
 │   └── build-search-index.py  # Builds search index
-├── annotated-projects/        # Annotated SDRF files (~300 datasets)
+├── annotated-projects/        # Migration notice and legacy compatibility pointers
 ├── demo_page/                 # Local preview output (git-ignored)
 ├── .github/workflows/         # CI/CD configuration
 │   └── build-docs.yml         # Website build workflow
@@ -85,7 +85,7 @@ The easiest way to build the documentation is using the provided build script. T
 1. **Converts AsciiDoc to HTML** using Asciidoctor with proper styling options
 2. **Copies static assets** (CSS, JavaScript, images)
 3. **Copies static HTML pages** (index.html, quickstart.html, sdrf-explorer.html, sdrf-editor.html, sdrf-terms.html)
-4. **Builds SDRF Explorer index** from annotated-projects
+4. **Builds SDRF Explorer index** using annotated project metadata and links
 5. **Injects navigation headers** into all generated HTML pages
 6. **Transforms SDRF links** to use the SDRF Explorer viewer
 7. **Transforms SDRF example tables** to add color-coded column styling
@@ -182,9 +182,8 @@ Example HTML in AsciiDoc:
 
 ## CI/CD Deployment
 
-The website is automatically built and deployed when pushing to:
-- `master` branch → Production site (https://sdrf.quantms.org/)
-- `dev` branch → Development site (https://sdrf.quantms.org/dev/)
+The website is automatically built and deployed when pushing to the `master`
+branch → https://sdrf.quantms.org/.
 
 The CI/CD workflow (`.github/workflows/build-docs.yml`) performs the same steps as the local build script, ensuring consistency.
 
@@ -196,8 +195,8 @@ The CI/CD workflow (`.github/workflows/build-docs.yml`) performs the same steps
 | Navigation headers | ✓ | ✓ |
 | SDRF link transformation | ✓ | ✓ |
 | Search index | ✓ | ✓ |
-| Dev banner | `--dev` flag | Auto for dev branch |
-| Output directory | Configurable | docs/ or docs/dev/ |
+| Dev banner | `--dev` flag (local only) | — |
+| Output directory | Configurable | `docs/` |
 
 ### Ensuring Consistency
 
@@ -213,14 +212,21 @@ To verify your local build matches the deployed version:
 
 ## Adding New Annotated Projects
 
-1. Create a folder in `annotated-projects/` with the PXD accession
-2. Add the SDRF file as `{PXD}.sdrf.tsv`
+Annotated datasets are now maintained in
+`bigbio/sdrf-annotated-datasets` using the layout
+`datasets/{ACCESSION}/{ACCESSION}.sdrf.tsv`.
+
+1. Create a folder in `datasets/` with the accession in
+   `bigbio/sdrf-annotated-datasets`
+2. Add the SDRF file as `{ACCESSION}.sdrf.tsv`
 3. Validate the file:
    ```bash
    pip install sdrf-pipelines
-   parse_sdrf validate-sdrf --sdrf_file annotated-projects/PXD000000/PXD000000.sdrf.tsv
+   parse_sdrf validate-sdrf --sdrf_file datasets/PXD000000/PXD000000.sdrf.tsv
    ```
-4. Rebuild the dataset index:
+4. Open a pull request in `bigbio/sdrf-annotated-datasets`
+5. If this repository needs a refreshed explorer index after migration updates,
+   rebuild it locally:
    ```bash
    python3 site/build-sdrf-index.py
    ```
@@ -243,16 +249,16 @@ This is supported by research <<ref1>>.
 ## Useful Commands
 
 ```bash
-# Validate all SDRF files in a folder
-for f in annotated-projects/*/*.sdrf.tsv; do
+# Validate all SDRF files in the annotated datasets repository
+for f in datasets/*/*.sdrf.tsv; do
   parse_sdrf validate-sdrf --sdrf_file "$f"
 done
 
 # Find all SDRF files with a specific column
-grep -l "characteristics\[disease\]" annotated-projects/*/*.sdrf.tsv
+grep -l "characteristics\[disease\]" datasets/*/*.sdrf.tsv
 
 # Count annotated projects
-ls -d annotated-projects/PXD* | wc -l
+ls -d datasets/PXD* | wc -l
 ```
 
 ## Troubleshooting
@@ -387,6 +393,5 @@ When templates change (new columns, new templates):
 1. Make changes to the appropriate AsciiDoc or HTML files
 2. Build locally and test: `./scripts/build-docs.sh --clean`
 3. View locally: `open demo_page/index.html`
-4. Create a pull request to the `dev` branch for review
-5. After approval and merge to dev, changes deploy to https://sdrf.quantms.org/dev/
-6. After merge to master, changes deploy to https://sdrf.quantms.org/
+4. Open a pull request against `master`
+5. After approval and merge, changes deploy to https://sdrf.quantms.org/

README.md

@@ -40,7 +40,13 @@ For information on building the documentation website locally, see the [Developm
 
 Contribute with the specification: you can contribute to the specification with ideas or refinements by adding an issue into the [issue tracker](https://github.com/bigbio/proteomics-metadata-standard/issues) or performing a PR.
 
-In the [annotated projects](https://github.com/bigbio/proteomics-metadata-standard/tree/master/annotated-projects) folder, users can see different public datasets that have been annotated so far by the contributors. If you would like to join these efforts, make a Fork of this repo and perform a pull request (PR) with your annotated project. If you don't have a project in mind, you can take one project from the [issues](https://github.com/bigbio/proteomics-metadata-standard/issues) and perform the annotation.
+Annotated SDRF dataset files are now maintained in the dedicated repository
+[`bigbio/sdrf-annotated-datasets`](https://github.com/bigbio/sdrf-annotated-datasets)
+under `datasets/{ACCESSION}/{ACCESSION}.sdrf.tsv`.
+If you would like to contribute annotations, open a pull request in that
+repository. If you do not have a project in mind, you can still pick one from
+the [issues](https://github.com/bigbio/proteomics-metadata-standard/issues) and
+prepare the SDRF annotation.
 
 Annotate a dataset in 5 steps:
 
@@ -63,7 +69,8 @@ parse_sdrf validate-sdrf --sdrf_file PXD020294.sdrf.tsv
 
 You can read more about the validator [here](https://github.com/bigbio/sdrf-pipelines).
 
-- Fork the current repository, add a folder with the ProteomeXchange accession and the annotated sdrf.tsv
+- Fork `bigbio/sdrf-annotated-datasets`, add a folder under `datasets/` with
+  the ProteomeXchange accession and the annotated `.sdrf.tsv` file.
 
 ## 30 Minutes Guide to SDRF-Proteomics
 

annotated-projects/README.md

@@ -0,0 +1,32 @@
+# Annotated projects migration notice
+
+The annotated SDRF datasets were moved to a dedicated repository:
+
+- https://github.com/bigbio/sdrf-annotated-datasets
+
+## Why this was moved
+
+The SDRF specification and annotated datasets now evolve on separate lifecycles:
+
+- `bigbio/proteomics-sample-metadata` focuses on specification, docs, and tooling.
+- `bigbio/sdrf-annotated-datasets` focuses on curated dataset annotations that can be updated frequently.
+
+This split was discussed in:
+- https://github.com/bigbio/proteomics-sample-metadata/issues/817
+
+## New canonical location
+
+Annotated files are now expected in:
+
+`datasets/{ACCESSION}/{ACCESSION}.sdrf.tsv`
+
+Example:
+
+`datasets/PXD000070/PXD000070.sdrf.tsv`
+
+## How to contribute annotated datasets
+
+1. Fork `bigbio/sdrf-annotated-datasets`.
+2. Add or update files in `datasets/{ACCESSION}/`.
+3. Validate with `parse_sdrf validate-sdrf`.
+4. Open a pull request in `bigbio/sdrf-annotated-datasets`.

psi-document/v1.1.0-dev/supplementary-files/quickstart-v1.1.0-dev.adoc

@@ -540,7 +540,7 @@ TIP: Don't worry about finding the exact ontology term initially. Just write the
 
 Browse real SDRF files from published datasets in ProteomeXchange:
 
-- https://github.com/bigbio/proteomics-metadata-standard/tree/master/annotated-projects[Annotated Projects on GitHub]
+- https://github.com/bigbio/sdrf-annotated-datasets/tree/main/datasets[Annotated Projects on GitHub]
 
 === Questions
 

psi-document/v1.1.0-dev/supplementary-files/tool-support-v1.1.0-dev.adoc

@@ -267,4 +267,4 @@ We welcome contributions from the community to expand the SDRF-Proteomics ecosys
 
 * link:README.adoc[SDRF-Proteomics Specification]
 * https://github.com/bigbio/proteomics-metadata-standard[GitHub Repository]
-* https://github.com/bigbio/proteomics-metadata-standard/tree/master/annotated-projects[Annotated Project Examples]
+* https://github.com/bigbio/sdrf-annotated-datasets/tree/main/datasets[Annotated Project Examples]