Merge pull request #93 from nextstrain/phylo/ppx

Phylo/ppx

Author: rneher

README.md

@@ -20,6 +20,16 @@ RSV sequences and metadata can be downloaded in the `/ingest` folder using
 The ingest pipeline is based on the Nextstrain mpox ingest workflow (<https://github.com/nextstrain/mpox/tree/master/ingest>).
 Running the ingest pipeline produces `ingest/data/{a,b}/metadata.tsv` and `ingest/data/{a,b}/sequences.fasta`.
 
+### Use locally ingested data
+
+Once you have run the ingest pipeline locally you can copy the files into the top-level `data` directory so that the main phylo workflow uses these files rather than downloading from s3:
+
+```sh
+mkdir -p data/{a,b}
+for i in ingest/data/*/{metadata.tsv,sequences.fasta}; do cp $i ${i#ingest/}; done
+```
+
+
 ### `ingest/vendored`
 
 This repository uses [`git subrepo`](https://github.com/ingydotnet/git-subrepo) to manage copies of ingest scripts in [`ingest/vendored`](./ingest/vendored), from [nextstrain/ingest](https://github.com/nextstrain/ingest). To pull new changes from the central ingest repository, first install `git subrepo`, then run:

config/auspice_config.json

@@ -8,6 +8,10 @@
     {
       "name": "GenBank",
       "url": "https://www.ncbi.nlm.nih.gov/genbank/"
+    },
+    {
+      "name": "Pathoplexus",
+      "url": "https://pathoplexus.org"
     }
   ],
   "build_url": "https://github.com/nextstrain/rsv",
@@ -27,6 +31,11 @@
       "title": "Country",
       "type": "categorical"
     },
+    {
+      "key": "dataUseTerms",
+      "title": "Data use terms",
+      "type": "categorical"
+    },
     {
       "key": "region",
       "title": "Region",
@@ -83,6 +92,11 @@
       "type": "ordinal"
     }
   ],
+  "metadata_columns": [
+    "PPX_accession",
+    "INSDC_accession",
+    "restrictedUntil"
+  ],
   "geo_resolutions": [
     "country",
     "region"

config/auspice_config_additional_colorings.json

@@ -0,0 +1,9 @@
+{
+  "colorings": [
+    {
+      "key": "clade",
+      "title": "Clade",
+      "type": "categorical"
+    }
+  ]
+}

config/configfile.yaml

@@ -6,7 +6,7 @@ builds_to_run: ["genome", "G", "F"]
 
 resolutions_to_run: ["all-time", "6y", "3y"]
 
-exclude: "config/outliers.txt"
+exclude: "config/outliers_ppx.txt"
 
 description: "config/description.md"
 
@@ -43,6 +43,7 @@ filter:
 
 files:
   auspice_config: "config/auspice_config.json"
+  auspice_config_additional_colorings: "config/auspice_config_additional_colorings.json"
 
 refine:
   coalescent: "opt"

config/description.md

@@ -1,4 +1,4 @@
-We gratefully acknowledge the authors, originating and submitting laboratories of the genetic sequences and metadata for sharing their work. Please note that although data generators have generously shared data in an open fashion, that does not mean there should be free license to publish on this data. Data generators should be cited where possible and collaborations should be sought in some circumstances. Please try to avoid scooping someone else's work. Reach out if uncertain.
+We gratefully acknowledge the authors, originating and submitting laboratories of the genetic sequences and metadata for sharing their work via INSDC or Pathoplexus. Please note that data from Pathoplexus comes with specific data use terms that need to be abided by. If data are shared under RESTRICTED terms, you can not use these data in publications without collaborating with the group that generated the data, please consult the [Data Use Terms of Pathoplexus](https://pathoplexus.org/about/terms-of-use/restricted-data) for details. Even if data are shared without restrictions, that does not mean there should be free license to publish on this data. Data generators should be cited where possible and collaborations should be sought in some circumstances. Please try to avoid scooping someone else's work. Reach out if uncertain.
 
 We maintain three views of human respiratory syncytial virus evolution for 3 time spans of each RSV subtype:
 
@@ -19,17 +19,18 @@ Our bioinformatic processing workflow can be found at [github.com/nextstrain/rsv
 - clade assignment via clade definitions defined here:
   [RSV-A](https://raw.githubusercontent.com/rsv-lineages/lineage-designation-A/main/.auto-generated/lineage.tsv)
   [RSV-B](https://raw.githubusercontent.com/rsv-lineages/lineage-designation-A/main/.auto-generated/lineage.tsv)
-  These clade definitions are based on the not-yet-published nomenclature of the RSV Genotyping Consensus Consortium.
+  These clade definitions are based on the [nomenclature proposal by the RSV Genotyping Consensus Consortium](https://wwwnc.cdc.gov/eid/article/30/8/24-0209_article).
 
 #### Underlying data
 
-We curate sequence data and metadata from the [NCBI Datasets command line tools](https://www.ncbi.nlm.nih.gov/datasets/docs/v2/download-and-install/)
-as starting point for these analyses. See our [ingest configuration file](https://github.com/nextstrain/rsv/blob/master/ingest/config/config.yaml)
-for the NCBI Taxonomy IDs used to fetch the virus genomes.
-Curated sequences and metadata are available as flat files at:
+We source sequence data and metadata from [Pathoplexus](https://pathoplexus.org) which ingests data from INSDC and provides data from INSDC together with data that were submitted directly to Pathoplexus. See our [ingest configuration file](https://github.com/nextstrain/rsv/blob/master/ingest/config/config.yaml).
+Curated sequences and metadata are available as flat files at the links below.
+The data in the files provided below is the subset of data from Pathoplexus under the OPEN [data use terms](https://pathoplexus.org/about/terms-of-use/data-use-terms). In the metadata files below, each sequence contains a field specifying the data use terms of this sequence and a link to the data use terms.
 
 - [RSV-A sequences](https://data.nextstrain.org/files/workflows/rsv/a/sequences.fasta.xz)
 - [RSV-A metadata](https://data.nextstrain.org/files/workflows/rsv/a/metadata.tsv.gz)
 
 - [RSV-B sequences](https://data.nextstrain.org/files/workflows/rsv/b/sequences.fasta.xz)
 - [RSV-B metadata](https://data.nextstrain.org/files/workflows/rsv/b/metadata.tsv.gz)
+
+ If you are interested in the RESTRICTED USE data, we ask you to obtain those directly from Pathoplexus.

config/include_a.txt

@@ -0,0 +1,4 @@
+# A.D.3.7
+LC862029
+OZ279430
+OZ279094

config/include_b.txt

@@ -0,0 +1,16 @@
+# A
+
+PP_001LE8W  # over diverged/misdated
+PP_001NQY1  # over diverged/misdated
+PP_001CJNE
+PP_0019Z0G
+PP_001TDF3
+
+
+# B
+PP_00226Q7  # over diverged/misdated
+PP_001B1YD
+PP_001B1ZB
+PP_0026MLS
+PP_00232B2
+PP_002S0BR

config/outliers_ppx.txt

@@ -8,7 +8,11 @@ def _get_all_targets(wildcards):
         "data/a/sequences.fasta",
         "data/a/metadata.tsv",
         "data/b/sequences.fasta",
-        "data/b/metadata.tsv"
+        "data/b/metadata.tsv",
+        "data/a/sequences_open.fasta",
+        "data/a/metadata_open.tsv",
+        "data/b/sequences_open.fasta",
+        "data/b/metadata_open.tsv"
     ]
 
     # Add additional targets based on upload config

ingest/Snakefile

@@ -0,0 +1,43 @@
+"""custom curate script to add URLs"""
+import sys
+import argparse
+from typing import Iterable
+
+from augur.curate import validate_records
+from augur.io.json import dump_ndjson, load_ndjson
+
+def run(args: argparse.Namespace, records: Iterable[dict]) -> Iterable[dict]:
+
+    for index, record in enumerate(records):
+        record = record.copy()
+
+        ppx_accession = record.get('PPX_accession', None) # versioned
+        insdc_accession = record.get('INSDC_accession', None) # versioned
+
+        # Add INSDC_accession__url and PPX_accession__url fields to NDJSON records
+        record['PPX_accession__url'] = f"https://pathoplexus.org/seq/{ppx_accession}" \
+            if ppx_accession \
+            else ""
+        record['INSDC_accession__url'] = f"https://www.ncbi.nlm.nih.gov/nuccore/{insdc_accession}" \
+            if insdc_accession \
+            else ""
+
+        yield record
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description=__doc__)
+    args = parser.parse_args()
+
+    records = load_ndjson(sys.stdin)
+
+    # Validate records have the same input fields
+    validated_input_records = validate_records(records, __doc__, True)
+
+    # Run this custom curate command to get modified records
+    modified_records = run(args, validated_input_records)
+
+    # Validate modified records have the same output fields
+    validated_output_records = validate_records(modified_records, __doc__, False)
+
+    dump_ndjson(validated_output_records)