Merge branch 'release/0.1.2'

Author: rhshah

.pyup.yml

@@ -6,3 +6,4 @@ Branch pre-fix: pyup-
 Close PRs: True
 Label PRs: False
 Update hashes: True
+schedule: "every week"

bam_collapsing

@@ -6,7 +6,6 @@ description: Workflow that creates all the bam files for the MSK-ACCESS fastq fi
 
 [![PyPI version](https://badge.fury.io/py/fastq_to_bam.svg)](https://badge.fury.io/py/fastq_to_bam) [![Build Status](https://travis-ci.com/msk-access/fastq_to_bam.svg?branch=master)](https://travis-ci.com/msk-access/fastq_to_bam) [![Updates](https://pyup.io/repos/github/msk-access/fastq_to_bam/shield.svg)](https://pyup.io/repos/github/msk-access/fastq_to_bam/) [![Python 3](https://pyup.io/repos/github/msk-access/fastq_to_bam/python-3-shield.svg)](https://pyup.io/repos/github/msk-access/fastq_to_bam/)
 
-
 * Free software: Apache Software License 2.0
 * Documentation: [https://msk-access.gitbook.io/fastq-to-bam](https://msk-access.gitbook.io/fastq-to-bam)
 

command_line_tools

@@ -22,8 +22,8 @@
 | **P** | Skip pairing for BWA; mate rescue performed unless -S also in use | true |
 | **M** | Mark shorter split hits as secondary \(for Picard/GATK compatibility\) | true |
 | **reference** | Reference fasta file |  |
-| **read\_group\_identifier** | Read Group ID  Default value: 1. This option can be set to 'null' to clear the default value  Required |  |
-| **read\_group\_sequencing\_center** | Read Group sequencing center name  Default value: null. Required. | MSKCC |
+| **read\_group\_identifier** | Read Group ID.  Required |  |
+| **read\_group\_sequencing\_center** | Read Group sequencing center name. Required. | MSKCC |
 | **read\_group\_library** | Read Group Library.  Required |  |
 | **read\_group\_platform\_unit** | Read Group platform unit \(eg. run barcode\)  Required. |  |
 | **read\_group\_sequencing\_platform** | Read Group platform \(e.g. illumina, solid\)  Required. | ILLUMINA |
@@ -60,11 +60,21 @@
     <tr>
       <td style="text-align:left"><b>bed_file</b>
       </td>
-      <td style="text-align:left">The genotype from positions in this bed file will be used as the consensus
-        base if <b>min_consensus_percent</b> threshold is not reached. Otherwise,
-        the reference base from the supplied <b>reference_fasta</b> will be used</td>
-      <td
-      style="text-align:left"></td>
+      <td style="text-align:left">
+        <p>Bed file of target regions in tsv format with the following columns:</p>
+        <ol>
+          <li>contig / interval name</li>
+          <li>interval start index</li>
+          <li>interval end index</li>
+          <li>+ / - to indicate strand</li>
+          <li>interval desciption</li>
+        </ol>
+        <p>The genotype from positions in this bed file will be used as the consensus
+          base if the marianas <b>min_consensus_percent</b> threshold is not reached.
+          Otherwise, the reference base from the supplied <b>reference_fasta</b> will
+          be used</p>
+      </td>
+      <td style="text-align:left"></td>
     </tr>
     <tr>
       <td style="text-align:left"><b>wobble</b>
@@ -76,15 +86,15 @@
     <tr>
       <td style="text-align:left"><b>reference_fasta</b>
       </td>
-      <td style="text-align:left">Currently, hg19</td>
+      <td style="text-align:left">Path to a reference fasta file to align to</td>
       <td style="text-align:left"></td>
     </tr>
     <tr>
       <td style="text-align:left"><b>mismatches</b>
       </td>
-      <td style="text-align:left">Allowable mismatch count in UMI bases for grouping UMI families</td>
-      <td
-      style="text-align:left">0</td>
+      <td style="text-align:left">Allowable number of mismatches between fragments / molecules in the same
+        UMI family</td>
+      <td style="text-align:left">0</td>
     </tr>
     <tr>
       <td style="text-align:left"><b>min_consensus_percent</b>
@@ -114,29 +124,26 @@
     <tr>
       <td style="text-align:left"><b>output_name_collapsed_gzip_R1</b>
       </td>
-      <td style="text-align:left">
-        <p></p>
-        <p>Filename for collapsed read 1 fastq</p>
-      </td>
+      <td style="text-align:left">Filename to give to collapsed read 1 fastq</td>
       <td style="text-align:left"></td>
     </tr>
     <tr>
       <td style="text-align:left"><b>output_name_collapsed_gzip_R2</b>
       </td>
-      <td style="text-align:left">Filename for collapsed read 2 fastq</td>
+      <td style="text-align:left">Filename to give to collapsed read 2 fastq</td>
       <td style="text-align:left"></td>
     </tr>
     <tr>
       <td style="text-align:left"><b>picard_output_file_name</b>
       </td>
-      <td style="text-align:left">Filename to be given to final re-aligned bam (SHOULD have DEFAULT?)</td>
+      <td style="text-align:left">Filename to give to final re-aligned bam (SHOULD have DEFAULT?)</td>
       <td
       style="text-align:left"></td>
     </tr>
     <tr>
       <td style="text-align:left"><b>aln_output_file_name</b>
       </td>
-      <td style="text-align:left">Filename to be given to intermediate sam file (SHOULD have DEFAULT?)</td>
+      <td style="text-align:left">Filename to give to intermediate sam file (SHOULD have DEFAULT?)</td>
       <td
       style="text-align:left"></td>
     </tr>
@@ -171,7 +178,7 @@
     <tr>
       <td style="text-align:left"><b>bqsr_read_filter</b>
       </td>
-      <td style="text-align:left">GATK READ_FILTER option to apply defferent set of ReadFilter</td>
+      <td style="text-align:left">GATK READ_FILTER option to apply different set of ReadFilter</td>
       <td style="text-align:left"></td>
     </tr>
     <tr>
@@ -279,70 +286,43 @@
 
 ```text
 {
-    "M": true,
-    "P": true,
-    "adapter": null,
-    "adapter2": null,
-    "assume_sorted": null,
-    "bed_file": {
-        "class": "File",
-        "path": "panel.bed"
-    },
-    "bqsr_read_filter": null,
-    "collapsing_aln_output_file_name": null,
-    "collapsing_picard_output_file_name": null,
-    "consensus_sequence": null,
-    "contig_anchor": null,
-    "create_bam_index": null,
+    "standard_aln_output_file_name": "test_standard.sam",
+    "standard_picard_addrg_output_filename": "test_standard.bam",
+    "collapsing_aln_output_file_name": "test_unfiltered.sam",
+    "collapsing_picard_output_file_name": "test_unfiltered.bam",
+    "output_name_collapsed_gzip_R1": "test_collapsed_R1_fastq.gz",
+    "output_name_collapsed_gzip_R2": "test_collapsed_R2_fastq.gz",
+    "sort_first_pass_output_file_name": "test_collapsing_first_pass.txt",
+    
+    "read_group_identifier": "test",
+    "read_group_library": 1,
+    "read_group_platform_unit": "IDT11",
+    "read_group_sample_name": "test",
+    
     "fastq1": {
         "class": "File",
-        "path": "TD-seracare-0-5per_IGO_05500_EZ_32_S25_R1_001.fastq.gz"
+        "path": "/path/to/test_R1_001.fastq.gz",
     },
     "fastq2": {
         "class": "File",
-        "path": "TD-seracare-0-5per_IGO_05500_EZ_32_S25_R2_001.fastq.gz"
+        "path": "/path/to/test_R2_001.fastq.gz",
+    },
+    "bed_file": {
+        "class": "File",
+        "path": "/path/to/panel.bed",
     },
-    "ignore_bad_assembly": null,
-    "key": null,
     "known_sites_1": {
         "class": "File",
-        "path": "dbsnp_137.b37.vcf"
+        "path": "/path/to/dbsnp_137_14_16.b37.vcf",
+    },
+    "known_sites_2": {
+        "class": "File",
+        "path": "/path/to/Mills_and_1000G_gold_standard-14_16.indels.b37.vcf",
     },
-    "known_sites_2": null,
-    "length": null,
-    "maximum_average_depth": null,
-    "maximum_mixmatch_rate": null,
-    "min_base_quality": null,
-    "min_consensus_percent": null,
-    "min_map_quality": null,
-    "mismatches": null,
-    "number_of_threads": null,
-    "option_bedgraph": null,
-    "output_name_collapsed_gzip_R1": null,
-    "output_name_collapsed_gzip_R2": null,
-    "quality": null,
-    "read_group_identifier": "test_sample",
-    "read_group_library": 0,
-    "read_group_platform_unit": "PU",
-    "read_group_sample_name": "test_sample",
-    "read_group_sequencing_platform": Illumina,
-    "read_group_sequnecing_center": MSKCC_test,
     "reference": {
         "class": "File",
-        "path": "Homo_sapiens_assembly19.fasta"
+        "path": "/path/to/Homo_sapiens_assembly19.fasta",
     },
-    "scoring_gap_alignments": null,
-    "soft_clip_contig": null,
-    "sort_first_pass_output_file_name": "test",
-    "sort_order": null,
-    "standard_aln_output_file_name": null,
-    "standard_picard_addrg_output_filename": null,
-    "stringency": null,
-    "trim_galore_number_of_threads": null,
-    "validation_stringency": null,
-    "window_size": null,
-    "wobble": null
 }
-
 ```
 

docs/README.md

@@ -14,78 +14,70 @@ If you are using cwltool only, please proceed using python 3.7 as done below:
 
 Here we can use either [virtualenv](https://virtualenv.pypa.io/) or [conda](https://docs.conda.io/en/latest/). Here we will use virtualenv.
 
-{% code-tabs %}
-{% code-tabs-item title="python3-virtualenv" %}
+{% code title="python3-virtualenv" %}
 ```bash
 pip3 install virtualenv
 python3 -m venv my_project
 source my_project/bin/activate
 ```
-{% endcode-tabs-item %}
-{% endcode-tabs %}
+{% endcode %}
 
 ### Option \(B\) - recommended for Juno HPC cluster
 
 If you are using toil, python 2 is required. Please install using Python 2.7 as done below:
 
 Here we can use either [virtualenv](https://virtualenv.pypa.io/) or [conda](https://docs.conda.io/en/latest/). Here we will use virtualenv.
 
-{% code-tabs %}
-{% code-tabs-item title="python2-virtaulenv" %}
+{% code title="python2-virtaulenv" %}
 ```bash
 pip install virtualenv
 virtualenv my_project
 source my_project/bin/activate
 ```
-{% endcode-tabs-item %}
-{% endcode-tabs %}
+{% endcode %}
 
 {% hint style="info" %}
 Once you execute the above command you will see your bash prompt something on this lines: 
 
-{% code-tabs %}
-{% code-tabs-item title="bash-prompt-example" %}
+{% code title="bash-prompt-example" %}
 ```bash
 (my_project)[server]$ 
 ```
-{% endcode-tabs-item %}
-{% endcode-tabs %}
+{% endcode %}
 {% endhint %}
 
 
 
 ## Step 2: Clone the repository
 
-{% code-tabs %}
-{% code-tabs-item title="git-clone-with-submodule" %}
+{% code title="git-clone-with-submodule" %}
 ```bash
-git clone --recursive https://github.com/msk-access/fastq_to_bam.git
-cd fastq_to_bam
-git submodule update --recursive --remote
+git clone --recursive --branch 0.1.1 https://github.com/msk-access/fastq_to_bam.git
 ```
-{% endcode-tabs-item %}
-{% endcode-tabs %}
+{% endcode %}
+
+{% hint style="info" %}
+**Note:** Change 0.1.1 to the latest stable release of the pipeline
+{% endhint %}
 
 ## Step 3: Install requirements using pip
 
 We have already specified the version of cwltool and other packages in the requirements.txt file. Please use this to install.
 
-{% code-tabs %}
-{% code-tabs-item title="python-package-installation-using-pip" %}
+{% code title="python-package-installation-using-pip" %}
 ```bash
 #python2
 pip install -r requirements.txt
 #python3
 pip3 install -r requirements.txt
 ```
-{% endcode-tabs-item %}
-{% endcode-tabs %}
+{% endcode %}
 
 ## Step 4: Generate an inputs file
 
 Next you must generate a proper input file in either [json](https://www.json.org/) or [yaml](https://yaml.org/) format.
 
-For details on how to create this file, please follow this example:
+For details on how to create this file, please follow this example \(there is a minimal example of what needs to be filled in at the end of the page\):
 
 {% page-ref page="inputs-description.md" %}
 
@@ -109,13 +101,11 @@ Here we show how to use [cwltool](https://github.com/common-workflow-language/cw
 
 #### Run the workflow with a given set of input using [cwltool](https://github.com/common-workflow-language/cwltool) on single machine
 
-{% code-tabs %}
-{% code-tabs-item title="cwltool-execution" %}
+{% code title="cwltool-execution" %}
 ```bash
 cwltool fastq_to_bam.cwl inputs.yaml
 ```
-{% endcode-tabs-item %}
-{% endcode-tabs %}
+{% endcode %}
 {% endtab %}
 
 {% tab title="Using toil-cwl-runner locally" %}
@@ -125,27 +115,27 @@ Once we have successfully installed the requirements we can now run the workflow
 
 #### Run the workflow with a given set of input using [toil](https://toil.readthedocs.io/en/latest/running/introduction.html) on single machine
 
-{% code-tabs %}
-{% code-tabs-item title="toil-local-execution" %}
+{% code title="toil-local-execution" %}
 ```bash
 toil-cwl-runner fastq_to_bam.cwl inputs.yaml
 ```
-{% endcode-tabs-item %}
-{% endcode-tabs %}
+{% endcode %}
 {% endtab %}
 
 {% tab title="Using toil-cwl-runner on JUNO" %}
 Here we show how to run the workflow using [toil-cwl-runner](https://toil.readthedocs.io/en/latest/running/introduction.html) on MSKCC internal compute cluster called JUNO which has [IBM LSF](https://www.ibm.com/support/knowledgecenter/en/SSETD4/product_welcome_platform_lsf.html) as a scheduler. 
 
-Note the use of `--singularity`to convert Docker containers into singularity containers, the `TMPDIR`envirionment variable to avoid writing temporary files to shared disk space, and `TOIl_LSF_ARGS` to specify any additional arguments to `bsub`commands that the jobs should have \(in this case, setting a max walltime of 6 hours\).
+Note the use of `--singularity`to convert Docker containers into singularity containers, the `TMPDIR` environment variable to avoid writing temporary files to shared disk space, the `_JAVA_OPTIONS` environment variable to specify java temporary directory to `/scratch`, using `SINGULARITY_BINDPATH`  environment variable to bind the `/scratch` when running singularity containers and `TOIl_LSF_ARGS` to specify any additional arguments to `bsub`commands that the jobs should have \(in this case, setting a max wall-time of 6 hours\).
 
 Run the workflow with a given set of input using [toil](https://toil.readthedocs.io/en/latest/running/introduction.html) on JUNO \(MSKCC Research Cluster\)
 
-{% code-tabs %}
-{% code-tabs-item title="toil-lsf-execution" %}
+{% code title="toil-lsf-execution" %}
+
 ```bash
 TMPDIR=$PWD
-TOIL_LSF_ARGS='-W 3600'
+TOIL_LSF_ARGS='-W 3600 -P test_fastq_to_bam -app anyOS -R select[type==CentOS7]'
+_JAVA_OPTIONS='-Djava.io.tmpdir=/scratch/'
+SINGULARITY_BINDPATH='/scratch:/scratch:rw'
 toil-cwl-runner \
        --singularity \
        --logFile ./example.log  \
@@ -156,19 +146,19 @@ toil-cwl-runner \
        --writeLogs ./example_log_folder/ \
        --logLevel DEBUG \
        --stats \
-       --retryCount 2 
+       --retryCount 2 \
        --disableCaching \
        --disableChaining \
-       --maxLogFileSize 20000000000 \
-       --cleanWorkDir onSuccess
        --preserve-environment TOIL_LSF_ARGS TMPDIR \
+       --maxLogFileSize 20000000000 \
+       --cleanWorkDir onSuccess \
        fastq_to_bam.cwl \
        inputs.yaml \
        > toil.stdout \
        2> toil.stderr &
 ```
-{% endcode-tabs-item %}
-{% endcode-tabs %}
+
+{% endcode %}
 {% endtab %}
 {% endtabs %}