Skip to content

Commit 109b10e

Browse files
khoroshevskyikhoroshevskyi
authored
Merge pull request #122 from pepkit/dev
Release v0.12.4
2 parents 7824861 + 73946b7 commit 109b10e

12 files changed

Lines changed: 764 additions & 676 deletions

‎docs/changelog.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# Changelog
22

3+
## [0.12.4] -- 2023-08-01
4+
- Fixed SRA convert
5+
- Added how to convert SRA
6+
37
## [0.12.3] -- 2023-06-21
48
- Fixed preserving order of project keys (#119)
59

‎docs/sra_convert.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,4 @@ This effectively makes it easier to interact with *project-level* management of
1515

1616
## Tutorial
1717

18-
See the [tutorial](raw-data-downloading.md) for an example of how to use `sraconvert`.
18+
See the [how-to](how_to_convert_fastq_from_sra.md) for an example of how to use `sraconvert`.

‎docs_jupyter/build/processed-data-downloading.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,11 @@ Calling geofetch will do 4 tasks:
2424

2525
Complete details about geofetch outputs is cataloged in the [metadata outputs reference](metadata_output.md).
2626

27+
from IPython.core.display import SVG
28+
SVG(filename='logo.svg')
29+
30+
![arguments_outputs.svg](attachment:arguments_outputs.svg)
31+
2732
## Download the data
2833

2934
First, create the metadata for processed data (by adding --processed and --just-metadata):

‎docs_jupyter/build/raw-data-downloading.md‎

Lines changed: 0 additions & 307 deletions
Original file line numberDiff line numberDiff line change
@@ -382,313 +382,6 @@ Writing: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/red_algae/GSE67
382382
383383
```
384384

385-
## Convert to fastq format
386-
387-
Now the `.sra` files have been downloaded. The project that was automatically created by GEO contained an amendment for sra file conversion. This project expects you to have an environment variable called `SRARAW` that points to the location where `prefetch` stores your `.sra` files. We also should define a `$SRAFQ` variable to point to where we ant the fastq files stored. In this command below, we set these on the fly for this command, but you can also just use globals.
388-
389-
We'll use `-d` first to do a dry run:
390-
391-
392-
```bash
393-
SRARAW=${HOME}/ncbi/public/sra/ SRAFQ=red_algae/fastq \
394-
looper run red_algae/red_algae_config.yaml -a sra_convert -p local -d
395-
```
396-
397-
```.output
398-
Looper version: 1.2.0-dev
399-
Command: run
400-
Using amendments: sra_convert
401-
Activating compute package 'local'
402-
## [1 of 4] sample: Cm_BlueLight_Rep1; pipeline: sra_convert
403-
Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_BlueLight_Rep1.sub
404-
Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_BlueLight_Rep1.sub
405-
Dry run, not submitted
406-
## [2 of 4] sample: Cm_BlueLight_Rep2; pipeline: sra_convert
407-
Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_BlueLight_Rep2.sub
408-
Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_BlueLight_Rep2.sub
409-
Dry run, not submitted
410-
## [3 of 4] sample: Cm_Darkness_Rep1; pipeline: sra_convert
411-
Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_Darkness_Rep1.sub
412-
Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_Darkness_Rep1.sub
413-
Dry run, not submitted
414-
## [4 of 4] sample: Cm_Darkness_Rep2; pipeline: sra_convert
415-
Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_Darkness_Rep2.sub
416-
Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_Darkness_Rep2.sub
417-
Dry run, not submitted
418-
419-
Looper finished
420-
Samples valid for job generation: 4 of 4
421-
Commands submitted: 4 of 4
422-
Jobs submitted: 4
423-
Dry run. No jobs were actually submitted.
424-
425-
```
426-
427-
And now the real thing:
428-
429-
430-
```bash
431-
SRARAW=${HOME}/ncbi/public/sra/ SRAFQ=red_algae/fastq \
432-
looper run red_algae/red_algae_config.yaml -a sra_convert -p local \
433-
--command-extra=--keep-sra
434-
```
435-
436-
```.output
437-
Looper version: 1.2.0-dev
438-
Command: run
439-
Using amendments: sra_convert
440-
Activating compute package 'local'
441-
## [1 of 4] sample: Cm_BlueLight_Rep1; pipeline: sra_convert
442-
Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_BlueLight_Rep1.sub
443-
Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_BlueLight_Rep1.sub
444-
Compute node: zither
445-
Start time: 2020-05-21 17:40:56
446-
Using outfolder: red_algae/results_pipeline/SRX969073
447-
### Pipeline run code and environment:
448-
449-
* Command: `/home/nsheff/.local/bin/sraconvert --srr /home/nsheff/ncbi/public/sra//SRR1930183.sra --sample-name SRX969073 -O red_algae/results_pipeline --keep-sra`
450-
* Compute host: zither
451-
* Working dir: /home/nsheff/code/geofetch/docs_jupyter
452-
* Outfolder: red_algae/results_pipeline/SRX969073/
453-
* Pipeline started at: (05-21 17:40:57) elapsed: 0.0 _TIME_
454-
455-
### Version log:
456-
457-
* Python version: 3.7.5
458-
* Pypiper dir: `/home/nsheff/.local/lib/python3.7/site-packages/pypiper`
459-
* Pypiper version: 0.12.1
460-
* Pipeline dir: `/home/nsheff/.local/bin`
461-
* Pipeline version: None
462-
463-
### Arguments passed to pipeline:
464-
465-
* `bamfolder`: ``
466-
* `config_file`: `sraconvert.yaml`
467-
* `format`: `fastq`
468-
* `fqfolder`: `red_algae/fastq`
469-
* `keep_sra`: `True`
470-
* `logdev`: `False`
471-
* `mode`: `convert`
472-
* `output_parent`: `red_algae/results_pipeline`
473-
* `recover`: `False`
474-
* `sample_name`: `['SRX969073']`
475-
* `silent`: `False`
476-
* `srafolder`: `/home/nsheff/ncbi/public/sra/`
477-
* `srr`: `['/home/nsheff/ncbi/public/sra//SRR1930183.sra']`
478-
* `verbosity`: `None`
479-
480-
----------------------------------------
481-
482-
Processing 1 of 1 files: SRR1930183
483-
Target to produce: `red_algae/fastq/SRR1930183_1.fastq.gz`
484-
485-
> `fastq-dump /home/nsheff/ncbi/public/sra//SRR1930183.sra --split-files --gzip -O red_algae/fastq` (9436)
486-
<pre>
487-
Read 1068319 spots for /home/nsheff/ncbi/public/sra//SRR1930183.sra
488-
Written 1068319 spots for /home/nsheff/ncbi/public/sra//SRR1930183.sra
489-
</pre>
490-
Command completed. Elapsed time: 0:00:38. Running peak memory: 0.067GB.
491-
PID: 9436; Command: fastq-dump; Return code: 0; Memory used: 0.067GB
492-
493-
Already completed files: []
494-
495-
### Pipeline completed. Epilogue
496-
* Elapsed time (this run): 0:00:38
497-
* Total elapsed time (all runs): 0:00:38
498-
* Peak memory (this run): 0.0666 GB
499-
* Pipeline completed time: 2020-05-21 17:41:35
500-
## [2 of 4] sample: Cm_BlueLight_Rep2; pipeline: sra_convert
501-
Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_BlueLight_Rep2.sub
502-
Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_BlueLight_Rep2.sub
503-
Compute node: zither
504-
Start time: 2020-05-21 17:41:36
505-
Using outfolder: red_algae/results_pipeline/SRX969074
506-
### Pipeline run code and environment:
507-
508-
* Command: `/home/nsheff/.local/bin/sraconvert --srr /home/nsheff/ncbi/public/sra//SRR1930184.sra --sample-name SRX969074 -O red_algae/results_pipeline --keep-sra`
509-
* Compute host: zither
510-
* Working dir: /home/nsheff/code/geofetch/docs_jupyter
511-
* Outfolder: red_algae/results_pipeline/SRX969074/
512-
* Pipeline started at: (05-21 17:41:36) elapsed: 0.0 _TIME_
513-
514-
### Version log:
515-
516-
* Python version: 3.7.5
517-
* Pypiper dir: `/home/nsheff/.local/lib/python3.7/site-packages/pypiper`
518-
* Pypiper version: 0.12.1
519-
* Pipeline dir: `/home/nsheff/.local/bin`
520-
* Pipeline version: None
521-
522-
### Arguments passed to pipeline:
523-
524-
* `bamfolder`: ``
525-
* `config_file`: `sraconvert.yaml`
526-
* `format`: `fastq`
527-
* `fqfolder`: `red_algae/fastq`
528-
* `keep_sra`: `True`
529-
* `logdev`: `False`
530-
* `mode`: `convert`
531-
* `output_parent`: `red_algae/results_pipeline`
532-
* `recover`: `False`
533-
* `sample_name`: `['SRX969074']`
534-
* `silent`: `False`
535-
* `srafolder`: `/home/nsheff/ncbi/public/sra/`
536-
* `srr`: `['/home/nsheff/ncbi/public/sra//SRR1930184.sra']`
537-
* `verbosity`: `None`
538-
539-
----------------------------------------
540-
541-
Processing 1 of 1 files: SRR1930184
542-
Target exists: `red_algae/fastq/SRR1930184_1.fastq.gz`
543-
Already completed files: []
544-
545-
### Pipeline completed. Epilogue
546-
* Elapsed time (this run): 0:00:00
547-
* Total elapsed time (all runs): 0:00:00
548-
* Peak memory (this run): 0 GB
549-
* Pipeline completed time: 2020-05-21 17:41:36
550-
## [3 of 4] sample: Cm_Darkness_Rep1; pipeline: sra_convert
551-
Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_Darkness_Rep1.sub
552-
Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_Darkness_Rep1.sub
553-
Compute node: zither
554-
Start time: 2020-05-21 17:41:36
555-
Using outfolder: red_algae/results_pipeline/SRX969075
556-
### Pipeline run code and environment:
557-
558-
* Command: `/home/nsheff/.local/bin/sraconvert --srr /home/nsheff/ncbi/public/sra//SRR1930185.sra --sample-name SRX969075 -O red_algae/results_pipeline --keep-sra`
559-
* Compute host: zither
560-
* Working dir: /home/nsheff/code/geofetch/docs_jupyter
561-
* Outfolder: red_algae/results_pipeline/SRX969075/
562-
* Pipeline started at: (05-21 17:41:36) elapsed: 0.0 _TIME_
563-
564-
### Version log:
565-
566-
* Python version: 3.7.5
567-
* Pypiper dir: `/home/nsheff/.local/lib/python3.7/site-packages/pypiper`
568-
* Pypiper version: 0.12.1
569-
* Pipeline dir: `/home/nsheff/.local/bin`
570-
* Pipeline version: None
571-
572-
### Arguments passed to pipeline:
573-
574-
* `bamfolder`: ``
575-
* `config_file`: `sraconvert.yaml`
576-
* `format`: `fastq`
577-
* `fqfolder`: `red_algae/fastq`
578-
* `keep_sra`: `True`
579-
* `logdev`: `False`
580-
* `mode`: `convert`
581-
* `output_parent`: `red_algae/results_pipeline`
582-
* `recover`: `False`
583-
* `sample_name`: `['SRX969075']`
584-
* `silent`: `False`
585-
* `srafolder`: `/home/nsheff/ncbi/public/sra/`
586-
* `srr`: `['/home/nsheff/ncbi/public/sra//SRR1930185.sra']`
587-
* `verbosity`: `None`
588-
589-
----------------------------------------
590-
591-
Processing 1 of 1 files: SRR1930185
592-
Target to produce: `red_algae/fastq/SRR1930185_1.fastq.gz`
593-
594-
> `fastq-dump /home/nsheff/ncbi/public/sra//SRR1930185.sra --split-files --gzip -O red_algae/fastq` (9607)
595-
<pre>
596-
Read 1707508 spots for /home/nsheff/ncbi/public/sra//SRR1930185.sra
597-
Written 1707508 spots for /home/nsheff/ncbi/public/sra//SRR1930185.sra
598-
</pre>
599-
Command completed. Elapsed time: 0:01:01. Running peak memory: 0.066GB.
600-
PID: 9607; Command: fastq-dump; Return code: 0; Memory used: 0.066GB
601-
602-
Already completed files: []
603-
604-
### Pipeline completed. Epilogue
605-
* Elapsed time (this run): 0:01:01
606-
* Total elapsed time (all runs): 0:01:01
607-
* Peak memory (this run): 0.0656 GB
608-
* Pipeline completed time: 2020-05-21 17:42:37
609-
## [4 of 4] sample: Cm_Darkness_Rep2; pipeline: sra_convert
610-
Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_Darkness_Rep2.sub
611-
Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_Darkness_Rep2.sub
612-
Compute node: zither
613-
Start time: 2020-05-21 17:42:38
614-
Using outfolder: red_algae/results_pipeline/SRX969076
615-
### Pipeline run code and environment:
616-
617-
* Command: `/home/nsheff/.local/bin/sraconvert --srr /home/nsheff/ncbi/public/sra//SRR1930186.sra --sample-name SRX969076 -O red_algae/results_pipeline --keep-sra`
618-
* Compute host: zither
619-
* Working dir: /home/nsheff/code/geofetch/docs_jupyter
620-
* Outfolder: red_algae/results_pipeline/SRX969076/
621-
* Pipeline started at: (05-21 17:42:38) elapsed: 0.0 _TIME_
622-
623-
### Version log:
624-
625-
* Python version: 3.7.5
626-
* Pypiper dir: `/home/nsheff/.local/lib/python3.7/site-packages/pypiper`
627-
* Pypiper version: 0.12.1
628-
* Pipeline dir: `/home/nsheff/.local/bin`
629-
* Pipeline version: None
630-
631-
### Arguments passed to pipeline:
632-
633-
* `bamfolder`: ``
634-
* `config_file`: `sraconvert.yaml`
635-
* `format`: `fastq`
636-
* `fqfolder`: `red_algae/fastq`
637-
* `keep_sra`: `True`
638-
* `logdev`: `False`
639-
* `mode`: `convert`
640-
* `output_parent`: `red_algae/results_pipeline`
641-
* `recover`: `False`
642-
* `sample_name`: `['SRX969076']`
643-
* `silent`: `False`
644-
* `srafolder`: `/home/nsheff/ncbi/public/sra/`
645-
* `srr`: `['/home/nsheff/ncbi/public/sra//SRR1930186.sra']`
646-
* `verbosity`: `None`
647-
648-
----------------------------------------
649-
650-
Processing 1 of 1 files: SRR1930186
651-
Target to produce: `red_algae/fastq/SRR1930186_1.fastq.gz`
652-
653-
> `fastq-dump /home/nsheff/ncbi/public/sra//SRR1930186.sra --split-files --gzip -O red_algae/fastq` (9780)
654-
<pre>
655-
Read 1224029 spots for /home/nsheff/ncbi/public/sra//SRR1930186.sra
656-
Written 1224029 spots for /home/nsheff/ncbi/public/sra//SRR1930186.sra
657-
</pre>
658-
Command completed. Elapsed time: 0:00:44. Running peak memory: 0.067GB.
659-
PID: 9780; Command: fastq-dump; Return code: 0; Memory used: 0.067GB
660-
661-
Already completed files: []
662-
663-
### Pipeline completed. Epilogue
664-
* Elapsed time (this run): 0:00:44
665-
* Total elapsed time (all runs): 0:00:44
666-
* Peak memory (this run): 0.0673 GB
667-
* Pipeline completed time: 2020-05-21 17:43:22
668-
669-
Looper finished
670-
Samples valid for job generation: 4 of 4
671-
Commands submitted: 4 of 4
672-
Jobs submitted: 4
673-
674-
```
675-
676-
Now that's done, let's take a look in the `red_algae/fastq` folder (where we set the `$SRAFQ` variable).
677-
678-
679-
```bash
680-
ls red_algae/fastq
681-
```
682-
683-
```.output
684-
SRR1930183_1.fastq.gz SRR1930184_2.fastq.gz SRR1930186_1.fastq.gz
685-
SRR1930183_2.fastq.gz SRR1930185_1.fastq.gz SRR1930186_2.fastq.gz
686-
SRR1930184_1.fastq.gz SRR1930185_2.fastq.gz
687-
688-
```
689-
690-
By default, the sra conversion script will delete the `.sra` files after they have been converted to fastq. You can keep them if you want by passing `--keep-sra`, which you can do by passing `--command-extra=--keep-sra` to your `looper run` command.
691-
692385

693386
## Finalize the project config and sample annotation
694387

0 commit comments

Comments
 (0)