Skip to content

Commit 8580cb7

Browse files
ao508ao508
committed
Initial commit - Spring Batch Importer Pipeline
General * added main class * added some data models (rest will be added as needed) * added a data file utils class (may not need ultimately) * added .gitignore * Standardized SQL statements and named parameters type Refactored importer: * removed persistence classes and models * using cbioportal/persistence repo instead Configurations * batch configuration, data source configuration, and datatype configurations added * step execution configurations for importing will be organized by datatype Job Execution: * added checks to make sure cancer study meta file exists and that the data was loaded before running rest of importer pipeline DB Compatibility: * Added db schema check to be used by job decider to determine whether or not to run the job Dao classes and repositories: * Created interfaces for Dao classes * Moved sql and database interactions to Dao classes Cancer Study: * moved step/flow configuration out of `BatchConfiguration` Clinical Data * step ex. configurations added for clinical data * all clinical datatypes use the same step components - execution is decided by step name * Updated clinical data writer and clinical data configurations * Changed import of new patients/samples to single imports vs batch * Added support for clinical supp files * cleaned up step/flow configuration in `ClinicalStepConfiguration` Mutation Data: * added a composite maf record class to hold all possible fields in a maf record * Added step configuration for importing mutation data * Added a reader, processor, writer, listener, and tasklets for mutation data * Added utils classes for mutation data processing Added documentation for: * main batch importer job * cancer study step * clinical data step * mutation data step Signed-off-by: Angelica Ochoa <aochoa4230@gmail.com>
1 parent 0dc63f9 commit 8580cb7

34 files changed

Lines changed: 6528 additions & 0 deletions

‎.gitignore‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
.gitignore
2+
/importer/nbproject/private/
3+
/importer/dist/
4+
/importer/build/
5+
/importer/target/
6+
/importer/src/main/resources/application.properties
7+
/importer/nbproject/
8+
/importer/nb-configuration.xml

‎docs/Importer-Workflow.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
## Outline:
2+
* [Introduction](introduction)
3+
* [Importer Workflow](importer-workflow)
4+
* [Job Configuration Specs](job-config-specs)
5+
* [Step Configuration Specs](step-config-specs)
6+
7+
# [Introduction](introduction)
8+
The Spring Batch importer comprises one job with a series of steps that are executed sequentially or in parallel depending on their datatype. For reference, the importer workflow diagram can be viewed [here](ImporterWorkflowDiagram.pdf).
9+
10+
The `application.properties` file requires the following:
11+
* `spring.batch.job.enabled`=false
12+
* `chunk.interval`= some integer that will be set as the chunk processing interval value
13+
* `db.user`= the db username
14+
* `db.password`= the db password
15+
* `db.driver`= the driver classname
16+
* `db.connection_string`= the db connection string
17+
* `db.portal_db_name`= the db name
18+
* `db.url`= the db url
19+
* `db.version=${db.version}`: db.version is defined in the master pom file
20+
21+
The user must pass one argument when running the importer (`-s`, `--staging`) which will be added to the `JobExecutionContext` as a `JobParameter` called `stagingDirectory`. The following is an example command for importing a cancer study at a specified directory:
22+
23+
```
24+
$JAVA_HOME/bin/java -jar importer/target/importer-0.1.0.jar -s /path/to/cancer/study
25+
```
26+
27+
# [Importer Workflow](importer-workflow):
28+
The following describes the components of the batch importer job and its steps.
29+
30+
## [Job Configuration Specs](job-config-specs):
31+
* `JobExecutionListener`: Used for letting user know whether cancer study data successfully imported or not.
32+
* `JobExecutionDecider`: Checks the DB schema version and stops the job if the DB version does not match what is expected for the portal. If the DB schema does not match what is expected then the `JobExecutionDecider` sets the `FlowExecutionStatus` to **STOPPED**. Otherwise, the `FlowExecutionStatus` is set to **CONTINUE**.
33+
* `Flow`: Defines the conditional flow for importing a cancer study based on the `FlowExecutionStatus` from the `JobExecutionDecider`. On **STOPPED**, the job execution is halted and the application closes. On **CONTINUE**, the job attempts to import a cancer study based on the staging directory passed on the command line.
34+
35+
## [Step Configuration Specs](step-config-specs)
36+
* [Cancer Study Step](cancer-study-step)
37+
* [Clinical Data Step](clinical-data-step)
38+
* [Mutation Data Step](mutation-data-step)
39+
40+
### [Cancer Study Step](cancer-study-step):
41+
Importing cancer study data will be executed by the `importCancerStudy` Step (executed by the main batch importer job described in [Job Configuration Specs](job-config-specs).
42+
* `StepExecutionListener`: Used for setting the `ExitStatus` to **STOPPED** after the step has executed if the cancer study metadata could not be imported or if the cancer study metadata could not be loaded from `meta_study.txt`.
43+
* `Tasklet`: Used for loading cancer study metadata from `meta_study.txt`. If a cancer study already exists by the cancer study identifier then the existing study is deleted and the new study is imported.
44+
* `JobExecutionDecider`: Used to determine whether the impor job should continue or not. If the cancer study metadata was successfully loaded and imported into the DB then the `FlowExecutionStatus` is set to **CONTINUE**, otherwise the `FlowExecutionStatus` is set to **STOPPED**.
45+
* `Flow`: Defines the conditional flow for importing other datatypes for a cancer study based on the `FlowExecutionStatus` from the `JobExecutionDecider`. If the `FlowExecutionStatus` is set to **CONTINUE** then the job will continue executing by first loading and importing clinical data and then executing the remaining import steps in parallel.
46+
47+
### [Clinical Data Step](clinical-data-step)
48+
The following datatypes are loaded and imported sequentially if metadata files are found for them:
49+
50+
| Datatype | Meta Filename | Data Filename | Status | Step Name |
51+
| ---------------- | ------------------------- | ------------------------- | ------------- | ------------------- |
52+
| clinical | meta_clinical.txt | data_clinical.txt | **Complete** | clinicalStep |
53+
| clinical-patient | meta_clinical_patient.txt | data_clinical_patient.txt | **Complete** | clinicalPatientStep |
54+
| clinical-sample | meta_clinical_sample.txt | data_clinical_sample.txt | **Complete** | clinicalSampleStep |
55+
| clinical-supp | meta_clinical.txt | data_clinical_supp*.txt | **Complete** | clinicalSuppStep |
56+
| bcr-clinical | meta_bcr_clinical*.txt | data_bcr_clinical*.txt | _In Progress_ | -- |
57+
| clinical-caises | meta_clinical_caises.txt | data_clinical_caises.xml | _In Progress_ | -- |
58+
59+
A main Flow (`clinicalStepFlow`) will execute the Steps for importing each clinical datatype sequentially. All clinical datatype steps will have the following components in common:
60+
* `JobExecutionDecider`: Searches the cancer study path for a datatype's meta filename and data filename. If the meta file exists and data files can be found then the `FlowExecutionStatus` is set to **RUN** and the current step name is injected into the `JobExecutionContext`, otherwise the `FlowExecutionStatus` is set to **SKIP**.
61+
* `Tasklet`: A universal tasklet loads clinical attributes from a list of clinical datafiles based on the data filename corresponding to the `currentStep` stored in the `JobExecutionContext`. It is assumed that metadata for clinical attributes are stored at the top of the file and that the clinical datafile(s) have passed validation. A HashMap of data filenames and clinical attributes loaded are injected into the `JobExecutionContext` to be used by the clinical data readers, processors, and writers.
62+
* `Step`: A universal step builder for clinical data implements the reading, processing, and writing of clinical data using the HashMap of datafile clinical attributes loaded from the `Tasklet` above.
63+
* `StepExecutionListener`: Before the clinical datatype step executes, the listener will add the cancer study imported from [Cancer Study Step](cancer-study-step) to the `StepExecutionContext`, as well as the datafile clinical attributes loaded from the `Tasklet` described above. After the clinical datatype step executes, the listener will report how many patient records, sample records, patient clinical data, and sample clinical data were loaded and imported into the db, as well as how many rollbacks or skips occurred during import.
64+
65+
### [Mutation Data Step](mutation-data-step)
66+
The following datatypes are loaded and imported in parallel if metadata files are found for them:
67+
68+
| Datatype | Meta Filename | Data Filename | Status | Step Name |
69+
| ------------------- | --------------------------- | -------------------------------------- | ------------- | ---------------------- |
70+
| mutation | meta_mutations_extended.txt | data_mutations_extended*.txt | **Complete** | mutationStep |
71+
| mutation-germline | meta_mutations_extended.txt | data_mutations_germline.txt | **Complete** | mutationGermlineStep |
72+
| mutation-foundation | meta_mutations_extended.txt | data_mutations_extended_foundation.txt | **Complete** | mutationFoundationStep |
73+
| mutation-unfiltered | meta_mutations_extended.txt | data_mutations_unfiltered.txt | _In Progress_ | -- |
74+
| mutation-manual | meta_mutations_extended.txt | data_mutations_manual.txt | _In Progress_ | -- |
75+
_* **Note:**_ `data_mutations_extended_foundation.txt` _is filtered out of the list of datafiles when searching for filename pattern_ `data_mutations_extended*.txt` _for datatype_ `mutation`.
76+
77+
A main Flow (`mutationStepFlow`) will execute the Steps for importing each mutation datatype sequentially. All mutation datatype steps will have the following components in common:
78+
* `JobExecutionDecider`: Searches the cancer study path for a datatype's meta filename and data filename. If the meta file exists and the data files can be found then the `FlowExecutionStatus` is set to **RUN** and the current step name is injected into the `JobExecutionContext`, otherwise the `FlowExecutionStatus` is set to **SKIP**. To avoid loading and importing the `data_mutations_extended_foundation.txt` during the general mutation step, the mutation-foundation data filename is filtered
79+
* `Tasklets`: Two universal tasklets are used for mutation data steps. One tasklet loads and imports a genetic profile from a datatype's metafile and injects the genetic profile into the `JobExecutionContext`. The second tasklet injects the list of datafiles into the `JobExecutionContext`, as well as a MultiKeyMap of the headers of each MAF file and number of records in the file (row count). These information are used by the mutation data readers, processors, and writers.
80+
* `Step`: A universal step builder for mutation data implements the reading, processing, and writing of mutation data using the genetic profile and the MultiKeyMap of MAF file metadata (headers and number of records in file) loaded from the `Tasklet`s above. A report of mutations filtered out is printed after each file is read.
81+
* `StepExecutionListener`: Before the mutation datatype step executes, the listener will add the data loaded from the `Tasklet`s above. After the mutation datatype step executes, the listener will report the total samples loaded, total genes loaded, samples skipped, and entries skipped, as well as the total records imported into MUTATION and MUTATION_EVENT for the current datatype.
82+

‎docs/ImporterWorkflowDiagram.pdf‎

47.8 KB
Binary file not shown.

‎importer/pom.xml‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
<name>cBioPortal Importer Pipeline</name>
5+
<description>Spring Batch importer pipeline</description>
6+
<artifactId>importer</artifactId>
7+
<version>0.1.0</version>
8+
9+
<parent>
10+
<groupId>org.cbio.portal.pipelines</groupId>
11+
<artifactId>master</artifactId>
12+
<version>0.1.0</version>
13+
</parent>
14+
15+
16+
<dependencies>
17+
<dependency>
18+
<groupId>org.mskcc.cbio</groupId>
19+
<artifactId>model</artifactId>
20+
<version>0.1.0</version>
21+
<type>jar</type>
22+
</dependency>
23+
<dependency>
24+
<groupId>org.mskcc.cbio</groupId>
25+
<artifactId>persistence-jdbc</artifactId>
26+
<version>0.1.0</version>
27+
<type>jar</type>
28+
</dependency>
29+
30+
<dependency>
31+
<groupId>org.springframework</groupId>
32+
<artifactId>spring-web</artifactId>
33+
</dependency>
34+
<dependency>
35+
<groupId>commons-lang</groupId>
36+
<artifactId>commons-lang</artifactId>
37+
<version>2.4</version>
38+
</dependency>
39+
<dependency>
40+
<groupId>commons-cli</groupId>
41+
<artifactId>commons-cli</artifactId>
42+
<version>1.3</version>
43+
</dependency>
44+
<dependency>
45+
<groupId>commons-collections</groupId>
46+
<artifactId>commons-collections</artifactId>
47+
<type>jar</type>
48+
</dependency>
49+
50+
<!-- for Spring JDBC, BasicDataSource usage -->
51+
<dependency>
52+
<groupId>org.springframework</groupId>
53+
<artifactId>spring-jdbc</artifactId>
54+
<type>jar</type>
55+
</dependency>
56+
<dependency>
57+
<groupId>commons-dbcp</groupId>
58+
<artifactId>commons-dbcp</artifactId>
59+
<type>jar</type>
60+
</dependency>
61+
<dependency>
62+
<groupId>mysql</groupId>
63+
<artifactId>mysql-connector-java</artifactId>
64+
</dependency>
65+
66+
<dependency>
67+
<groupId>com.google.guava</groupId>
68+
<artifactId>guava</artifactId>
69+
<version>19.0</version>
70+
<type>jar</type>
71+
</dependency>
72+
73+
</dependencies>
74+
75+
76+
<build>
77+
<!-- this plugin will allow us to share resources with children -->
78+
<!-- <plugins>
79+
<plugin>
80+
<groupId>org.apache.maven.plugins</groupId>
81+
<artifactId>maven-resources-plugin</artifactId>
82+
<version>2.6</version>
83+
</plugin>
84+
<plugin>
85+
<groupId>org.apache.maven.plugins</groupId>
86+
<artifactId>maven-site-plugin</artifactId>
87+
<version>3.4</version>
88+
</plugin>
89+
<plugin>
90+
<groupId>org.apache.maven.plugins</groupId>
91+
<artifactId>maven-release-plugin</artifactId>
92+
<version>2.5.3</version>
93+
</plugin>
94+
</plugins>-->
95+
96+
<pluginManagement>
97+
<plugins>
98+
<plugin>
99+
<groupId>org.apache.maven.plugins</groupId>
100+
<artifactId>maven-compiler-plugin</artifactId>
101+
<version>3.5.1</version>
102+
<configuration>
103+
<source>1.8</source>
104+
<target>1.8</target>
105+
</configuration>
106+
</plugin>
107+
</plugins>
108+
</pluginManagement>
109+
</build>
110+
<repositories>
111+
<repository>
112+
<id>persistence-mvn-repo</id>
113+
<url>https://raw.github.com/angelicaochoa/persistence/mvn-repo/</url>
114+
<snapshots>
115+
<enabled>true</enabled>
116+
<updatePolicy>always</updatePolicy>
117+
</snapshots>
118+
</repository>
119+
</repositories>
120+
</project>

‎importer/src/main/java/org/cbio/portal/pipelines/importer/ImporterPipeline.java‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
/*
2+
* Copyright (c) 2015 Memorial Sloan-Kettering Cancer Center.
3+
*
4+
* This library is distributed in the hope that it will be useful, but WITHOUT
5+
* ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS
6+
* FOR A PARTICULAR PURPOSE. The software and documentation provided hereunder
7+
* is on an "as is" basis, and Memorial Sloan-Kettering Cancer Center has no
8+
* obligations to provide maintenance, support, updates, enhancements or
9+
* modifications. In no event shall Memorial Sloan-Kettering Cancer Center be
10+
* liable to any party for direct, indirect, special, incidental or
11+
* consequential damages, including lost profits, arising out of the use of this
12+
* software and its documentation, even if Memorial Sloan-Kettering Cancer
13+
* Center has been advised of the possibility of such damage.
14+
*/
15+
16+
/*
17+
* This file is part of cBioPortal.
18+
*
19+
* cBioPortal is free software: you can redistribute it and/or modify
20+
* it under the terms of the GNU Affero General Public License as
21+
* published by the Free Software Foundation, either version 3 of the
22+
* License.
23+
*
24+
* This program is distributed in the hope that it will be useful,
25+
* but WITHOUT ANY WARRANTY; without even the implied warranty of
26+
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
27+
* GNU Affero General Public License for more details.
28+
*
29+
* You should have received a copy of the GNU Affero General Public License
30+
* along with this program. If not, see <http://www.gnu.org/licenses/>.
31+
*/
32+
33+
package org.cbio.portal.pipelines.importer;
34+
35+
import org.cbio.portal.pipelines.importer.config.BatchConfiguration;
36+
37+
import java.io.*;
38+
import java.util.*;
39+
import org.apache.commons.cli.*;
40+
import org.apache.commons.logging.*;
41+
42+
import org.springframework.boot.SpringApplication;
43+
import org.springframework.boot.autoconfigure.SpringBootApplication;
44+
import org.springframework.context.ConfigurableApplicationContext;
45+
import org.springframework.batch.core.*;
46+
import org.springframework.batch.core.launch.JobLauncher;
47+
48+
/**
49+
*
50+
* @author ochoaa
51+
*/
52+
@SpringBootApplication
53+
public class ImporterPipeline {
54+
55+
private static final Log LOG = LogFactory.getLog(ImporterPipeline.class);
56+
57+
private static Options getOptions(String[] args) {
58+
Options gnuOptions = new Options();
59+
gnuOptions.addOption("h", "help", false, "shows this help document and quits.")
60+
.addOption("s", "staging", true, "Staging directory");
61+
return gnuOptions;
62+
}
63+
64+
private static void help(Options gnuOptions, int exitStatus) {
65+
HelpFormatter helpFormatter = new HelpFormatter();
66+
helpFormatter.printHelp("ImporterPipeline", gnuOptions);
67+
System.exit(exitStatus);
68+
}
69+
70+
private static void launchJob(String[] args, String stagingDirectory) throws Exception {
71+
72+
SpringApplication app = new SpringApplication(ImporterPipeline.class);
73+
ConfigurableApplicationContext ctx = app.run(args);
74+
JobLauncher jobLauncher = ctx.getBean(JobLauncher.class);
75+
76+
Job batchImporterJob = ctx.getBean(BatchConfiguration.BATCH_STUDY_IMPORTER_JOB, Job.class);
77+
78+
JobParameters jobParameters = new JobParametersBuilder()
79+
.addString("stagingDirectory", stagingDirectory)
80+
.addDate("date", new Date())
81+
.toJobParameters();
82+
83+
JobExecution jobExecution = jobLauncher.run(batchImporterJob, jobParameters);
84+
if (jobExecution.getExitStatus().getExitCode().equals("STOPPED")) {
85+
LOG.error("Error importing cancer study.");
86+
}
87+
88+
LOG.info("Shutting down ImporterPipeline.");
89+
ctx.close();
90+
}
91+
92+
public static void main(String[] args) throws Exception {
93+
Options gnuOptions = ImporterPipeline.getOptions(args);
94+
CommandLineParser parser = new GnuParser();
95+
CommandLine commandLine = parser.parse(gnuOptions, args);
96+
if (commandLine.hasOption("h") ||
97+
!commandLine.hasOption("s")) {
98+
help(gnuOptions, 0);
99+
}
100+
String stagingDirectory = commandLine.getOptionValue("s");
101+
if (!(new File(stagingDirectory).exists())) {
102+
LOG.error("Staging directory does not exist - please check argument: " + stagingDirectory);
103+
System.exit(2);
104+
}
105+
launchJob(args, stagingDirectory);
106+
107+
}
108+
109+
}

0 commit comments

Comments
 (0)