Repository navigation
Release v008 - #164
Merged
Merged
Release v008#164
Conversation
…ts (#109) sourceInputBuilderScripts() now accepts configBlock and pipelineVersion and exposes them to builder scripts via an inputBuilderEnv object, so scripts can create execution settings without hard-coding either value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orts script name sourceInputBuilderScripts() now loops over configBlock like execStudyPipeline(), exposing the current block via inputBuilderEnv. makeInputBuilderScript() now writes build_dependent_cohorts.R (the name that is actually sourced) and a warning flags the legacy build_dependent_cohorts_cohort.R name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
execute_pipeline() previously built execution settings and generated cohorts for configBlock[1] only, while tasks ran for every block. Cohort generation now loops over all config blocks before tasks run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sourceInputBuilderScripts() now requires all six builder scripts and aborts before sourcing if one is missing or an unrecognized .R file is present, instead of skipping silently. The builder templates now run without error when unpopulated: manifests are created on first run, ATLAS is only contacted when there is ATLAS work to do, and Capr is not loaded until used. updateAtlasCohorts()/updateAtlasConceptSets() no longer error on a manifest without ATLAS entries. The warnMissing argument is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ersion The config.yml version is read before execStudyPipeline() increments it, and a test run would silently fall back to it and target the production cohort table. Default to "prod" like createExecutionSettingsFromConfig() and require test runs to pass their namespace explicitly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conditional ATLAS logic in the import templates could skip steps silently during development. Every step is now commented out with instructions to uncomment as needed, so an unpopulated template is a no-op and a populated one fails loudly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Imported definitions are saved as <atlasId>_<ATLAS name> in snake_case, not by label, and file_name is not a load file column. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The load csvs are kept in the study repo as the record of which ATLAS cohorts/concept sets a study uses. Keep the stopIfExists = TRUE default but stop describing the csv as a throwaway in templates, method docs, comments, error hints and vignettes, and explain stopIfExists = FALSE as the alternative for re-importing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sourceInputBuilderScripts() and sourceDisseminationScripts() gain stopOnError (default TRUE): the first failing script stops the run, with the script's error as the cause. stopOnError = FALSE warns and continues, returning errors in error_summary. Also fixes sourceDisseminationScripts() never recording errors (local assignment in the handler) and removes the outer tryCatch that could swallow failures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
checkAtlasCohorts()/updateAtlasCohorts() and the concept set equivalents warned and skipped entries whose fetch failed, so a builder script could finish with the manifest silently out of date. A failed fetch now aborts, naming the ATLAS id and label, with the original error as the cause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ches checkAtlasCohorts()/updateAtlasCohorts() and the concept set equivalents gain stopOnError (default TRUE). With FALSE, an entry whose fetch fails is skipped with a warning and the rest are still synced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he pipeline Builder scripts can make database-specific manifest changes (e.g. concept ids in custom SQL), which were overwritten by later databases' builders before cohorts were generated. execStudyPipeline()/testStudyPipeline() now source the builders for each config block, then generate that block's cohorts and run its tasks before moving on; main.R no longer sources builders itself. Pre-flight warns instead of failing when the cohort manifest does not exist yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tale flag Builders can make database-specific changes, so the shared manifest flips definitions as the pipeline moves through databases and marked derived cohorts stale on every pass, forcing regeneration in every database. The derived-cohort checksum now also covers the cohort's own rendered SQL, and the checksum stored in each database's work schema alone decides whether to skip. Stale is cleared when a cohort is skipped as up to date. Note: derived cohorts regenerate once per database after upgrading, since their stored checksums use the old formula. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Rework ATLAS import builder templates around keeping the load csv and importing with stopIfExists = FALSE; reorder sections and fix stale notes (#112) - Run renv::restore() by default in generated main.R (#113) - Fix wrong/missing params in main.R and test_main.R, and builder script names in main.R comments (#115) - buildCompositeCohort(): clarify minEventCount and default it to the number of criteria cohorts (#126) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert the default change for #126 to avoid a breaking change. minEventCount stays 1L, and a warning recommends setting it to the number of criteria cohorts when it is not supplied. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ort template Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds of pipeline results - Add ConceptSetManifest$getManifestHash() (includes label/category/tags) and record concept_set_manifest_hash in task_run_history (#125) - Hash renv.lock package/R versions into renv_lock_hash (#118) - validateStudyTask() fails tasks that read files from pipeline results and warns on other untracked data-file reads (#118) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Changes to files a task reads are not tracked yet, so a task with any data-file read is always rerun. Tracking those files is left as a TODO. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
testStudyPipeline(), execStudyPipeline(), and testStudyTask() accept forceRerun = TRUE (all tasks) or a vector of task file names. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Handling of tasks that read data files will be addressed separately. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-item definition hashes (CohortDef$getSqlHash(), ConceptSetDef$getHash()) cover only the SQL/JSON. New getTaskRerunHash() on both manifests combines each definition hash with label, category, and tags, since tasks select cohorts and concept sets by that metadata. CohortManifest$getManifestHash() is kept as a deprecated alias. NEWS entries move to a development-version section since 0.0.7 is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CohortManifest$getManifestHash() keeps its name and now covers label, category, and tags. ConceptSetManifest$getManifestHash() matches it and hashes normalized expression JSON so reformatting does not rerun tasks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…checksum Pins the property that lets the per-database checksum alone decide cohort regeneration: hashes are recorded in topological order, so editing a base cohort's file changes every descendant's dependency hash while unrelated cohorts keep theirs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… default execStudyPipeline()/testStudyPipeline() gain skipInputBuilders (default TRUE), so the pipeline no longer sources the input builder scripts unless asked, preserving the previous behavior. The generated main.R and test_main.R source the builders before running the pipeline again. Pre-flight only treats a missing cohort manifest as a warning when the pipeline will run the builders. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parameterize input builders per DB; fail loudly on missing builders and script errors
…cksum columns - initUlyssesRepo() works in an existing empty directory and gains allowExisting to add only missing scaffold files to a non-empty one (#148) - Read checksum-table query results by position instead of uppercase column names (#149) - ExecutionSettings accepts any DatabaseConnectorConnection, e.g. DuckDB (#150) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PostgreSQL and SQL Server reject DISTINCT in window functions, so compute the per-subject distinct cohort count in a grouped subquery instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Close some minor bugs in issue log
Fix cross-DB compatibility bugs, allow init in existing directory
Non-interactive guard for extractSourceCodes() and vignette corrections for the six required input builder scripts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
picard 0.0.8
Breaking Changes
sourceInputBuilderScripts()now requires all six builder scripts and abortsbefore sourcing anything if one is missing or if
inputs/cohorts/R/orinputs/conceptSets/R/contains any other.Rfile (previously these wereskipped silently). Studies that deleted unused builders should recreate them
with
makeInputBuilderScript(), and studies withbuild_dependent_cohorts_cohort.Rshould rename it tobuild_dependent_cohorts.R. Helper code belongs in a subfolder such asinputs/cohorts/R/src/. The now-meaninglesswarnMissingargument isremoved (Source input builder scripts silently skips files with the wrong name #116).
Task Change Detection
CohortManifest$getManifestHash()now also covers each cohort's label,category, and tags, so renaming, recategorizing, or retagging a cohort reruns
tasks — tasks often select cohorts by that metadata. A cohort's definition
hash (
CohortDef$getSqlHash()) still covers only its SQL.ConceptSetManifest$getManifestHash(), consistent with the cohortversion: each concept set's normalized expression JSON plus its label,
category, and tags.
shouldRerunTask()now reruns a task when the concept set manifestchanges (Add another trigger for shouldRerunTask #125), e.g. merging two categories, or when
renv.lockchanges — the R version or any package's version, source, or remote SHA
(Task Change Detection Updates #118).
testStudyPipeline(),execStudyPipeline(), andtestStudyTask()gain aforceRerunargument (defaultFALSE) that bypasses change detection:TRUEreruns every task, and a vector of task file names reruns only those.Forced runs are still recorded in
task_run_history.csvand noted in thepipeline log.
exec/logs/task_run_history.csvgainsconcept_set_manifest_hashandrenv_lock_hashcolumns. Existing history rows have no recorded value forthem, and the cohort hash now includes metadata, so every task reruns once
after upgrading. Studies without a concept set manifest or
renv.lockhashto a stable sentinel and are not rerun on every run.
New Features
Parameterized Input Builder Scripts
sourceInputBuilderScripts()gainsconfigBlockandpipelineVersionarguments. Builder scripts run once per config block, with the current block
and pipeline version exposed as an
inputBuilderEnvobject so scripts canbuild execution settings (e.g. to resolve concept sets for SQL cohorts)
without hard-coding either value. The generated
main.RpassesdbIds.createInputBuilderEnv()builds the same object for interactive use (Parameterize config block and pipeline version in input builder scripts #109).execStudyPipeline()/testStudyPipeline()gainskipInputBuilders(default
TRUE). WithFALSE, the pipeline sources the builder scripts foreach config block right before generating that block's cohorts, so
database-specific builder changes are not overwritten by the next database.
By default the builders are still sourced from
main.R/test_main.R.Per-Config-Block Pipeline Execution
execStudyPipeline()/testStudyPipeline()now walk the config blocks oneat a time — input builders, then that block's cohort generation, then its
tasks — instead of generating every database's cohorts up front. A builder
that makes database-specific manifest changes is therefore no longer
overwritten by the next database's builders before its cohorts are built.
table rather than from the manifest's shared
staleflag, so a cohortalready generated in one database is not skipped in the next.
Initializing Into an Existing Directory
UlyssesStudy$initUlyssesRepo()now works in an existing empty directory,and gains an
allowExistingargument (defaultFALSE) to initialize into anon-empty one: only missing scaffold files are added, existing files are
never overwritten, and the required entries are merged into an existing
.gitignore. An existing git repository is left for the user to commit(Allow initUlyssesRepo() to initialize in an existing directory #148).
Fail Fast on Script and ATLAS Errors
sourceInputBuilderScripts()andsourceDisseminationScripts()stop at thefirst failing script instead of running the rest and reporting afterwards, so
a broken builder cannot leave a half-built manifest for the next script to
work from. Both gain
stopOnError(defaultTRUE);FALSErestores thecollect-and-report behavior (sourceDisseminationScripts and sourceInputBuilderScripts should fail by default instead of continuing silently #123).
$checkAtlasCohorts()/$updateAtlasCohorts()and their concept setcounterparts now abort when a definition fetch from ATLAS fails, rather than
skipping it silently and reporting the manifest as synced. Both gain
stopOnError(defaultTRUE);FALSEskips the failed fetches andcontinues.
Manifest Builders and Queries
$buildStratifiedCohorts()gainsstopIfExists(defaultTRUE), matchingthe other derived builders:
FALSEupdates already-registered strata inplace and creates any new ones, so a stratification can be re-run after
editing its strata.
$retrieveCohortCounts()returns counts only for cohorts registered in themanifest. Requested ids are intersected with the manifest, and an empty
result is returned without querying when none match, so leftover cohort ids
in a shared cohort table no longer appear in
cohortCounts.csv.cohortManifestSnapshot.csvwritten byrunPostProcessing()now uses thewide tag format — one column per tag key — so the provenance snapshot is
readable in a spreadsheet.
Bug Fixes
ExecutionSettingsnow accepts anyDatabaseConnectorConnection, includingthe DBI-backed connections used for DuckDB and SQLite, not only JDBC
connections (Allow DatabaseConnectorConnection in addition to DatabaseConnectorJdbcConnection in ExecutionSettings #150).
checksums are found regardless of the column-name case a DBMS returns
(Uppercase column name refs break across database platforms #149).
buildCompositeCohort()SQL no longer usesCOUNT(DISTINCT ...) OVER (...),which PostgreSQL and SQL Server reject, so composite cohorts now generate on
those platforms (createCompositeCohort.sql fails on PostgreSQL: DISTINCT is not implemented for window functions #155).
execStudyPipeline()/testStudyPipeline()now generate cohorts in everyconfig block passed to
configBlock. Previously cohorts were only generatedin the first block, so tasks for the other databases ran against cohort
tables that had not been built for that run.
stopIfExists = FALSEnowrewrites the definition at its registered file path instead of writing a
second file named after the new label and leaving the old one behind. The
derived builders refuse to move a registered cohort to a different path.
tabulateManifest()no longer errors on a tag whose value isnull; suchtags come back with an
NAvalue.$extractSourceCodes()no longer aborts in a non-interactive session.utils::menu()cannot run non-interactively, so the source-vocabularyprompt errored out whenever the method was called from a script rather than
the console. The suggestion is now offered only when
interactive();otherwise the
sourceVocabsargument is used as given.DatabaseConnector::existsTable(),and a failing check aborts naming the schema, table, and database instead of
being reported as "table missing" — a connection, permission, or identifier
problem no longer looks like an absent table. A failure to create a cohort
table is likewise an error rather than a danger message the run continued
past.
makeInputBuilderScript(type = "buildDependentCohorts")now writesbuild_dependent_cohorts.Rinstead ofbuild_dependent_cohorts_cohort.R, whichsourceInputBuilderScripts()never sourced.ATLAS builders are fully commented out, with instructions to uncomment each
step as needed (previously they created a blank load csv and connected to
ATLAS on every run); the Capr, SQL and dependent-cohort builders create their
manifest on first run; and the Capr builders no longer require Capr until
Capr code is added (Source input builder scripts silently skips files with the wrong name #116).
$updateAtlasCohorts()/$updateAtlasConceptSets()no longer error when themanifest has no ATLAS entries.
json/without registration are deleted as orphans, and the concept set template listed the wrong load csv columns (categoryis required;domain/sourceCodeare optional tags) and wrongly claimed dropped-in JSON files are auto-registered. Both templates now note thatstopIfExists = FALSEmatches rows by label (rename via$updateCohortLabel()/$updateConceptSetLabel()), and their headers use the generated file names.if/elseblock rather than two brace-lessifstatements.main.Rnow runsrenv::restore()by default (renv restore should always be uncommented in main #113).$buildCompositeCohort()documentation now describesminEventCountaccurately (the number of criteria cohorts a subject must belong to, default 1 = any of them), and a warning is raised when it is not supplied, recommending the number of criteria cohorts for an intersection (Update composite cohort description and default value #126).main.R(main.R issues #115):execStudyPipeline()is passed its requiredupdateType,runPostProcessing()is called withpipelineVersion/dbIdsinstead of nonexistent arguments, andsourceDisseminationScripts()uses the new pipeline version.test_main.Rgets the samerunPostProcessing()fix.date: todayinstead oflast-modified, and the results page setsecho: false/warning: false,so a rendered hub no longer shows chunk code or warnings.
docs/folder in thestudy repo tree;
initUlyssesRepo()does not create one.analysts to delete the builder scripts they do not use, which contradicted
the six-script requirement above (Source input builder scripts silently skips files with the wrong name #116). They now describe keeping all six
in place and what an unpopulated template does and does not do.