Skip to content

Release v008 - #164

Merged
mdlavallee92 merged 48 commits into
mainfrom
develop
Nov 18, 2026
Merged

mdlavallee92 merged 48 commits into
mainfrom
develop

Conversation

@mdlavallee92

Copy link
Copy Markdown
Collaborator

picard 0.0.8

Breaking Changes

  • sourceInputBuilderScripts() now requires all six builder scripts and aborts
    before sourcing anything if one is missing or if inputs/cohorts/R/ or
    inputs/conceptSets/R/ contains any other .R file (previously these were
    skipped silently). Studies that deleted unused builders should recreate them
    with makeInputBuilderScript(), and studies with
    build_dependent_cohorts_cohort.R should rename it to
    build_dependent_cohorts.R. Helper code belongs in a subfolder such as
    inputs/cohorts/R/src/. The now-meaningless warnMissing argument is
    removed (Source input builder scripts silently skips files with the wrong name #116).

Task Change Detection

  • CohortManifest$getManifestHash() now also covers each cohort's label,
    category, and tags, so renaming, recategorizing, or retagging a cohort reruns
    tasks — tasks often select cohorts by that metadata. A cohort's definition
    hash (CohortDef$getSqlHash()) still covers only its SQL.
  • New ConceptSetManifest$getManifestHash(), consistent with the cohort
    version: each concept set's normalized expression JSON plus its label,
    category, and tags.
  • shouldRerunTask() now reruns a task when the concept set manifest
    changes (Add another trigger for shouldRerunTask #125), e.g. merging two categories, or when renv.lock
    changes — the R version or any package's version, source, or remote SHA
    (Task Change Detection Updates #118).
  • testStudyPipeline(), execStudyPipeline(), and testStudyTask() gain a
    forceRerun argument (default FALSE) that bypasses change detection:
    TRUE reruns every task, and a vector of task file names reruns only those.
    Forced runs are still recorded in task_run_history.csv and noted in the
    pipeline log.
  • exec/logs/task_run_history.csv gains concept_set_manifest_hash and
    renv_lock_hash columns. Existing history rows have no recorded value for
    them, and the cohort hash now includes metadata, so every task reruns once
    after upgrading. Studies without a concept set manifest or renv.lock hash
    to a stable sentinel and are not rerun on every run.

New Features

Parameterized Input Builder Scripts

  • sourceInputBuilderScripts() gains configBlock and pipelineVersion
    arguments. Builder scripts run once per config block, with the current block
    and pipeline version exposed as an inputBuilderEnv object so scripts can
    build execution settings (e.g. to resolve concept sets for SQL cohorts)
    without hard-coding either value. The generated main.R passes dbIds.
    createInputBuilderEnv() builds the same object for interactive use (Parameterize config block and pipeline version in input builder scripts #109).
  • execStudyPipeline() / testStudyPipeline() gain skipInputBuilders
    (default TRUE). With FALSE, the pipeline sources the builder scripts for
    each config block right before generating that block's cohorts, so
    database-specific builder changes are not overwritten by the next database.
    By default the builders are still sourced from main.R / test_main.R.

Per-Config-Block Pipeline Execution

  • execStudyPipeline() / testStudyPipeline() now walk the config blocks one
    at a time — input builders, then that block's cohort generation, then its
    tasks — instead of generating every database's cohorts up front. A builder
    that makes database-specific manifest changes is therefore no longer
    overwritten by the next database's builders before its cohorts are built.
  • Whether a cohort needs regenerating is decided from that database's checksum
    table rather than from the manifest's shared stale flag, so a cohort
    already generated in one database is not skipped in the next.

Initializing Into an Existing Directory

  • UlyssesStudy$initUlyssesRepo() now works in an existing empty directory,
    and gains an allowExisting argument (default FALSE) to initialize into a
    non-empty one: only missing scaffold files are added, existing files are
    never overwritten, and the required entries are merged into an existing
    .gitignore. An existing git repository is left for the user to commit
    (Allow initUlyssesRepo() to initialize in an existing directory #148).

Fail Fast on Script and ATLAS Errors

  • sourceInputBuilderScripts() and sourceDisseminationScripts() stop at the
    first failing script instead of running the rest and reporting afterwards, so
    a broken builder cannot leave a half-built manifest for the next script to
    work from. Both gain stopOnError (default TRUE); FALSE restores the
    collect-and-report behavior (sourceDisseminationScripts and sourceInputBuilderScripts should fail by default instead of continuing silently #123).
  • $checkAtlasCohorts() / $updateAtlasCohorts() and their concept set
    counterparts now abort when a definition fetch from ATLAS fails, rather than
    skipping it silently and reporting the manifest as synced. Both gain
    stopOnError (default TRUE); FALSE skips the failed fetches and
    continues.

Manifest Builders and Queries

  • $buildStratifiedCohorts() gains stopIfExists (default TRUE), matching
    the other derived builders: FALSE updates already-registered strata in
    place and creates any new ones, so a stratification can be re-run after
    editing its strata.
  • $retrieveCohortCounts() returns counts only for cohorts registered in the
    manifest. Requested ids are intersected with the manifest, and an empty
    result is returned without querying when none match, so leftover cohort ids
    in a shared cohort table no longer appear in cohortCounts.csv.
  • The cohortManifestSnapshot.csv written by runPostProcessing() now uses the
    wide tag format — one column per tag key — so the provenance snapshot is
    readable in a spreadsheet.

Bug Fixes

  • ExecutionSettings now accepts any DatabaseConnectorConnection, including
    the DBI-backed connections used for DuckDB and SQLite, not only JDBC
    connections (Allow DatabaseConnectorConnection in addition to DatabaseConnectorJdbcConnection in ExecutionSettings #150).
  • Cohort generation reads checksum-table query results by position, so
    checksums are found regardless of the column-name case a DBMS returns
    (Uppercase column name refs break across database platforms #149).
  • buildCompositeCohort() SQL no longer uses COUNT(DISTINCT ...) OVER (...),
    which PostgreSQL and SQL Server reject, so composite cohorts now generate on
    those platforms (createCompositeCohort.sql fails on PostgreSQL: DISTINCT is not implemented for window functions #155).
  • execStudyPipeline() / testStudyPipeline() now generate cohorts in every
    config block passed to configBlock. Previously cohorts were only generated
    in the first block, so tasks for the other databases ran against cohort
    tables that had not been built for that run.
  • Renaming a derived cohort through its builder with stopIfExists = FALSE now
    rewrites the definition at its registered file path instead of writing a
    second file named after the new label and leaving the old one behind. The
    derived builders refuse to move a registered cohort to a different path.
  • tabulateManifest() no longer errors on a tag whose value is null; such
    tags come back with an NA value.
  • $extractSourceCodes() no longer aborts in a non-interactive session.
    utils::menu() cannot run non-interactively, so the source-vocabulary
    prompt errored out whenever the method was called from a script rather than
    the console. The suggestion is now offered only when interactive();
    otherwise the sourceVocabs argument is used as given.
  • Cohort table existence is now checked with DatabaseConnector::existsTable(),
    and a failing check aborts naming the schema, table, and database instead of
    being reported as "table missing" — a connection, permission, or identifier
    problem no longer looks like an absent table. A failure to create a cohort
    table is likewise an error rather than a danger message the run continued
    past.
  • makeInputBuilderScript(type = "buildDependentCohorts") now writes
    build_dependent_cohorts.R instead of build_dependent_cohorts_cohort.R, which
    sourceInputBuilderScripts() never sourced.
  • The input builder templates now run without error when unpopulated: the
    ATLAS builders are fully commented out, with instructions to uncomment each
    step as needed (previously they created a blank load csv and connected to
    ATLAS on every run); the Capr, SQL and dependent-cohort builders create their
    manifest on first run; and the Capr builders no longer require Capr until
    Capr code is added (Source input builder scripts silently skips files with the wrong name #116).
  • $updateAtlasCohorts() / $updateAtlasConceptSets() no longer error when the
    manifest has no ATLAS entries.
  • Corrected the ATLAS import builder templates (Update ATLAS Concept Set Import Template to Reflect Recommended Workflow #112): the import step now comes before the sync step, both templates note that JSON files dropped into json/ without registration are deleted as orphans, and the concept set template listed the wrong load csv columns (category is required; domain/sourceCode are optional tags) and wrongly claimed dropped-in JSON files are auto-registered. Both templates now note that stopIfExists = FALSE matches rows by label (rename via $updateCohortLabel()/$updateConceptSetLabel()), and their headers use the generated file names.
  • The ATLAS concept set builder template's one-time import is now a single
    if/else block rather than two brace-less if statements.
  • The generated main.R now runs renv::restore() by default (renv restore should always be uncommented in main #113).
  • $buildCompositeCohort() documentation now describes minEventCount accurately (the number of criteria cohorts a subject must belong to, default 1 = any of them), and a warning is raised when it is not supplied, recommending the number of criteria cohorts for an intersection (Update composite cohort description and default value #126).
  • Fixed the generated main.R (main.R issues #115): execStudyPipeline() is passed its required updateType, runPostProcessing() is called with pipelineVersion/dbIds instead of nonexistent arguments, and sourceDisseminationScripts() uses the new pipeline version. test_main.R gets the same runPostProcessing() fix.
  • The Study Hub templates date themselves with date: today instead of
    last-modified, and the results page sets echo: false / warning: false,
    so a rendered hub no longer shows chunk code or warnings.
  • The repository structure vignette no longer lists a docs/ folder in the
    study repo tree; initUlyssesRepo() does not create one.
  • The Loading Inputs and Running the Pipeline vignettes no longer tell
    analysts to delete the builder scripts they do not use, which contradicted
    the six-script requirement above (Source input builder scripts silently skips files with the wrong name #116). They now describe keeping all six
    in place and what an unpopulated template does and does not do.

Katy Sadowski and others added 30 commits September 30, 2026 19:11
…ts (#109)

sourceInputBuilderScripts() now accepts configBlock and pipelineVersion and
exposes them to builder scripts via an inputBuilderEnv object, so scripts can
create execution settings without hard-coding either value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orts script name

sourceInputBuilderScripts() now loops over configBlock like execStudyPipeline(),
exposing the current block via inputBuilderEnv. makeInputBuilderScript() now
writes build_dependent_cohorts.R (the name that is actually sourced) and a
warning flags the legacy build_dependent_cohorts_cohort.R name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
execute_pipeline() previously built execution settings and generated cohorts
for configBlock[1] only, while tasks ran for every block. Cohort generation now
loops over all config blocks before tasks run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sourceInputBuilderScripts() now requires all six builder scripts and aborts
before sourcing if one is missing or an unrecognized .R file is present,
instead of skipping silently. The builder templates now run without error
when unpopulated: manifests are created on first run, ATLAS is only
contacted when there is ATLAS work to do, and Capr is not loaded until used.
updateAtlasCohorts()/updateAtlasConceptSets() no longer error on a manifest
without ATLAS entries. The warnMissing argument is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ersion

The config.yml version is read before execStudyPipeline() increments it, and
a test run would silently fall back to it and target the production cohort
table. Default to "prod" like createExecutionSettingsFromConfig() and require
test runs to pass their namespace explicitly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conditional ATLAS logic in the import templates could skip steps silently
during development. Every step is now commented out with instructions to
uncomment as needed, so an unpopulated template is a no-op and a populated
one fails loudly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Imported definitions are saved as <atlasId>_<ATLAS name> in snake_case, not
by label, and file_name is not a load file column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The load csvs are kept in the study repo as the record of which ATLAS
cohorts/concept sets a study uses. Keep the stopIfExists = TRUE default but
stop describing the csv as a throwaway in templates, method docs, comments,
error hints and vignettes, and explain stopIfExists = FALSE as the
alternative for re-importing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sourceInputBuilderScripts() and sourceDisseminationScripts() gain
stopOnError (default TRUE): the first failing script stops the run, with the
script's error as the cause. stopOnError = FALSE warns and continues,
returning errors in error_summary. Also fixes sourceDisseminationScripts()
never recording errors (local assignment in the handler) and removes the
outer tryCatch that could swallow failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
checkAtlasCohorts()/updateAtlasCohorts() and the concept set equivalents
warned and skipped entries whose fetch failed, so a builder script could
finish with the manifest silently out of date. A failed fetch now aborts,
naming the ATLAS id and label, with the original error as the cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ches

checkAtlasCohorts()/updateAtlasCohorts() and the concept set equivalents
gain stopOnError (default TRUE). With FALSE, an entry whose fetch fails is
skipped with a warning and the rest are still synced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he pipeline

Builder scripts can make database-specific manifest changes (e.g. concept ids
in custom SQL), which were overwritten by later databases' builders before
cohorts were generated. execStudyPipeline()/testStudyPipeline() now source the
builders for each config block, then generate that block's cohorts and run its
tasks before moving on; main.R no longer sources builders itself. Pre-flight
warns instead of failing when the cohort manifest does not exist yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tale flag

Builders can make database-specific changes, so the shared manifest flips
definitions as the pipeline moves through databases and marked derived
cohorts stale on every pass, forcing regeneration in every database. The
derived-cohort checksum now also covers the cohort's own rendered SQL, and
the checksum stored in each database's work schema alone decides whether to
skip. Stale is cleared when a cohort is skipped as up to date.

Note: derived cohorts regenerate once per database after upgrading, since
their stored checksums use the old formula.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Rework ATLAS import builder templates around keeping the load csv and
  importing with stopIfExists = FALSE; reorder sections and fix stale notes (#112)
- Run renv::restore() by default in generated main.R (#113)
- Fix wrong/missing params in main.R and test_main.R, and builder script
  names in main.R comments (#115)
- buildCompositeCohort(): clarify minEventCount and default it to the number
  of criteria cohorts (#126)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert the default change for #126 to avoid a breaking change. minEventCount
stays 1L, and a warning recommends setting it to the number of criteria cohorts
when it is not supplied.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ort template

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds of pipeline results

- Add ConceptSetManifest$getManifestHash() (includes label/category/tags)
  and record concept_set_manifest_hash in task_run_history (#125)
- Hash renv.lock package/R versions into renv_lock_hash (#118)
- validateStudyTask() fails tasks that read files from pipeline results
  and warns on other untracked data-file reads (#118)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Changes to files a task reads are not tracked yet, so a task with any
data-file read is always rerun. Tracking those files is left as a TODO.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
testStudyPipeline(), execStudyPipeline(), and testStudyTask() accept
forceRerun = TRUE (all tasks) or a vector of task file names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Handling of tasks that read data files will be addressed separately.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-item definition hashes (CohortDef$getSqlHash(), ConceptSetDef$getHash())
cover only the SQL/JSON. New getTaskRerunHash() on both manifests combines
each definition hash with label, category, and tags, since tasks select
cohorts and concept sets by that metadata. CohortManifest$getManifestHash()
is kept as a deprecated alias. NEWS entries move to a development-version
section since 0.0.7 is released.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CohortManifest$getManifestHash() keeps its name and now covers label,
category, and tags. ConceptSetManifest$getManifestHash() matches it and
hashes normalized expression JSON so reformatting does not rerun tasks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…checksum

Pins the property that lets the per-database checksum alone decide cohort
regeneration: hashes are recorded in topological order, so editing a base
cohort's file changes every descendant's dependency hash while unrelated
cohorts keep theirs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… default

execStudyPipeline()/testStudyPipeline() gain skipInputBuilders (default TRUE),
so the pipeline no longer sources the input builder scripts unless asked,
preserving the previous behavior. The generated main.R and test_main.R source
the builders before running the pipeline again. Pre-flight only treats a
missing cohort manifest as a warning when the pipeline will run the builders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parameterize input builders per DB; fail loudly on missing builders and script errors
mdlavallee92 and others added 18 commits October 9, 2026 09:53
…cksum columns

- initUlyssesRepo() works in an existing empty directory and gains
  allowExisting to add only missing scaffold files to a non-empty one (#148)
- Read checksum-table query results by position instead of uppercase
  column names (#149)
- ExecutionSettings accepts any DatabaseConnectorConnection, e.g. DuckDB (#150)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PostgreSQL and SQL Server reject DISTINCT in window functions, so compute
the per-subject distinct cohort count in a grouped subquery instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Close some minor bugs in issue log
Fix cross-DB compatibility bugs, allow init in existing directory
Non-interactive guard for extractSourceCodes() and vignette corrections
for the six required input builder scripts.
@mdlavallee92
mdlavallee92 merged commit 87324d7 into main Nov 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants