Skip to content

v006 correction - #58

Merged
mdlavallee92 merged 39 commits into
mainfrom
develop
Jul 20, 2026
Merged

mdlavallee92 merged 39 commits into
mainfrom
develop

Conversation

@mdlavallee92

@mdlavallee92 mdlavallee92 commented Jul 18, 2026 •

Copy link
Copy Markdown
Collaborator

forgot publish stuff

mdlavallee92 and others added 30 commits July 16, 2026 11:46
Overwrite the registered JSON in place for a cohort/concept set added
via addCaprCohort()/addCaprConceptSet(), keeping its ID and file path,
refreshing the manifest hash, and (for cohorts) cascading 'stale' to
derived dependents. Unchanged definitions are a no-op that leaves the
file untouched.

Also skips the known-broken getManifestStatus test (active_count
miscount pending fix).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With stopIfExists = FALSE (mirroring addSqlCohort's parameter), adding a
label that already exists updates the registered definition in place via
updateCaprCohort()/updateCaprConceptSet() instead of erroring — same ID
and file path, with category (and tags, when supplied) replacing the
registered metadata. addCaprConceptSet also now raises a friendly
duplicate-label error instead of a raw SQLite unique-index failure.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ethods

importAtlasCohorts()/importAtlasConceptSets() gain updateExisting
(default FALSE): when TRUE, already-registered ATLAS definitions are
re-checked against ATLAS and changed ones updated in place via
updateAtlasCohorts()/updateAtlasConceptSets(), so re-running the load
script propagates ATLAS edits without a separate manual step. The
generated study templates enable it by default.

Bug fixes in the existing check/update methods:
- checkAtlasConceptSets: referenced undefined cm_atlas_subset and
  atlas_id, and selected a nonexistent filePath column (crash on call)
- updateAtlasConceptSets: UPDATE targeted nonexistent 'timestamp'
  column instead of updated_at
- updateAtlasCohorts: never refreshed the in-memory manifest (old TODO)
- all four methods: error handlers in tryCatch didn't skip failed
  fetches (stale/undefined def used afterwards), and early no-op exits
  fell through instead of returning

Covered by new tests driving the import→update flow with a fake ATLAS
connection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stale-regeneration flow was unreachable: cascadeStaleDownstream()
marks derived dependents 'stale', but build_dependency_graph(),
load_manifest_from_db(), and compute_dependency_hash() all filtered to
status = 'active', so stale cohorts silently dropped out of the
generation loop and could never be regenerated or re-activated —
despite evaluate_cohort_skip_status() and the stale->active reset in
generate_single_cohort() being built for exactly that path. All three
now include 'stale' rows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- deleteCohort() now refuses when the cohort has active/stale derived
  dependents, listing them; a new cascade = TRUE flag soft-deletes the
  whole dependent subtree (deepest-first), including DBMS cleanup when
  dropFromDBMS = TRUE. Also fixes the missing early return when the id
  doesn't exist and refreshes the in-memory manifest after deletion
  (old TODO).
- syncManifest() now cascades soft-deletion to derived dependents when
  a parent's file has gone missing (action 'cascade_deleted') — they
  could never generate without their parent.
- topological_sort() distinguishes dangling parent references from
  genuine cycles: orphaned dependents previously crashed generateCohorts()
  with a misleading 'possible circular dependency' error.
- findTransitiveDependents() extracted from cascadeStaleDownstream()
  and shared by all three call sites.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All eight template-based build methods (union, subsetTemporal,
complement, composite, demographic, oprior, tprior, censor) gain
stopIfExists (default TRUE, matching the add* methods). With FALSE,
building under an existing label updates the registered derived cohort
in place: same ID and file path, SQL re-rendered with the new parents
and build parameters, depends_on/dependency_rule/hash replaced, the
cohort marked 'stale' for regeneration, and stale cascaded to its own
dependents. Previously the only way to change a derived cohort's
parents was resetCohortManifest(scope = 'derived'), which wipes every
derived cohort and assigns new IDs.

The upsert path refuses to update source (circe/custom) cohorts and
rejects parent sets that would create a dependency cycle (a cohort
cannot depend on itself or its own dependents).

Shared plumbing: resolve_derived_upsert() and upsert_derived_cohort()
privates; builders now delegate registration to the latter.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With stopIfExists = FALSE, re-registering an existing label updates the
custom_derived cohort in place (same ID): the dependent cohort IDs
(depends_on + dependency_rule parameter mapping) and the SQL file
registration (path and content hash) are replaced, the cohort is marked
'stale' for regeneration, and staleness cascades to its dependents. The
Picard write-contract validation and cycle guard apply on the update
path too, and a file-path conflict with the upsert target itself is no
longer treated as a conflict. Plain addSqlCohort() behavior unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Entries may now hold an integer vector of cohort IDs, not just a single
ID: all IDs land in depends_on (so stale cascades and generation
ordering see every parent), the dependency_rule stores the vector, and
get_custom_derived_sql_params() flattens JSON-parsed values back to
atomic vectors so SqlRender renders them comma-separated — suitable for
IN (@PARAM) clauses.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s-run sync

The load csv is a transient one-time import mechanism, not part of the
pipeline, so updates no longer flow through it:

- importAtlasCohorts()/importAtlasConceptSets() lose updateExisting and
  now fail fast (before importing anything) when load rows are already
  registered — by atlasId or by label — with pointers to the update
  paths.
- addAtlasCohort()/addAtlasConceptSet() gain stopIfExists (default
  TRUE): FALSE fetches the current ATLAS definition and updates the
  registered entry in place — same ID and file path, hash refreshed,
  category/tags/atlasId replaced, derived dependents marked stale
  (cohorts), no-op when unchanged.
- The importAtlas templates now run
  updateAtlasCohorts()/updateAtlasConceptSets() unconditionally in a
  dedicated sync section, so re-sourcing the load script is the step
  that propagates ATLAS edits. Sync stays in input loading rather than
  pipeline execution so per-database runs never depend on ATLAS access
  and all databases execute identical frozen definitions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sourceInputBuilderScripts() (called by main.R before execStudyPipeline)
re-sources the importAtlas builder scripts on every run. With the load
csv now erroring on already-registered rows, an unguarded import would
abort the script before the updateAtlas sync section. The templates now
skip the one-time import when the csv has been deleted (the documented
post-import step) and always reach the sync — so every main.R run
fetches current ATLAS definitions before the pipeline executes.
Guards are written brace-free because makeInputBuilderScript() renders
templates through glue::glue(), which interprets curly braces.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…port

sourceInputBuilderScripts() swallowed builder-script errors: the per-file
tryCatch printed an alert and continued, and the handlers assigned to
handler-local copies of the errors list (missing <<-), so error_summary
was always empty. main.R therefore proceeded to execStudyPipeline() even
when input building failed, running the pipeline on an incomplete or
stale manifest. Errors are now aggregated across all scripts (fixed
scoping) and raised as a single abort listing every failure, so a failed
input build can never be followed by a pipeline run.

The importAtlas templates also now sync before importing, so even a
stale load csv (which errors by design) cannot prevent the ATLAS sync
from running.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ypes

With stopIfExists = FALSE, a typo'd label could silently replace an
unrelated cohort's definition — same ID, changed meaning, no error.
Accidental collisions almost always cross a provenance boundary the
manifest already records, so the upsert paths now verify identity:

- Atlas (cohorts + concept sets): the registered atlasId tag must match
  the atlasId being passed; entries without an atlasId tag cannot be
  updated via the Atlas route. Conversely, registering the same atlasId
  under a second label is rejected as a duplicate import.
- Capr: updateCaprCohort() requires the registered route tag 'capr'
  (an ATLAS cohort can no longer be clobbered by a Capr update);
  updateCaprConceptSet() refuses ATLAS-registered concept sets, and
  addCaprConceptSet() now records route = 'capr' for provenance parity
  with cohorts.
- Derived builders: the existing cohort_type must match the type the
  builder creates (a union can only be updated by buildUnionCohort,
  etc.); intentional type changes require delete + rebuild.

Intentional cross-boundary changes remain possible as explicit
two-steps (retag with updateCohortTags()/updateConceptSetTags(), or
delete and rebuild). Same-route same-type collisions remain visible
via the 'updating in place' message and git review.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n Capr skill

The picard-capr-cohorts skill now tells the agent to always pass
stopIfExists = FALSE in addCaprCohort()/addCaprConceptSet() registrations,
so definitions it writes can be revised and re-sourced later, and to ask
the user for label, category, and tags as part of the Step 1 clarification
before writing any cohort-building code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cm_test_add_sql_cohort registered the shared fixture at its real path in
test_files/, so any test that hard-deletes a cohort (deleteCohort unlinks
file_path) wiped the fixture from the working tree and silently skipped
every test seeded through cm_test_seed_manifest_for_queries. Register a
temp copy instead, restore the fixture, and fix a conceptSetIds/cohortIds
typo in a tags test that the skip had been masking.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…kip getManifestStatus test

validateManifest selected filePath instead of file_path, so the query
errored, the tryCatch swallowed it, and getManifestStatus reported zero
counts — the 'active_count miscount' the skipped test referenced. Fixing
it unmasked cleanupMissing passing a nonexistent reason argument to
deleteCohort/deleteConceptSet (now confirm = TRUE), and both cleanup
methods missing return() on the nothing-to-clean early exit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r concept sets

upsert_derived_cohort (addDependentCustomCohort and all build* methods)
and addCaprConceptSet now clear registered tags when none are supplied,
matching the addAtlasCohort/addCaprCohort upsert semantics. Plain
addSqlCohort is unchanged: it has no label-keyed upsert, so there is no
metadata-refresh path there.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hods

stopIfExists = FALSE now updates a registered custom SQL cohort in place:
same ID, file path registration and hash refreshed, category/tags replaced
wholesale (cleared when none supplied), derived dependents marked stale on
definition change. Overwriting a non-custom cohort is rejected, mirroring
the circe guard in addAtlasCohort/updateCaprCohort.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Katy Sadowski and others added 9 commits July 18, 2026 21:44
…n Capr skill

The picard-capr-cohorts skill now tells the agent to always pass
stopIfExists = FALSE in addCaprCohort()/addCaprConceptSet() registrations,
so definitions it writes can be revised and re-sourced later, and to ask
the user for label, category, and tags as part of the Step 1 clarification
before writing any cohort-building code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Make upsert behavior consistent across builder functions
Updates to agent instruction files
@mdlavallee92
mdlavallee92 merged commit 93c02bf into main Jul 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants