Repository navigation
v006 correction - #58
Merged
Merged
Conversation
Overwrite the registered JSON in place for a cohort/concept set added via addCaprCohort()/addCaprConceptSet(), keeping its ID and file path, refreshing the manifest hash, and (for cohorts) cascading 'stale' to derived dependents. Unchanged definitions are a no-op that leaves the file untouched. Also skips the known-broken getManifestStatus test (active_count miscount pending fix). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With stopIfExists = FALSE (mirroring addSqlCohort's parameter), adding a label that already exists updates the registered definition in place via updateCaprCohort()/updateCaprConceptSet() instead of erroring — same ID and file path, with category (and tags, when supplied) replacing the registered metadata. addCaprConceptSet also now raises a friendly duplicate-label error instead of a raw SQLite unique-index failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ethods importAtlasCohorts()/importAtlasConceptSets() gain updateExisting (default FALSE): when TRUE, already-registered ATLAS definitions are re-checked against ATLAS and changed ones updated in place via updateAtlasCohorts()/updateAtlasConceptSets(), so re-running the load script propagates ATLAS edits without a separate manual step. The generated study templates enable it by default. Bug fixes in the existing check/update methods: - checkAtlasConceptSets: referenced undefined cm_atlas_subset and atlas_id, and selected a nonexistent filePath column (crash on call) - updateAtlasConceptSets: UPDATE targeted nonexistent 'timestamp' column instead of updated_at - updateAtlasCohorts: never refreshed the in-memory manifest (old TODO) - all four methods: error handlers in tryCatch didn't skip failed fetches (stale/undefined def used afterwards), and early no-op exits fell through instead of returning Covered by new tests driving the import→update flow with a fake ATLAS connection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stale-regeneration flow was unreachable: cascadeStaleDownstream() marks derived dependents 'stale', but build_dependency_graph(), load_manifest_from_db(), and compute_dependency_hash() all filtered to status = 'active', so stale cohorts silently dropped out of the generation loop and could never be regenerated or re-activated — despite evaluate_cohort_skip_status() and the stale->active reset in generate_single_cohort() being built for exactly that path. All three now include 'stale' rows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- deleteCohort() now refuses when the cohort has active/stale derived dependents, listing them; a new cascade = TRUE flag soft-deletes the whole dependent subtree (deepest-first), including DBMS cleanup when dropFromDBMS = TRUE. Also fixes the missing early return when the id doesn't exist and refreshes the in-memory manifest after deletion (old TODO). - syncManifest() now cascades soft-deletion to derived dependents when a parent's file has gone missing (action 'cascade_deleted') — they could never generate without their parent. - topological_sort() distinguishes dangling parent references from genuine cycles: orphaned dependents previously crashed generateCohorts() with a misleading 'possible circular dependency' error. - findTransitiveDependents() extracted from cascadeStaleDownstream() and shared by all three call sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All eight template-based build methods (union, subsetTemporal, complement, composite, demographic, oprior, tprior, censor) gain stopIfExists (default TRUE, matching the add* methods). With FALSE, building under an existing label updates the registered derived cohort in place: same ID and file path, SQL re-rendered with the new parents and build parameters, depends_on/dependency_rule/hash replaced, the cohort marked 'stale' for regeneration, and stale cascaded to its own dependents. Previously the only way to change a derived cohort's parents was resetCohortManifest(scope = 'derived'), which wipes every derived cohort and assigns new IDs. The upsert path refuses to update source (circe/custom) cohorts and rejects parent sets that would create a dependency cycle (a cohort cannot depend on itself or its own dependents). Shared plumbing: resolve_derived_upsert() and upsert_derived_cohort() privates; builders now delegate registration to the latter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With stopIfExists = FALSE, re-registering an existing label updates the custom_derived cohort in place (same ID): the dependent cohort IDs (depends_on + dependency_rule parameter mapping) and the SQL file registration (path and content hash) are replaced, the cohort is marked 'stale' for regeneration, and staleness cascades to its dependents. The Picard write-contract validation and cycle guard apply on the update path too, and a file-path conflict with the upsert target itself is no longer treated as a conflict. Plain addSqlCohort() behavior unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Entries may now hold an integer vector of cohort IDs, not just a single ID: all IDs land in depends_on (so stale cascades and generation ordering see every parent), the dependency_rule stores the vector, and get_custom_derived_sql_params() flattens JSON-parsed values back to atomic vectors so SqlRender renders them comma-separated — suitable for IN (@PARAM) clauses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s-run sync The load csv is a transient one-time import mechanism, not part of the pipeline, so updates no longer flow through it: - importAtlasCohorts()/importAtlasConceptSets() lose updateExisting and now fail fast (before importing anything) when load rows are already registered — by atlasId or by label — with pointers to the update paths. - addAtlasCohort()/addAtlasConceptSet() gain stopIfExists (default TRUE): FALSE fetches the current ATLAS definition and updates the registered entry in place — same ID and file path, hash refreshed, category/tags/atlasId replaced, derived dependents marked stale (cohorts), no-op when unchanged. - The importAtlas templates now run updateAtlasCohorts()/updateAtlasConceptSets() unconditionally in a dedicated sync section, so re-sourcing the load script is the step that propagates ATLAS edits. Sync stays in input loading rather than pipeline execution so per-database runs never depend on ATLAS access and all databases execute identical frozen definitions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sourceInputBuilderScripts() (called by main.R before execStudyPipeline) re-sources the importAtlas builder scripts on every run. With the load csv now erroring on already-registered rows, an unguarded import would abort the script before the updateAtlas sync section. The templates now skip the one-time import when the csv has been deleted (the documented post-import step) and always reach the sync — so every main.R run fetches current ATLAS definitions before the pipeline executes. Guards are written brace-free because makeInputBuilderScript() renders templates through glue::glue(), which interprets curly braces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…port sourceInputBuilderScripts() swallowed builder-script errors: the per-file tryCatch printed an alert and continued, and the handlers assigned to handler-local copies of the errors list (missing <<-), so error_summary was always empty. main.R therefore proceeded to execStudyPipeline() even when input building failed, running the pipeline on an incomplete or stale manifest. Errors are now aggregated across all scripts (fixed scoping) and raised as a single abort listing every failure, so a failed input build can never be followed by a pipeline run. The importAtlas templates also now sync before importing, so even a stale load csv (which errors by design) cannot prevent the ATLAS sync from running. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e-update-capr-cohort
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ypes With stopIfExists = FALSE, a typo'd label could silently replace an unrelated cohort's definition — same ID, changed meaning, no error. Accidental collisions almost always cross a provenance boundary the manifest already records, so the upsert paths now verify identity: - Atlas (cohorts + concept sets): the registered atlasId tag must match the atlasId being passed; entries without an atlasId tag cannot be updated via the Atlas route. Conversely, registering the same atlasId under a second label is rejected as a duplicate import. - Capr: updateCaprCohort() requires the registered route tag 'capr' (an ATLAS cohort can no longer be clobbered by a Capr update); updateCaprConceptSet() refuses ATLAS-registered concept sets, and addCaprConceptSet() now records route = 'capr' for provenance parity with cohorts. - Derived builders: the existing cohort_type must match the type the builder creates (a union can only be updated by buildUnionCohort, etc.); intentional type changes require delete + rebuild. Intentional cross-boundary changes remain possible as explicit two-steps (retag with updateCohortTags()/updateConceptSetTags(), or delete and rebuild). Same-route same-type collisions remain visible via the 'updating in place' message and git review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Manifest Function Updates
…n Capr skill The picard-capr-cohorts skill now tells the agent to always pass stopIfExists = FALSE in addCaprCohort()/addCaprConceptSet() registrations, so definitions it writes can be revised and re-sourced later, and to ask the user for label, category, and tags as part of the Step 1 clarification before writing any cohort-building code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cm_test_add_sql_cohort registered the shared fixture at its real path in test_files/, so any test that hard-deletes a cohort (deleteCohort unlinks file_path) wiped the fixture from the working tree and silently skipped every test seeded through cm_test_seed_manifest_for_queries. Register a temp copy instead, restore the fixture, and fix a conceptSetIds/cohortIds typo in a tags test that the skip had been masking. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…kip getManifestStatus test validateManifest selected filePath instead of file_path, so the query errored, the tryCatch swallowed it, and getManifestStatus reported zero counts — the 'active_count miscount' the skipped test referenced. Fixing it unmasked cleanupMissing passing a nonexistent reason argument to deleteCohort/deleteConceptSet (now confirm = TRUE), and both cleanup methods missing return() on the nothing-to-clean early exit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r concept sets upsert_derived_cohort (addDependentCustomCohort and all build* methods) and addCaprConceptSet now clear registered tags when none are supplied, matching the addAtlasCohort/addCaprCohort upsert semantics. Plain addSqlCohort is unchanged: it has no label-keyed upsert, so there is no metadata-refresh path there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hods stopIfExists = FALSE now updates a registered custom SQL cohort in place: same ID, file path registration and hash refreshed, category/tags replaced wholesale (cleared when none supplied), derived dependents marked stale on definition change. Overwriting a non-custom cohort is rejected, mirroring the circe guard in addAtlasCohort/updateCaprCohort. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n Capr skill The picard-capr-cohorts skill now tells the agent to always pass stopIfExists = FALSE in addCaprCohort()/addCaprConceptSet() registrations, so definitions it writes can be revised and re-sourced later, and to ask the user for label, category, and tags as part of the Step 1 clarification before writing any cohort-building code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Make upsert behavior consistent across builder functions
Updates to agent instruction files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
forgot publish stuff