Skip to content

v007 - #100

Merged
mdlavallee92 merged 111 commits into
mainfrom
develop
Sep 9, 2026
Merged

v007#100
mdlavallee92 merged 111 commits into
mainfrom
develop

Conversation

@mdlavallee92

Copy link
Copy Markdown
Collaborator

picard 0.0.7

New Features

Code State Escape Hatches

  • Scoped escape hatches for the "Code state" pre-flight check, for study repos where files under inputs/ churn without a deliberate edit. Both default to off, so unchanged studies get exactly the previous behavior.
    • execStudyPipeline(ignoreUncommittedPaths = ...) tolerates uncommitted changes confined to the listed repo-relative paths; changes anywhere else (notably analysis/) still fail the check. The list can be set once per study as ignoreUncommittedPaths in the default: block of config.yml, with the argument overriding it for a single run.
    • execStudyPipeline(skipCodeStateCheck = TRUE) skips the check entirely as a last resort. Deliberately not settable from config.yml.
    • Neither hatch is silent: Code state is reported as a warning rather than a pass, a banner names every ignored file, and the pipeline log records the code state and commit SHA.
    • exec/logs/task_run_history.csv gains commit_sha and code_state columns (clean, dirty-ignored, unverified-skipped, unverified-test-mode, unrecorded) so the audit trail never implies a clean tree when the tree was not clean. Existing history files are read and back-filled as unrecorded.

Tabulate and View Manifest

  • Added viewManifest() convenience method for both CohortManifest and ConceptSetManifest to interactively explore manifest metadata in RStudio viewer with streamlined columns (id, label, category, tags, file_path).
  • Enhanced tabulateManifest() with tags_format parameter supporting three formats: "nested" (default, parsed tags as tibble with tag_name/tag_value), "json" (raw JSON string for backward compatibility), and "wide" (expanded tags as individual columns for spreadsheet-like analysis).

Building a disseminationEnv Interactively

Study Metadata and Publishing

  • Added the optional studyDescription field to makeStudyMeta(). When supplied, it is inserted into the generated README; when omitted, the existing description placeholder is retained.
  • Added publishStudyHubPosit() to render and publish a Study Hub through Quarto's public Posit Connect publishing API.

Bug Fixes

  • Fixed several problems with the 'stale' cohort status (see Issue Various issues with "stale" status #74). 'stale' now consistently means "registered, but needs regenerating", rather than removing a cohort from the study:
    • Re-running a derived cohort builder (e.g. addDependentCustomCohort(), buildUnionCohort()) with stopIfExists = FALSE and unchanged inputs no longer marks the cohort — and everything downstream of it — stale. Staleness is now cascaded only on a real definition change (content hash, file path, parents, or build rule), and a metadata-only edit no longer forces regeneration.
    • Stale cohorts are once again returned by tabulateManifest(filter = "active") and therefore by all query...() methods, so they can be queried by label/tag/category and used as parents for new derived cohorts.
    • generateCohorts() now lists stale cohorts in its confirmation prompt (they are exactly the cohorts that need generating) and flags them as such.
    • print() and getManifestStatus() count stale cohorts as registered and break them out separately; getManifestStatus() gains stale_count.
    • Fixed viewManifest() erroring on an undefined tags_format argument; it now also shows the status column.
  • Manifest tag parsing no longer assumes the tags column holds a JSON string (see Issue Robustly handle non-string tags in manifest parsing #78). The query*ByTagName() methods default to tags_format = "nested", which returns tags as a tag_name/tag_value tibble, so internal callers that re-parsed that column failed with Argument 'txt' must be a JSON string, URL or file. This broke importAtlasCohorts()/importAtlasConceptSets() (and therefore sourceInputBuilderScripts()), as well as listAllUniqueTags() and getTagValuesSummary(). Tag parsing now goes through a shared helper that accepts JSON strings, nested tibbles, and already-parsed lists, and treats missing/empty/malformed tags as no tags.
  • add $queryConceptSetsByCategory() it was missing
  • Pipeline pre-flight checks were impossible to pass (Impossible to pass execStudyPipeline checks #84):
    • The manifest metadata writers issued an UPDATE even when every value already matched the stored row, bumping updated_at and rewriting cohortManifest.sqlite / conceptSetManifest.sqlite on every metadata refresh and every ATLAS re-import. Unchanged assignments are now dropped, so a re-run leaves the manifest files byte-identical and validateCodeState() no longer sees uncommitted changes. cascadeStaleDownstream() likewise skips rows that are already stale.
    • validateEnvironment() always reported "Environment drift detected!" because it tested renv::status() for NULL, and renv::status() always returns a list. It now reads the synchronized flag and compares the library and lockfile package records, and treats source-only mismatches as a warning rather than a blocker.
    • runPreflightChecks() ran renv::snapshot() immediately after validating that the working tree was clean, so the check could dirty the tree it had just approved and leave the next run failing its code-state check. The lockfile is now recorded rather than refreshed; keeping renv.lock current is the analyst's call, via snapshotEnvironment().
  • add stopIfExists option to ATLAS csv load builders (see Issue Add stopIfExists to ATLAS csv loaders #65)
  • Custom dependent cohort builder is now consistent with the other dependent cohort builders - the rendered SQL query is written to the file system, and it accepts cohort objects as inputs instead of IDs.
  • Custom dependent cohort builder now also allows user to specify non-cohort-ID params, so the rendered SQL query can be generated in a single function call (see Issue Allow user to specify non-cohort ID params in custom dependent SQL builder #66)
  • Builder console messages clarified (see Issue Update builder messages to indicate when metadata has been updated #70)
  • dplyr::select(-.data$tags_list) in the tag-expansion helpers replaced with select(-"tags_list"); use of .data in tidyselect expressions was deprecated in tidyselect 1.2.0 and would eventually have errored.
  • on resetManifest turn archive default to FALSE
  • Fixed two dissemination bugs (see Issue Dissemination bugs #87):
    • prepareDisseminationData() asserted that four concatenated logical flags had length 1, so it aborted on every call — including all-default calls — and the formatting step could never run.
    • The dissemination script template checked merged results for a database_id column, but importAndBind() labels them with camelCase databaseId, so the check was always false. The later references to database_id are correct, as they operate on data already passed through cleanColumnNames().
  • standardizeDataTypes() no longer discards columns it cannot coerce. The default "_id$" rule matched the character database_id column and as.integer() silently turned it into NA, since failed coercion raises a warning rather than an error. A conversion that would turn a non-missing value into NA is now skipped and reported.
  • Refactor query...Manifest methods to use $tabulateManifest() internally to be more consistent.
  • Show current manifest id after import (see Issue Console print for initial CS manifest load is missing IDs #68)
  • remove function forced tags route = atlas or route = capr (See Issue Move Capr/ATLAS/etc. tags to a column in the manifest #67)
  • in $grabConceptInfoFromSet() change input to conceptSetRef which accepts either a manifest id or a label (See issue Add an option to get concept info by label to conceptSetManifest$grabConceptInfoFromSet() #56)
  • Add task prefix to the importAndBind() save to avoid similar output names between tasks.
  • Concept sets fetched from ATLAS came back with their concepts in a nondeterministic order, so an unchanged definition serialized to different JSON — and a different content hash — on every fetch, dirtying the study repo's git status (part of Impossible to pass execStudyPipeline checks #84). Concept set expressions are now canonicalized before serialization: items are ordered by CONCEPT_ID (ties broken by vocabulary, code and inclusion flags) and circe ConceptSets by their id. Ordering is left untouched when any concept set lacks a usable id, so CodesetId references cannot break.
  • Add method to ExecutionSettings to reviewConnectionDetails() pulls the connectionDetails used.
  • Modify getServerCredentials() to not evaluate full yml file, only pull the specific block and evaluate (See issue Can't Run Pipeline in Background Job When ATLAS Creds Stored in keyring #30, Add useKeyring as pipeline input var #33).
  • Fix makeDisseminationScript() change the glue brackets to carrots
  • Fix bug in $updateAtlasCohorts()/$updateAtlasConceptSets() to use tags in json mode after tab format change.
  • Regenerated inst/agent/05-manifest-management.md from vignettes/manifest_management.Rmd. It had not been synced since viewManifest() and the tabulateManifest(tags_format = ...) options were added, so agents in generated study repos were reading stale manifest guidance.
  • execStudyPipeline()/testStudyPipeline() no longer go silent during a run (see Issue Make execStudyPipeline and testStudyPipeline output more verbose #85). execute_pipeline() used sink(type = "output") to capture console output into the pipeline log, which redirected every cli info bullet and error away from the console for the entire run — nothing showed up live, including the detail of a task's actual error. The sink is removed entirely: console output now streams normally, and the log file is instead written explicitly with high-level milestones (config block/task start, success, failure) plus a full error detail block (class, call, message) on failure. Separately, execute_task() was calling stop() from inside its per-expression error handler, which skipped the subsequent recordTaskExecution(..., errorMessage = ...) call entirely — so task_run_history.csv never recorded the real error message for a failed task, only whatever generic message the caller happened to re-throw. The detailed message is now recorded correctly.
  • Improved cohort-generation reporting (see Issue Cohort generation console print issues #77): failures now use prominent danger-level messages with the failed cohort, label, and underlying error, remaining cohorts are reported as not generated, and the final summary says generation failed when appropriate. The cohort lookup now checks for missing manifest entries before accessing their fields, and cancellation guidance points to builder scripts and manifest methods instead of only cohortsLoad.csv.
  • generateCohorts() now delegates table creation to executeCohortGeneration(confirm = FALSE), avoiding duplicate table checks and connection cycles while still creating missing tables automatically. The full cohort-count table is no longer printed to the console; a compact count summary and the saved cohortCounts.csv path are reported instead.
  • initUlyssesRepo() now aborts when its target repository directory already exists, preventing initialization from overwriting an existing repository.

mdlavallee92 and others added 30 commits July 22, 2026 10:27
`tabulateManifest()` and the `query*()` methods default to
`tags_format = "nested"`, which returns the `tags` column as a
tag_name/tag_value tibble rather than the raw JSON string stored in
sqlite. Several call sites re-parsed that column with
`jsonlite::fromJSON()`, which aborts with

    Argument 'txt' must be a JSON string, URL or file.

when handed a tibble. This broke `check_which_atlas_exist()` — and so
`importAtlasCohorts()` / `importAtlasConceptSets()` and any
`sourceInputBuilderScripts()` run that calls them — plus
`listAllUniqueTags()` and `getTagValuesSummary()`, which read the
manifest with the default format.

- Add `safe_parse_tags()` / `safe_tag_value()` in manifest_helpers.R.
  They normalise a JSON string, a nested tag_name/tag_value tibble, or
  an already-parsed list to a named list, and degrade to `list()` for
  NULL/NA/empty/malformed input.
- Route every manifest tag parse through them, replacing the duplicated
  inline `fromJSON()` parsers. A cohort or concept set with no tags no
  longer errors in `getCohortTags()` / `getConceptSetTags()`.
- Have the ATLAS import and update paths request `tags_format = "json"`
  explicitly, and coerce the `atlasId` tag rather than assuming it comes
  back as an integer.
- Fix the roxygen for the `query*()` methods, which advertised "json" as
  the default when `match.arg()` selects "nested".

Also regenerates man/ExecutionSettings.Rd and man/importAndBind.Rd,
which were stale on develop.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 'stale' status conflated two orthogonal things: lifecycle (active /
deleted / purged) and generation freshness. Because it lived in the same
`status` column, every `WHERE status = 'active'` lookup dropped stale
cohorts — so a cohort that merely needed regenerating fell out of the
study entirely.

Three reported symptoms, one root cause each:

1. Re-running addDependentCustomCohort() marked the cohort stale.
   upsert_derived_cohort() computed definition_changed but used it only
   to word the console message, then set status = 'stale' and cascaded
   downstream unconditionally — producing the contradictory "no changes
   detected (marked stale for regeneration)". Since the generated derived
   SQL is deterministic, an accidental re-run hashed identically and
   still flipped the cohort and every dependent to stale. All the
   build*Cohort() methods share this path. Staleness is now cascaded only
   on a real definition change, compared across content hash, file path,
   depends_on and dependency_rule; a metadata-only edit updates
   category/tags without forcing regeneration, and a full no-op returns
   early so a cohort already queued for regeneration stays stale.

2. Querying a stale cohort by label reported it did not exist. The query
   methods all build on tabulateManifest(filter = "active"), which meant
   status = 'active' literally. "active" now means registered — 'active'
   or 'stale' — and the returned status column tells them apart;
   filter = "stale" still narrows to just those. The same widening is
   applied to the label/file-path/parent validators, the ATLAS
   registration lookup, the tag operations, update_cohort_def(),
   reviewDependentCohorts(), statusReport() and syncManifest(), so a
   stale cohort keeps its label and stays usable as a parent. The unique
   partial indexes are widened to match, in place, on manifest load.

3. generateCohorts() listed only active cohorts in its confirmation
   prompt — omitting precisely the cohorts that needed generating. Stale
   entries are now listed and flagged.

Also: stale cohorts fell into neither bucket of print()'s active+deleted
total, so "Total cohorts" appeared to shrink when cohorts went stale.
They now count as active with their own Stale line, getManifestStatus()
gains stale_count, and its missing_count covers stale rows whose file is
missing. evaluate_cohort_skip_status() no longer lets a parentless stale
cohort be skipped on a checksum match. Fixed viewManifest() calling
match.arg() on an undeclared tags_format argument, which errored on
every call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Handle non-JSON tag columns in manifest parsing (#78)
Make 'stale' mean "needs regenerating", not "gone" (#74)
disseminationEnv was only ever built as a side effect of
sourceDisseminationScripts(), so an analyst writing a dissemination
script had no way to get a populated object at the console.

- Add exported createDisseminationEnv() that builds the object standalone
  (pipelineVersion, databaseIds, outputPath, resultsPath), with version
  auto-detection from config.yml via the new detect_pipeline_version()
  helper.
- Refactor sourceDisseminationScripts() to call it, so construction lives
  in exactly one place.
- Show the interactive pattern in the dissemination script template and
  in the post_processing vignette (agent doc regenerated from it).
- Add tests/testthat/test-createDisseminationEnv.R.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
prepareDisseminationData() asserted a length of 1 on the length-4 vector
of its four logical flags, so every call aborted before any formatting
ran. Assert len = 4 instead.

The dissemination script template checked merged results for a
`database_id` column, but importAndBind() labels merged rows with
`databaseId`, so the database summary never printed. The snake_case
references later in the template are correct because they apply to data
already passed through prepareDisseminationData(clean_names = TRUE); the
pivot example is updated to use the cleaned name and the real
pivotForComparison() argument names.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
The default "_id$" rule matches the character database_id column that
cleanColumnNames() produces, and as.integer() silently turned it into NA.
The surrounding tryCatch only caught errors, and coercion failure is a
warning, so the loss was silent. Skip any conversion that would introduce
NA for a non-missing value and report the column instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
Two independent causes made execStudyPipeline() unrunnable.

1. Manifest sqlite churn. update_cohort_def() and update_concept_set_def()
   issued an UPDATE unconditionally, even when every value already matched
   the stored row. Because the cohort write also sets
   updated_at = CURRENT_TIMESTAMP, a metadata refresh or an ATLAS
   re-import that changed nothing still rewrote cohortManifest.sqlite, so
   validateCodeState() saw uncommitted changes. Both now drop unchanged
   assignments before building the UPDATE and return early when nothing is
   left, and cascadeStaleDownstream() skips rows that are already stale
   (re-flagging a stale cohort is a no-op that only bumps updated_at).

   upsert_derived_cohort() had the same problem, but #74 already fixed it
   on develop with a finer definition/metadata split; that version is kept
   as-is and this branch only adds byte-level assertions around it.

2. False "Environment drift detected!". validateEnvironment() treated a
   non-NULL renv::status() as drift, but renv::status() always returns a
   list, so the check failed on every project including freshly
   snapshotted ones. It now reads the `synchronized` flag and, as a
   fallback, diffs the library and lockfile package records; source-only
   mismatches (the "unknown source" case) warn rather than block.

A third cause — nondeterministic ATLAS concept set ordering churning the
on-disk JSON and its content hash — is split out into its own branch,
fix-84-atlas-concept-ordering, because canonicalizing the bytes written to
disk is a larger judgment call than the fixes here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
runPreflightChecks() ran renv::snapshot() immediately after validating that
the working tree was clean, so the check could dirty the very tree it had
just approved and leave the next run failing its code-state check.

Record the lockfile instead of refreshing it: captureLockfile() hashes
renv.lock as it stands and archives the versioned copy post-processing needs
for provenance. validateEnvironment() still fails the run on real drift and
already tells the analyst to run snapshotEnvironment() themselves.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
Production runs require a clean working tree so results can be tied to a
known commit. That is impossible to satisfy in study repos where files
under inputs/ churn without a deliberate edit -- manifest sqlite files
rewritten on re-import, ATLAS-sourced JSON refetched.

Two opt-in escape hatches, both defaulting to off so unchanged studies
get exactly the previous behavior:

- ignoreUncommittedPaths tolerates uncommitted changes confined to the
  listed repo-relative paths; anything outside them (notably analysis/)
  still fails. Settable once per study in the default: block of
  config.yml, with the execStudyPipeline() argument overriding it for a
  single run.
- skipCodeStateCheck skips the check entirely as a last resort. Argument
  only -- deliberately not settable from config.yml so it cannot be baked
  permanently into a study.

Neither hatch is silent. Code state is reported as a warning rather than
a pass, a banner names every ignored file, the pipeline log records the
code state and commit SHA, and task_run_history.csv gains commit_sha and
code_state columns (clean / dirty-ignored / unverified-skipped /
unverified-test-mode / unrecorded) so the audit trail never implies a
clean tree when the tree was not clean. Legacy history files are read
permissively and back-filled as unrecorded.

The branch guard is untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
It was filed under 0.0.6; DESCRIPTION is at 0.0.7 and 0.0.7 is the open section.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
inst/agent/05-manifest-management.md had drifted from
vignettes/manifest_management.Rmd, missing the viewManifest() and
tabulateManifest(tags_format=) documentation added in 0.0.7.
Regenerated via sync_agent_docs_from_vignettes(); the other eight
agent docs were already in sync.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
@mdlavallee92
mdlavallee92 merged commit ba2df97 into main Sep 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants