Repository navigation
v007 - #100
Merged
Merged
v007#100
Conversation
Small Fixes
Ml fixes
`tabulateManifest()` and the `query*()` methods default to
`tags_format = "nested"`, which returns the `tags` column as a
tag_name/tag_value tibble rather than the raw JSON string stored in
sqlite. Several call sites re-parsed that column with
`jsonlite::fromJSON()`, which aborts with
Argument 'txt' must be a JSON string, URL or file.
when handed a tibble. This broke `check_which_atlas_exist()` — and so
`importAtlasCohorts()` / `importAtlasConceptSets()` and any
`sourceInputBuilderScripts()` run that calls them — plus
`listAllUniqueTags()` and `getTagValuesSummary()`, which read the
manifest with the default format.
- Add `safe_parse_tags()` / `safe_tag_value()` in manifest_helpers.R.
They normalise a JSON string, a nested tag_name/tag_value tibble, or
an already-parsed list to a named list, and degrade to `list()` for
NULL/NA/empty/malformed input.
- Route every manifest tag parse through them, replacing the duplicated
inline `fromJSON()` parsers. A cohort or concept set with no tags no
longer errors in `getCohortTags()` / `getConceptSetTags()`.
- Have the ATLAS import and update paths request `tags_format = "json"`
explicitly, and coerce the `atlasId` tag rather than assuming it comes
back as an integer.
- Fix the roxygen for the `query*()` methods, which advertised "json" as
the default when `match.arg()` selects "nested".
Also regenerates man/ExecutionSettings.Rd and man/importAndBind.Rd,
which were stale on develop.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 'stale' status conflated two orthogonal things: lifecycle (active / deleted / purged) and generation freshness. Because it lived in the same `status` column, every `WHERE status = 'active'` lookup dropped stale cohorts — so a cohort that merely needed regenerating fell out of the study entirely. Three reported symptoms, one root cause each: 1. Re-running addDependentCustomCohort() marked the cohort stale. upsert_derived_cohort() computed definition_changed but used it only to word the console message, then set status = 'stale' and cascaded downstream unconditionally — producing the contradictory "no changes detected (marked stale for regeneration)". Since the generated derived SQL is deterministic, an accidental re-run hashed identically and still flipped the cohort and every dependent to stale. All the build*Cohort() methods share this path. Staleness is now cascaded only on a real definition change, compared across content hash, file path, depends_on and dependency_rule; a metadata-only edit updates category/tags without forcing regeneration, and a full no-op returns early so a cohort already queued for regeneration stays stale. 2. Querying a stale cohort by label reported it did not exist. The query methods all build on tabulateManifest(filter = "active"), which meant status = 'active' literally. "active" now means registered — 'active' or 'stale' — and the returned status column tells them apart; filter = "stale" still narrows to just those. The same widening is applied to the label/file-path/parent validators, the ATLAS registration lookup, the tag operations, update_cohort_def(), reviewDependentCohorts(), statusReport() and syncManifest(), so a stale cohort keeps its label and stays usable as a parent. The unique partial indexes are widened to match, in place, on manifest load. 3. generateCohorts() listed only active cohorts in its confirmation prompt — omitting precisely the cohorts that needed generating. Stale entries are now listed and flagged. Also: stale cohorts fell into neither bucket of print()'s active+deleted total, so "Total cohorts" appeared to shrink when cohorts went stale. They now count as active with their own Stale line, getManifestStatus() gains stale_count, and its missing_count covers stale rows whose file is missing. evaluate_cohort_skip_status() no longer lets a parentless stale cohort be skipped on a checksum match. Fixed viewManifest() calling match.arg() on an undeclared tags_format argument, which errored on every call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Handle non-JSON tag columns in manifest parsing (#78)
Make 'stale' mean "needs regenerating", not "gone" (#74)
disseminationEnv was only ever built as a side effect of sourceDisseminationScripts(), so an analyst writing a dissemination script had no way to get a populated object at the console. - Add exported createDisseminationEnv() that builds the object standalone (pipelineVersion, databaseIds, outputPath, resultsPath), with version auto-detection from config.yml via the new detect_pipeline_version() helper. - Refactor sourceDisseminationScripts() to call it, so construction lives in exactly one place. - Show the interactive pattern in the dissemination script template and in the post_processing vignette (agent doc regenerated from it). - Add tests/testthat/test-createDisseminationEnv.R. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
prepareDisseminationData() asserted a length of 1 on the length-4 vector of its four logical flags, so every call aborted before any formatting ran. Assert len = 4 instead. The dissemination script template checked merged results for a `database_id` column, but importAndBind() labels merged rows with `databaseId`, so the database summary never printed. The snake_case references later in the template are correct because they apply to data already passed through prepareDisseminationData(clean_names = TRUE); the pivot example is updated to use the cleaned name and the real pivotForComparison() argument names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
The default "_id$" rule matches the character database_id column that cleanColumnNames() produces, and as.integer() silently turned it into NA. The surrounding tryCatch only caught errors, and coercion failure is a warning, so the loss was silent. Skip any conversion that would introduce NA for a non-missing value and report the column instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
Two independent causes made execStudyPipeline() unrunnable. 1. Manifest sqlite churn. update_cohort_def() and update_concept_set_def() issued an UPDATE unconditionally, even when every value already matched the stored row. Because the cohort write also sets updated_at = CURRENT_TIMESTAMP, a metadata refresh or an ATLAS re-import that changed nothing still rewrote cohortManifest.sqlite, so validateCodeState() saw uncommitted changes. Both now drop unchanged assignments before building the UPDATE and return early when nothing is left, and cascadeStaleDownstream() skips rows that are already stale (re-flagging a stale cohort is a no-op that only bumps updated_at). upsert_derived_cohort() had the same problem, but #74 already fixed it on develop with a finer definition/metadata split; that version is kept as-is and this branch only adds byte-level assertions around it. 2. False "Environment drift detected!". validateEnvironment() treated a non-NULL renv::status() as drift, but renv::status() always returns a list, so the check failed on every project including freshly snapshotted ones. It now reads the `synchronized` flag and, as a fallback, diffs the library and lockfile package records; source-only mismatches (the "unknown source" case) warn rather than block. A third cause — nondeterministic ATLAS concept set ordering churning the on-disk JSON and its content hash — is split out into its own branch, fix-84-atlas-concept-ordering, because canonicalizing the bytes written to disk is a larger judgment call than the fixes here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
runPreflightChecks() ran renv::snapshot() immediately after validating that the working tree was clean, so the check could dirty the very tree it had just approved and leave the next run failing its code-state check. Record the lockfile instead of refreshing it: captureLockfile() hashes renv.lock as it stands and archives the versioned copy post-processing needs for provenance. validateEnvironment() still fails the run on real drift and already tells the analyst to run snapshotEnvironment() themselves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
Production runs require a clean working tree so results can be tied to a known commit. That is impossible to satisfy in study repos where files under inputs/ churn without a deliberate edit -- manifest sqlite files rewritten on re-import, ATLAS-sourced JSON refetched. Two opt-in escape hatches, both defaulting to off so unchanged studies get exactly the previous behavior: - ignoreUncommittedPaths tolerates uncommitted changes confined to the listed repo-relative paths; anything outside them (notably analysis/) still fails. Settable once per study in the default: block of config.yml, with the execStudyPipeline() argument overriding it for a single run. - skipCodeStateCheck skips the check entirely as a last resort. Argument only -- deliberately not settable from config.yml so it cannot be baked permanently into a study. Neither hatch is silent. Code state is reported as a warning rather than a pass, a banner names every ignored file, the pipeline log records the code state and commit SHA, and task_run_history.csv gains commit_sha and code_state columns (clean / dirty-ignored / unverified-skipped / unverified-test-mode / unrecorded) so the audit trail never implies a clean tree when the tree was not clean. Legacy history files are read permissively and back-filled as unrecorded. The branch guard is untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
It was filed under 0.0.6; DESCRIPTION is at 0.0.7 and 0.0.7 is the open section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
inst/agent/05-manifest-management.md had drifted from vignettes/manifest_management.Rmd, missing the viewManifest() and tabulateManifest(tags_format=) documentation added in 0.0.7. Regenerated via sync_agent_docs_from_vignettes(); the other eight agent docs were already in sync. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QquaXJZxgCaEFtUTHVioG1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
picard 0.0.7
New Features
Code State Escape Hatches
inputs/churn without a deliberate edit. Both default to off, so unchanged studies get exactly the previous behavior.execStudyPipeline(ignoreUncommittedPaths = ...)tolerates uncommitted changes confined to the listed repo-relative paths; changes anywhere else (notablyanalysis/) still fail the check. The list can be set once per study asignoreUncommittedPathsin thedefault:block ofconfig.yml, with the argument overriding it for a single run.execStudyPipeline(skipCodeStateCheck = TRUE)skips the check entirely as a last resort. Deliberately not settable fromconfig.yml.Code stateis reported as a warning rather than a pass, a banner names every ignored file, and the pipeline log records the code state and commit SHA.exec/logs/task_run_history.csvgainscommit_shaandcode_statecolumns (clean,dirty-ignored,unverified-skipped,unverified-test-mode,unrecorded) so the audit trail never implies a clean tree when the tree was not clean. Existing history files are read and back-filled asunrecorded.Tabulate and View Manifest
viewManifest()convenience method for bothCohortManifestandConceptSetManifestto interactively explore manifest metadata in RStudio viewer with streamlined columns (id, label, category, tags, file_path).tabulateManifest()withtags_formatparameter supporting three formats:"nested"(default, parsed tags as tibble with tag_name/tag_value),"json"(raw JSON string for backward compatibility), and"wide"(expanded tags as individual columns for spreadsheet-like analysis).Building a disseminationEnv Interactively
Study Metadata and Publishing
studyDescriptionfield tomakeStudyMeta(). When supplied, it is inserted into the generated README; when omitted, the existing description placeholder is retained.publishStudyHubPosit()to render and publish a Study Hub through Quarto's public Posit Connect publishing API.Bug Fixes
'stale'cohort status (see Issue Various issues with "stale" status #74).'stale'now consistently means "registered, but needs regenerating", rather than removing a cohort from the study:addDependentCustomCohort(),buildUnionCohort()) withstopIfExists = FALSEand unchanged inputs no longer marks the cohort — and everything downstream of it — stale. Staleness is now cascaded only on a real definition change (content hash, file path, parents, or build rule), and a metadata-only edit no longer forces regeneration.tabulateManifest(filter = "active")and therefore by allquery...()methods, so they can be queried by label/tag/category and used as parents for new derived cohorts.generateCohorts()now lists stale cohorts in its confirmation prompt (they are exactly the cohorts that need generating) and flags them as such.print()andgetManifestStatus()count stale cohorts as registered and break them out separately;getManifestStatus()gainsstale_count.viewManifest()erroring on an undefinedtags_formatargument; it now also shows thestatuscolumn.tagscolumn holds a JSON string (see Issue Robustly handle non-string tags in manifest parsing #78). Thequery*ByTagName()methods default totags_format = "nested", which returns tags as atag_name/tag_valuetibble, so internal callers that re-parsed that column failed withArgument 'txt' must be a JSON string, URL or file.This brokeimportAtlasCohorts()/importAtlasConceptSets()(and thereforesourceInputBuilderScripts()), as well aslistAllUniqueTags()andgetTagValuesSummary(). Tag parsing now goes through a shared helper that accepts JSON strings, nested tibbles, and already-parsed lists, and treats missing/empty/malformed tags as no tags.$queryConceptSetsByCategory()it was missingUPDATEeven when every value already matched the stored row, bumpingupdated_atand rewritingcohortManifest.sqlite/conceptSetManifest.sqliteon every metadata refresh and every ATLAS re-import. Unchanged assignments are now dropped, so a re-run leaves the manifest files byte-identical andvalidateCodeState()no longer sees uncommitted changes.cascadeStaleDownstream()likewise skips rows that are already stale.validateEnvironment()always reported "Environment drift detected!" because it testedrenv::status()forNULL, andrenv::status()always returns a list. It now reads thesynchronizedflag and compares the library and lockfile package records, and treats source-only mismatches as a warning rather than a blocker.runPreflightChecks()ranrenv::snapshot()immediately after validating that the working tree was clean, so the check could dirty the tree it had just approved and leave the next run failing its code-state check. The lockfile is now recorded rather than refreshed; keepingrenv.lockcurrent is the analyst's call, viasnapshotEnvironment().stopIfExistsoption to ATLAS csv load builders (see Issue Add stopIfExists to ATLAS csv loaders #65)dplyr::select(-.data$tags_list)in the tag-expansion helpers replaced withselect(-"tags_list"); use of.datain tidyselect expressions was deprecated in tidyselect 1.2.0 and would eventually have errored.resetManifestturn archive default to FALSEprepareDisseminationData()asserted that four concatenated logical flags had length 1, so it aborted on every call — including all-default calls — and the formatting step could never run.database_idcolumn, butimportAndBind()labels them with camelCasedatabaseId, so the check was always false. The later references todatabase_idare correct, as they operate on data already passed throughcleanColumnNames().standardizeDataTypes()no longer discards columns it cannot coerce. The default"_id$"rule matched the characterdatabase_idcolumn andas.integer()silently turned it intoNA, since failed coercion raises a warning rather than an error. A conversion that would turn a non-missing value intoNAis now skipped and reported.query...Manifestmethods to use$tabulateManifest()internally to be more consistent.route = atlasorroute = capr(See Issue Move Capr/ATLAS/etc. tags to a column in the manifest #67)$grabConceptInfoFromSet()change input to conceptSetRef which accepts either a manifest id or a label (See issue Add an option to get concept info by label toconceptSetManifest$grabConceptInfoFromSet()#56)importAndBind()save to avoid similar output names between tasks.CONCEPT_ID(ties broken by vocabulary, code and inclusion flags) and circeConceptSetsby theirid. Ordering is left untouched when any concept set lacks a usableid, soCodesetIdreferences cannot break.reviewConnectionDetails()pulls the connectionDetails used.getServerCredentials()to not evaluate full yml file, only pull the specific block and evaluate (See issue Can't Run Pipeline in Background Job When ATLAS Creds Stored in keyring #30, Add useKeyring as pipeline input var #33).makeDisseminationScript()change the glue brackets to carrots$updateAtlasCohorts()/$updateAtlasConceptSets()to use tags in json mode after tab format change.inst/agent/05-manifest-management.mdfromvignettes/manifest_management.Rmd. It had not been synced sinceviewManifest()and thetabulateManifest(tags_format = ...)options were added, so agents in generated study repos were reading stale manifest guidance.execStudyPipeline()/testStudyPipeline()no longer go silent during a run (see Issue Make execStudyPipeline and testStudyPipeline output more verbose #85).execute_pipeline()usedsink(type = "output")to capture console output into the pipeline log, which redirected everycliinfo bullet and error away from the console for the entire run — nothing showed up live, including the detail of a task's actual error. The sink is removed entirely: console output now streams normally, and the log file is instead written explicitly with high-level milestones (config block/task start, success, failure) plus a full error detail block (class, call, message) on failure. Separately,execute_task()was callingstop()from inside its per-expression error handler, which skipped the subsequentrecordTaskExecution(..., errorMessage = ...)call entirely — sotask_run_history.csvnever recorded the real error message for a failed task, only whatever generic message the caller happened to re-throw. The detailed message is now recorded correctly.cohortsLoad.csv.generateCohorts()now delegates table creation toexecuteCohortGeneration(confirm = FALSE), avoiding duplicate table checks and connection cycles while still creating missing tables automatically. The full cohort-count table is no longer printed to the console; a compact count summary and the savedcohortCounts.csvpath are reported instead.initUlyssesRepo()now aborts when its target repository directory already exists, preventing initialization from overwriting an existing repository.