Skip to content

Improve performance and offline robustness of taxonomic resource loading - #266

Merged
dfalster merged 10 commits into
masterfrom
improve_download_system
Mar 23, 2026
Merged

Improve performance and offline robustness of taxonomic resource loading#266
dfalster merged 10 commits into
masterfrom
improve_download_system

Conversation

@dfalster

@dfalster dfalster commented Mar 23, 2026

Copy link
Copy Markdown
Member

Address #263

Summary

  • Session-level cache: load_taxonomic_resources() now caches results in a package-private environment (.pkg_cache) so repeated calls within a session return immediately without re-downloading or re-processing. Adds clear_cached_resources() to force a reload. This is CRAN-compliant — the cache lives in the package namespace, not .GlobalEnv.
  • Offline fallback: When no internet connection is available, default_version() falls back to the most recently downloaded local version, and dataset_get() reads cached parquet files directly. The package now works fully offline as long as resources have been downloaded at least once.
  • CRAN-compliant offline tests: Replaced the single skipped connection test with comprehensive tests covering every network-touching function. Tests simulate offline via NETWORK_UP=FALSE so they require no internet and run on CRAN.

Test plan

  • devtools::test() passes with no failures
  • Verify load_taxonomic_resources() returns cached result on second call (check for "Using cached" message)
  • Verify load_taxonomic_resources() works with NETWORK_UP=FALSE when parquet files are locally present
  • Verify clear_cached_resources() forces a fresh load on next call
  • Run R CMD check to confirm CRAN compliance

🤖 Generated with Claude Code

Cache loaded resources in a package-private environment (.pkg_cache) so
repeated calls to load_taxonomic_resources() within a session return
immediately without re-downloading or re-processing. Only "stable" data
is cached (keyed by version); "current" data is always reloaded. Adds
clear_cached_resources() to allow users to force a reload, and tests
covering all cache behaviours.
When no internet connection is available, default_version() now falls
back to the most recently downloaded local version rather than returning
NULL. dataset_get() reads local parquet files directly when offline
instead of bailing out. Adds local_cached_versions() helper to discover
locally available versions. Tests cover the offline fallback path for
default_version(), dataset_get(), and load_taxonomic_resources().
Replace the single skipped connection test with thorough offline tests
covering every network-touching function (default_version,
dataset_get, dataset_access_function, load_taxonomic_resources,
get_versions). Tests simulate offline via NETWORK_UP=FALSE so they
require no internet and run on CRAN. Each test verifies both that a
message is emitted and that NULL is returned rather than an error.
@dfalster
dfalster requested review from ehwenk and wcornwell March 23, 2026 04:31
@wcornwell

Copy link
Copy Markdown
Contributor

my claude tells me that your claude build this:

  1. Session-level in-memory cache (lines 53–58)
    When you call load_taxonomic_resources() with stable_or_current_data = "stable" (the default), the processed result is stored in .pkg_cache — a private package environment. On subsequent calls with the same version, it returns immediately with "Using cached taxonomic resources." and skips all downloading and processing.

  2. Disk-level parquet cache (lines 575–586 in dataset_get)
    The raw APC/APNI parquet files are downloaded once and stored in tools::R_user_dir("APCalign") (a persistent local directory). On subsequent calls, if the parquet files already exist on disk, they are read directly without re-downloading.

So the flow is:

Same version called again this session → returns from memory instantly
New session, version already on disk → reads from local parquet files (no download)
New session, version not on disk → downloads from GitHub, saves to disk
To force a full reload, call clear_cached_resources() (clears the in-memory cache only — disk files remain).

Note: "current" data is never cached at either level, since it's meant to be live.

@wcornwell

Copy link
Copy Markdown
Contributor

is there any way to cache the version number somehow? currently default_version() looks at github for the most recent version but this fails off line. if we are going to support offline use, default_version() needs to 1) check for internet access and 2) if it's down then find the local version number somehow, 3) use that version number to find the cache?

@dfalster
dfalster force-pushed the improve_download_system branch from f58e536 to ea4838b Compare March 23, 2026 06:17

@ehwenk ehwenk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the main code changes and changed documentation, then turned to testing functionality and got the expected results with and without internet, without and without clearing cached resources, trying all combinations. It enormously increased the speed of functions running.

@dfalster
dfalster merged commit e72f605 into master Mar 23, 2026
4 checks passed
@dfalster
dfalster deleted the improve_download_system branch March 23, 2026 07:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants