Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 11 additions & 9 deletions paper.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,15 @@ Paleoclimate researchers use measurements from natural archives, such as tree ri

# Statement of need

NOAA NCEI for Paleoclimatology and PANGAEA are the two main repositories used worldwide by the paleoclimate community to archive and access data. NOAA NCEI for Paleoclimatology is dedicated to paleoclimate data, whereas PANGAEA is a broader Earth and environmental science repository. Because datasets routinely used in paleoclimate research may be archived in either or both repositories, researchers often need to search across both systems, particularly for synthesis efforts that compile hundreds of records across regions, archive types, proxy systems, or time intervals.
NOAA's World Data Service for Paleoclimatology (WDS-Paleo) is dedicated to paleoclimate data, whereas PANGAEA is a broader Earth and environmental science repository. Because datasets routinely used in paleoclimate research may be archived in either or both repositories, researchers often need to search across both systems, particularly for synthesis efforts that compile hundreds of records across regions, archive types, proxy systems, or time intervals.

This cross-repository access problem affects more than software convenience. Paleoclimate scientists use existing records to compare with new observations, build syntheses, generate climate field reconstructions, and address questions that require large compilations, such as changes in climate variability or tipping points in the climate system. Differences in repository search interfaces, metadata models, and returned search results can affect which records are identified, how complete a compilation is, and whether the search process can be transparently documented and reproduced.
This cross-repository access problem affects the scientific conclusions researchers can draw, not just their workflow convenience. Paleoclimate scientists use existing records to compare with new observations, build syntheses, generate climate field reconstructions, and address questions that require large compilations, such as changes in climate variability or tipping points in the climate system. Differences in repository search interfaces, metadata models, and returned search results can affect which records are identified, how complete a compilation is, and whether the search process can be transparently documented and reproduced.
Comment on lines +44 to +46

`PyleoTUPS` fills the gap between repository-level discovery and analysis-ready paleoclimate workflows by providing consistent Python access to NOAA NCEI for Paleoclimatology and PANGAEA. The package harmonizes common access patterns, including search parameters and functions for retrieving publications, geographic metadata, data tables, and variable-level metadata, while preserving repository-specific information when scientific meaning would be lost through full homogenization. Results are returned as `pandas.DataFrame` objects that can be inspected, filtered, documented, and passed to downstream analysis workflows.
`PyleoTUPS` fills the gap between repository-level discovery and analysis-ready paleoclimate workflows by providing consistent Python access to WDS-Paleo and PANGAEA. The package unifies common access patterns, including search parameters and functions for retrieving publications, geographic metadata, data tables, and variable-level metadata, while preserving repository-specific information when scientific meaning would be lost through full homogenization.

# State of the field

Researchers can currently access NOAA NCEI for Paleoclimatology through its web interface and API [@Morrill2021], and PANGAEA through its web interface and tools such as `pangaeapy` [@pangaeapy]. These resources provide authoritative access to repository holdings, but they are repository-specific and expose different search parameters, metadata structures, and data-return formats. As a result, researchers who need records from both repositories often rely on custom scripts to discover records, retrieve metadata, and load associated data tables. These scripts address the first step in many paleoclimate workflows: finding and extracting relevant records from public archives.
Researchers can currently access WDS-Paleo through its web interface and API [@Morrill2021], and PANGAEA through its web interface and tools such as `pangaeapy` [@pangaeapy]. These resources provide authoritative access to repository holdings, but they are repository-specific and expose different search parameters, metadata structures, and data-return formats. As a result, researchers who need records from both repositories often rely on custom scripts to discover records, retrieve metadata, and load associated data tables. These scripts address the first step in many paleoclimate workflows: finding and extracting relevant records from public archives.

Once records have been identified and retrieved, they may need to be curated into a standardized representation before they can be reused in synthesis or analysis workflows. The Linked Paleo Data format (LiPD) [@McKay2016] and `PyLiPD` [@Ratnakar2025] address this later step. LiPD provides an intermediate archival format in which paleoclimate data and metadata can be normalized, aligned with controlled vocabularies, and represented consistently across datasets. `PyLiPD` provides Python tools for reading, querying, editing, and writing datasets once they have been encoded in this format. `PyleoTUPS` operates upstream of this standardization step by helping researchers discover and retrieve records from NOAA NCEI for Paleoclimatology and PANGAEA before manual curation or transformation into LiPD.

Expand All @@ -60,7 +60,7 @@ The decision to build `PyleoTUPS` rather than contribute only to `pangaeapy`, `P

# Software design

`PyleoTUPS` is built on top of the NOAA NCEI for Paleoclimatology API [@Morrill2021] and the Python `pangaeapy` library [@pangaeapy]. The package provides two primary dataset interfaces, `NOAADataset` and `PangaeaDataset`, that encapsulate repository-specific access methods while exposing a common set of user-facing functions. This two-object design reflects an explicit trade-off: `PyleoTUPS` harmonizes search and retrieval workflows where possible, but avoids forcing NOAA NCEI for Paleoclimatology and PANGAEA records into a single semantic model when doing so could obscure scientifically meaningful metadata differences.
`PyleoTUPS` is built on top of the WDS-Paleo API [@Morrill2021] and the Python `pangaeapy` library [@pangaeapy]. The package provides two primary dataset interfaces, `NOAADataset` and `PangaeaDataset`, that encapsulate repository-specific access methods while exposing a common set of user-facing functions. This two-object design reflects an explicit trade-off: `PyleoTUPS` standardizes search and retrieval workflows where possible, but avoids forcing NOAA WDS-Paleo and PANGAEA records into a single semantic model when doing so could obscure scientifically meaningful metadata differences.

For NOAA records, search queries are submitted through the NOAA API, which returns study-level metadata in JSON format. `PyleoTUPS` uses these metadata to identify associated paleoclimate data tables, then parses the tables into `pandas.DataFrame` objects. For PANGAEA records, `PyleoTUPS` builds on `pangaeapy` while providing a paleoclimate-oriented Python interface for constructing search queries and retrieving matching records through a workflow similar to the NOAA interface.

Expand All @@ -70,18 +70,20 @@ Both dataset objects provide common methods, including `get_publications`, `get_

`PyleoTUPS` is released through [PyPI](https://pypi.org/project/pyleotups/) and [GitHub](https://github.com/LinkedEarth/PyleoTUPS) and is accompanied by [public documentation](https://pyleotups.readthedocs.io/en/latest/) and a tutorial Jupyter Book [@pyleotups_tutorials]. The tutorials include introductory material, examples for working with `NOAADataset` and `PangaeaDataset` objects, and scientific workflows that demonstrate how repository data can be retrieved, inspected, and used in paleoclimate analyses. The package has been downloaded approximately 1.4k times from PyPI and is under active development, with issue-driven testing and refinement on GitHub.

Although `PyleoTUPS` is a new package, it is already being tested in an active paleoclimate data-compilation workflow. An undergraduate researcher with no prior experience in Python or paleoclimate science is using the package to help retrieve and transform repository data into LiPD format for an update to the PAGES2k compilation [@PAGES2k2017]. This ongoing use case suggests that `PyleoTUPS` can lower the technical barrier to cross-repository data access and support workflows that feed into community-scale paleoclimate products.

`PyleoTUPS` is also designed to connect repository discovery with downstream scientific software. The tutorials demonstrate interoperability with `Pyleoclim` [@Khider2022], and the same `pandas.DataFrame`-based outputs can support workflows using packages such as `cfr` [@Zhu2024].
`PyleoTUPS` is also designed to connect repository discovery with downstream scientific software. The tutorials demonstrate interoperability with `Pyleoclim` [@Khider2022], and the same `pandas.DataFrame`-based outputs can support workflows using packages such as `cfr` [@Zhu2024].


Although `PyleoTUPS` is a new package, it is already being tested in an active paleoclimate data-compilation workflow. An undergraduate researcher with no prior experience in Python or paleoclimate science is using the package to help retrieve and transform repository data into LiPD format for an update to the PAGES2k compilation [@PAGES2k2017]. This ongoing use case suggests that `PyleoTUPS` can lower the technical barrier to cross-repository data access and support workflows that feed into community-scale paleoclimate products. By lowering the barrier to reproducible cross-repository search, `PyleoTUPS` supports exactly the kind of large-scale synthesis effort this paper opened with — the PAGES2k update now underway is an early example.

# AI usage disclosure

AI code completion was used to help fix some functionalities, generate API documentation, and write unit tests. Generative AI tools, including Claude and ChatGPT, were used to help draft the manuscript and tighten the prose in the tutorials. All tutorial code was written manually by the authors to test and demonstrate scientific use of the package.
We used AI code completion to help fix some functionality, generate API documentation, and write unit tests. We used generative AI tools, including Claude and ChatGPT, to help draft the manuscript and tighten the prose in the tutorials. The authors wrote all tutorial code manually to test and demonstrate scientific use of the package.


# Acknowledgements

The authors were supported by NSF Award #2411267 and #2411268. We thank Dr Georgina Falster at The University of Adelaide for testing the alpha version of this software and providing feedback on the early implementation.
The authors were supported by NSF Award #2411267 and #2411268. We thank Dr. Georgina Falster at The University of Adelaide for testing the alpha version of this software and providing feedback on the early implementation.


# References
Loading