Skip to content

Fix get_data() dropping files when a DataTableID has multiple URLs - #61

Merged
khider merged 1 commit into
mainfrom
multiplefiles
Aug 12, 2026
Merged

Fix get_data() dropping files when a DataTableID has multiple URLs#61
khider merged 1 commit into
mainfrom
multiplefiles

Conversation

@khider

@khider khider commented Aug 12, 2026

Copy link
Copy Markdown
Member

This PR fixes #58

Problem

NOAADataset.get_data() assumed a 1:1 mapping between DataTableID and file URL — it only ever fetched paleo_data.file_url, a shortcut to the first file NOAA attached to that table, silently ignoring the rest. In practice, one DataTableID can carry several files of different formats (a NOAA-templated .txt, the original contributed .txt, a .csv, a directory-listing page, or proprietary tree-ring formats), and these aren't always the same table in another format — e.g. a tree-ring DataTableID can bundle a separate "Raw Measurements" table and a "Chronology" table under one ID. The old code returned at most one of them, and would outright crash if the first-listed file happened to be a non-.txt link (e.g. a folder).

Separately, DataTableIDs were assumed unique; if the same ID ever appeared under two different Study/Site pairs, the internal index silently kept only the last one.

Fix

  1. pyleotups/core/NOAADataset.py

get_data(dataTableIDs=...) now attempts every file under a DataTableID, ranked easiest-first (NOAA structured .txt → Original Contributed .txt → .csv → web page/folder link via new _rank_file_priority()), and keeps a DataFrame for every file that parses instead of stopping at the first success. A failing file is logged and skipped rather than aborting the rest; it only raises if none of a table's files could be parsed.

Duplicate DataTableIDs (same ID across different Study/Site pairs) are now surfaced via _check_duplicate_datatable_ids(), called from both get_tables() and get_data(). A new data_table_index_all multi-map lets get_data() open files from every colliding Study/Site rather than only the one left in the old single-entry index.

_process_file() gains support for .csv (via pandas.read_csv) and for web pages / directory-listing links (no extension or .html): it fetches the page, extracts links with the stdlib html.parser (no new dependency), ranks them by the same priority, and attempts each — addressing the optional "detect folder, extract URLs" ask.

Proprietary tree-ring formats (.crn, .rwl, .fhx, .lpd) remain explicitly unsupported and are still rejected, just now as one skipped attempt among several rather than a hard stop.

  1. pyleotups/tests/test_NOAADataset.py

New tests covering: priority ranking, falling back to the next file when the easiest one fails, raising when all files fail, keeping every DataFrame when multiple files succeed, duplicate-ID warnings in get_tables()/get_data(), opening files from every colliding Study/Site, .csv parsing, and folder-page link extraction/parsing.

Updated one existing assertion (test_get_data_t05_unsupported_file_type_raises) to match the more general "unsupported file type" message now that .csv and web pages are supported types.

Verification

Full suite: 121 passed (117 pre-existing + new tests), no regressions.

Checked against the live NOAA API, not just mocks:
NOAAStudyId=6400: 3 DataTableIDs each with a folder-link + an alternate .txt. Old code crashed on the very first ID (folder link has no .txt extension); fixed code returns all 3 successfully. This study was also added to the tutorials as an example.

xmlId=6400 (tree-ring study, DataTableID 6773, 5 files): now correctly returns 2 DataFrames (Chronology + Raw Measurements, the two NOAA-template .txt files) while clearly warning and skipping the 3 genuinely unparseable files (a prose correlation-stats report, .crn, .rwl).

Use of AI

The fix was generated by Claude (including new test suites). Human reviewed the verification against NOAA API manually.

@khider

khider commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

Problem with failure of tests comes from the new Pangaea update and will require new fixes. Merging things for now.

@khider
khider merged commit 6a68108 into main Aug 12, 2026
1 check failed
@khider
khider deleted the multiplefiles branch August 12, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multiple File URLs for one DataTableID

1 participant