fix: be more cautious when guessing what a backend can open #10804

ianhi · 2025-09-30T16:59:36Z

Previously pydap or netcdf if installed would grab any remote URL according the order of backend resolution.

Closes netcdf4 backend claims **all** remote files - preventing reading zarr #10801
Tests added
User visible changes (including notable bug fixes) are documented in whats-new.rst
[NA] New functions/methods are listed in api.rst

shoyer

Looks great, thanks @ianhi !

xarray/tests/test_backends.py

doc/whats-new.rst

ianhi · 2025-09-30T19:38:13Z

xarray/backends/pydap_.py

+        if not (isinstance(filename_or_obj, str) and is_remote_uri(filename_or_obj)):
+            return False
+
+        # Check file extension to avoid claiming non-OPeNDAP URLs (e.g., remote Zarr stores)
+        _, ext = os.path.splitext(filename_or_obj.rstrip("/"))
+        # Pydap handles OPeNDAP endpoints, which typically have no extension or .nc/.nc4
+        # Reject URLs with non-OPeNDAP extensions like .zarr
+        return ext not in {".zarr", ".zip", ".tar", ".gz"}


Not 100% sure on this. We could go further and require "dap" to be in the URL

I don't think there's a standard extension for OpenDAP URLs. @Mikejmnez do you know?

I checked with a co-worker on slack. He said:

There's no standard extension for DAP URLs. Explicitly excluding .zarr seems good enough for this disambiguation.

Interesting. Yes, there is no standard extension for opendap urls. OPeNDAP servers produce urls with the filename at the end, but for example NASA does something completely different. Excluding .zarr should be good.

What I am trying to push for this, is an opendap protocol-ization via the URL scheme. This is "dap2://<file_url>" vs "dap4://<file_url>". I already added it to the documentation back then dap2vdap4 Right now, if an opendap begins with http, then it is assumed to be dap2. This is completely on the client side and not a server thing. But pydap and python-netcdf4 support this, some NASA subsetting tools do this. Perhaps this may help separating opendap urls from non-opendap urls

Well, actually, Thredds (TDS) does have this "standard" way to specify the protocol that may help to discern between opendap url vs non-opendap url: a TDS dap2 url will have a dodsC in its urls. A TDS dap4 url will have a dap4 in its url. (see here). However, an organization running an opendap server may decide how their own urls are exposed.

shoyer · 2025-10-01T01:04:40Z

Conceptually, I think there is an ambiguity about guess_can_open. Does it means that a backend possibly or definitively can open a dataset?

Based on how it's used in open_dataset, I think we should pick defaults closer to "definitely" (which is what you do in this PR). It's a better user experience to require an explicit engine than to guess wrong and raise a less informative downstream error.

ianhi · 2025-10-01T01:47:35Z

I think there is an ambiguity about guess_can_open. Does it means that a backend possibly or definitively can open a dataset?

Based on how it's used in open_dataset, I think we should pick defaults closer to "definitely" (which is what you do in this PR). It's a better user experience to require an explicit engine than to guess wrong and raise a less informative downstream error.

I agree. I can make a follow up PR to the backends page to try make this more explicit.

For the case of the dap protocol I skimmed the specification https://opendap.github.io/dap4-specification/DAP4.html and didn't see anything in particular that made me feel confident about searching the URL for dap or opendap even if those are likely common, i think they are too restrictive on a URL. But this would also be a great question for @dcherian when he is back from vacation.

But for this PR I feel pretty happy with where it is and would defer further dap improvements to later, modulo a small fix to lower case the extension to be slightly more robust that I'll push shortly

ianhi · 2025-10-01T01:54:18Z

modulo a small fix to lower case the extension to be slightly more robust that I'll push shortly

Actually im not sure that's reasonable. Other backends do a similar check without thinking about case:

xarray/xarray/backends/scipy_.py

Lines 365 to 367 in eed12c4

    
           if isinstance(filename_or_obj, str | os.PathLike): 
        
               _, ext = os.path.splitext(filename_or_obj) 
        
               return ext in {".nc", ".nc4", ".cdf", ".gz"}

is that more correct?

shoyer · 2025-10-01T05:35:16Z

modulo a small fix to lower case the extension to be slightly more robust that I'll push shortly

Actually im not sure that's reasonable. Other backends do a similar check without thinking about case:

xarray/xarray/backends/scipy_.py

Lines 365 to 367 in eed12c4

if isinstance(filename_or_obj, str | os.PathLike):

_, ext = os.path.splitext(filename_or_obj)

return ext in {".nc", ".nc4", ".cdf", ".gz"}

is that more correct?

Really hard to say.

Honestly, I think even this list of extensions is pretty generous:

Scipy doesn't read netCDF v4 files, so .nc4 seems unlikely
Scipy can only read .gz files if they contain a netCDF v3 file. So maybe .nc.gz would be more appropriate than allowing any .gz file.

xarray/backends/scipy_.py

Mikejmnez · 2025-10-01T15:49:06Z

For the case of the dap protocol I skimmed the specification https://opendap.github.io/dap4-specification/DAP4.html and didn't see anything in particular that made me feel confident about searching the URL for dap or opendap even if those are likely common, i think they are too restrictive on a URL. But this would also be a great question for @dcherian when he is back from vacation.

.das .dds .dods (standard dap2 extensions), and .dmr, .dap (standard dap4 extensions) trigger downloads of either metadata or the entire file's binary data (.dods and .dap). URLS should not have those extensions...
These extensions are added by the backend (pydap or python-netcdf4), to create the python dataset objects. A url exposed by a server like THREDDS or Hyrax is likely not going to have this extension (it is possible only when a user manually creates it, which should not be entirely ruled out).

The only think that I can think of, that would be opendap specific, and part of the spec (dap4), appears when using a constraint expression in the url. These can often be part of the user-provided URL. For example:

url = "http://test.opendap.org/opendap/dap4/StaggeredGrid.nc4" # full data
url_ce2 = "http://test.opendap.org/opendap/dap4/StaggeredGrid.nc4?dap4.ce=/time[0:1:0];/Y[0:1:39];/X[0:1:39];/Eta[0:1:0][0:1:39][0:1:39]"
url_ce2 ="http://test.opendap.org/opendap/dap4/StaggeredGrid.nc4?dap4.ce=/time=[0:1:0];/Y=[0:1:39];/X=[0:1:39];/time;/X;/Y;/Eta"

url_ce1 and url_ce2 make use of the two ways to subset via its url, and as such the dap4.ce= must appear in the query parameter. That is a exclusively opendap that is implemented by any dap4 server.

ianhi · 2025-10-02T18:13:10Z

dap4.ce= must appear in the query parameter. That is a exclusively opendap that is implemented by any dap4 server.

hese can often be part of the user-provided URL.

Since it is often but not "always" what would you recommend conditioning guessing true on these? It would be a signfiicant increase in strictness over what is there now. The current changes are a medium increase. I'm not an dap user so I don't have a strong opinion.

shoyer · 2025-10-03T02:00:26Z

guess_can_open should be pretty conservative, something like 95% confident that the given backend can open the given file. It is a way better user experience to ask them to explicitly supply an engine than to guess wrong and get a mysterious error message. So pydap should require something explict to identify DAP, rather than guessing it can open any HTTP url.

…

On Thu, Oct 2, 2025 at 11:13 AM Ian Hunt-Isaak ***@***.***> wrote: *ianhi* left a comment (pydata/xarray#10804) <#10804 (comment)> dap4.ce= must appear in the query parameter. That is a exclusively opendap that is implemented by any dap4 server. hese can often be part of the user-provided URL. Since it is often but not "always" what would you recommend conditioning guessing true on these? It would be a signfiicant increase in strictness over what is there now. The current changes are a medium increase. I'm not an dap user so I don't have a strong opinion. — Reply to this email directly, view it on GitHub <#10804 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJJFVRUMOHILDO2VR5JWMD3VVTMXAVCNFSM6AAAAACH5N2T7SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNRSGQYTOMJVGY> . You are receiving this because you commented.Message ID: ***@***.***>

Mikejmnez · 2025-10-03T02:11:43Z

If the url has either dap2 or dap4 as its scheme, it will 100% be an opendap url. For example, dap4://opendap.earthdata.nasa.gov/collections/... and dap2://opendap.earthdata.... Both pydap and netcdf4-python understand this (as so do other non-python tools). In NASA land, there is a push for all tutorials have dap4 urls, for example. (these are not valid http schemes, but a client-side parse-able approach to specify the dap protocol somebody came up with some time ago).

ianhi · 2025-10-03T15:07:15Z

Both pydap and netcdf4-python understand this

So the NETCDF4 backend should also say yes to urls with dap in them?

I tried our your URL with both pydap and netcdf4 backends, but ran into an error for both, unfortunately

backend=netcdf4

OSError: [Errno -70] NetCDF: DAP server error: 'http://test.opendap.org/opendap/dap4/StaggeredGrid.nc4'

backend=pydap

seems to work but then fails with

RetryError: HTTPConnectionPool(host='test.opendap.org', port=80): Max retries exceeded with url: /opendap/dap4/StaggeredGrid.nc4.dds (Caused by ResponseError('too many 503 error responses'))

So i wasn't able to manually verify

shoyer · 2025-10-03T16:21:42Z

So the NETCDF4 backend should also say yes to urls with dap in them?

I'll defer to @Mikejmnez, but my impression is that we should definitely be preferring pydap to netCDF4 for DAP. I think DAP support is optional in netCDF-C.

So I would lean towards not claiming DAP urls in the netcdf4 backend, or maybe just being sure we try pydap before netcdf4.

Mikejmnez · 2025-10-03T16:56:20Z

So the NETCDF4 backend should also say yes to urls with dap in them?

I tried our your URL with both pydap and netcdf4 backends, but ran into an error for both, unfortunately

It looks like the test server was down overnight, and got restarted this morning. I tried it and it worked for me.

With pydap, all http, dap2, and dap4 work. http | https defaults to dap2.
With netcdf4 only http and dap4 work. dap2 does not, but http | https defaults to dap2.

My preference, would be to have any dap2 | dap4 scheme in the url, to be automatically assigned to pydap.

dopplershift · 2025-10-03T17:30:48Z

@shoyer While it's possible to build netcdf-c without DAP support, it's almost always built on and is frequently used. For instance, the netcdf4-python wheels are built with DAP support turned on in netCDF-C, as are the pacakges on conda-forge. So, PLEASE leave netcdf4-python able to read DAP by default.

EDIT: My default environment has neither h5netcdf nor pydap, just netcdf4. I'd really like for that environment to not suddenly start breaking my existing code/examples.

Mikejmnez · 2025-10-03T17:42:35Z

And I am up for keeping the defaults as is, and definitely NOT break people's workflows. I think the issue at hand is how to identify dap urls.

ianhi · 2025-10-03T17:50:12Z

So I would lean towards not claiming DAP urls in the netcdf4 backend, or maybe just being sure we try pydap before netcdf4.

I had a thought the other day when working on a custom zarr backend, that it would be nice to have a robust and cross-file-type system for expressing user preference for backend resolution order. It currently seems to be alphabetical, but with a special case of reordering the 3 built in netcdf backend. So the default guessing order will be:

  1. h5netcdf (from netcdf_engine_order[0])
  2. scipy (from netcdf_engine_order[1])
  3. netcdf4 (from netcdf_engine_order[2])
  4. pydap (alphabetically after netcdf4)
  5. store (alphabetically)
  6. zarr (alphabetically)

from the sorting here:

xarray/xarray/backends/plugins.py

Lines 94 to 100 in eed12c4

    
           for be_name in OPTIONS["netcdf_engine_order"]: 
        
               if backend_entrypoints.get(be_name) is not None: 
        
                   ordered_backends_entrypoints[be_name] = backend_entrypoints.pop(be_name) 
        
           ordered_backends_entrypoints.update( 
        
               {name: backend_entrypoints[name] for name in sorted(backend_entrypoints)} 
        
           ) 
        
           return ordered_backends_entrypoints

but if i add a zarr backend for a subtype of zarr it will only come first due to alphbetical order, but if i had a backend named like z-zarr it would not be used.

So, PLEASE leave netcdf4-python able to read DAP by default.

I will make sure to not remove the current situation where netcdf4 can report as being able to a.

This is just about the automatic engine resolution, not changing anything for an explicit engine=... but that definitely could break things

ianhi · 2025-10-03T17:53:58Z

This is also related to this issue i guess: #10810 (comment)

ianhi · 2025-10-08T19:01:02Z

Ok. Following the meeting today I am now happy with the state of this PR. It improves on the current state by having engine's not guess things that cannot read. However, it does not change the resolution order in the case that a backend can read something. This means we are still left the ambiguity of what a url ending in .nc means. However, this is existing ambiguity, not introduced here.

I have also added a section to the io docs about the resolution order and briefly discuss the .nc ambiguity.

xarray/backends/pydap_.py

xarray/backends/netCDF4_.py

dcherian · 2025-10-13T21:56:56Z

xarray/tests/test_backends.py

+        ("DAP4://example.com/dataset", "pydap"),  # uppercase scheme
+        ("https://example.com/services/DAP2/dataset", "pydap"),  # uppercase in path


Following #10804 (comment) can we check that netCDF4 gets chosen for these if pydap is not installed; you can use has_pydap from tests/__init__.py for the check and remove the requires_pydap decorator

dcherian

LGTM. Just one minor test request

ianhi · 2025-10-15T21:17:03Z

The remaining failures are not casued by the content of this PR.

Docs build: failing due to intersphinx isssue

flaky tests. Not caused by this PR but related. This URL is returning a 404 http://test.opendap.org/opendap/dap4/unaligned_simple_datatree.nc.h5.dmr i think the test server is down? Or possibly the content of the server was modified. looking here http://test.opendap.org/opendap/dap4/contents.html it sseems that the file we were looking for is not present @Mikejmnez maybe knows more here?

dcherian · 2025-10-16T02:03:03Z

Ok let's try this out!

fix: be more more caution when claiming a backend can open a URL

2af21ab

ianhi changed the title ~~fix: be more more caution when claiming a backend can open a URL~~ fix: be more cautious when claiming a backend can open a URL Sep 30, 2025

github-actions bot added topic-backends io labels Sep 30, 2025

add whats new entry

1a3e7df

shoyer approved these changes Sep 30, 2025

View reviewed changes

xarray/tests/test_backends.py Outdated Show resolved Hide resolved

shoyer reviewed Sep 30, 2025

View reviewed changes

doc/whats-new.rst Outdated Show resolved Hide resolved

fixes from review

d6a47b7

ianhi commented Sep 30, 2025

View reviewed changes

ianhi force-pushed the fix-netcdf4-remote-zarr-detection branch from b06a0a5 to 7ed1f0a Compare October 1, 2025 15:21

more caution in scipy netcdf backend

7ed1f0a

ianhi commented Oct 1, 2025

View reviewed changes

xarray/backends/scipy_.py Outdated Show resolved Hide resolved

correct suffix detection for scipy backend

60c1158

ianhi added 2 commits October 3, 2025 14:17

stricter URL detection for netcdf/dap

d2334e4

no query params for h5netcdf

ef3e07c

Merge branch 'main' into fix-netcdf4-remote-zarr-detection

e0e2da2

kmuehlbauer mentioned this pull request Oct 7, 2025

New h5netcdf engine may create files which netCDF can't read #10819

Open

5 tasks

ianhi added 4 commits October 8, 2025 11:43

Merge branch 'main' into fix-netcdf4-remote-zarr-detection

bfefb21

review comments

a50b2f6

fix windows failures

10d6edd

docs on backend resolution

8c77986

more complete table

079b290

ianhi force-pushed the fix-netcdf4-remote-zarr-detection branch from 66c9e1a to 079b290 Compare October 8, 2025 19:05

ianhi added 5 commits October 8, 2025 15:23

no horizontal scroll on table

6ee2910

Merge branch 'main' into fix-netcdf4-remote-zarr-detection

418ceee

fix whats new header

e32e93a

correct description

f445045

case insensitivity to DAP: vs dap:

4a717e7

dcherian reviewed Oct 13, 2025

View reviewed changes

xarray/backends/pydap_.py Outdated Show resolved Hide resolved

dcherian reviewed Oct 13, 2025

View reviewed changes

xarray/backends/netCDF4_.py Outdated Show resolved Hide resolved

dcherian reviewed Oct 13, 2025

View reviewed changes

dcherian approved these changes Oct 13, 2025

View reviewed changes

ianhi added 7 commits October 14, 2025 16:54

Merge branch 'main' into fix-netcdf4-remote-zarr-detection

7dc0995

thredds

00d07ee

move import

f29d7d8

claude import rules

d98d6dd

has_pydap instead of requires pydap

8f150dd

Merge branch 'main' into fix-netcdf4-remote-zarr-detection

cc64d7c

Merge branch 'main' into fix-netcdf4-remote-zarr-detection

8123beb

dcherian added the plan to merge Final call for comments label Oct 15, 2025

dcherian merged commit b5e4b0e into pydata:main Oct 16, 2025
44 of 47 checks passed

ianhi deleted the fix-netcdf4-remote-zarr-detection branch October 16, 2025 15:35

		("DAP4://example.com/dataset", "pydap"), # uppercase scheme
		("https://example.com/services/DAP2/dataset", "pydap"), # uppercase in path

Uh oh!

fix: be more cautious when guessing what a backend can open #10804

fix: be more cautious when guessing what a backend can open #10804

Uh oh!

Conversation

ianhi commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shoyer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ianhi Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

shoyer Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

ianhi Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Mikejmnez Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mikejmnez Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

shoyer commented Oct 1, 2025

Uh oh!

ianhi commented Oct 1, 2025

Uh oh!

ianhi commented Oct 1, 2025

Uh oh!

shoyer commented Oct 1, 2025

Uh oh!

Uh oh!

Mikejmnez commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianhi commented Oct 2, 2025

Uh oh!

shoyer commented Oct 3, 2025 via email

Uh oh!

Mikejmnez commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianhi commented Oct 3, 2025

Uh oh!

shoyer commented Oct 3, 2025

Uh oh!

Mikejmnez commented Oct 3, 2025

Uh oh!

dopplershift commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mikejmnez commented Oct 3, 2025

Uh oh!

ianhi commented Oct 3, 2025

Uh oh!

ianhi commented Oct 3, 2025

Uh oh!

ianhi commented Oct 8, 2025

Uh oh!

Uh oh!

Uh oh!

dcherian Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

dcherian left a comment

Choose a reason for hiding this comment

Uh oh!

ianhi commented Oct 15, 2025

Uh oh!

dcherian commented Oct 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

ianhi commented Sep 30, 2025 •

edited

Loading

Mikejmnez Oct 1, 2025 •

edited

Loading

Mikejmnez commented Oct 1, 2025 •

edited

Loading

Mikejmnez commented Oct 3, 2025 •

edited

Loading

dopplershift commented Oct 3, 2025 •

edited

Loading