Remove globbing from the spec

[RubenVerborgh]

I strongly think that globbing should be removed from the spec.

Reasons for removing

1. No one really wants globbing. People want cross-file data access, and there are better ways of achieving that. Globbing has always been a hack for accessing data in multiple files. It was not thought through well (see below).

2. Globbing is expensive on the server-side.

3. Globbing can lead to denial of service (on server and client).

4. Whatever can be achieved through globbing, can be achieved as efficiently without.
   With HTTP/2, there is zero overhead in just going through the files on the client side.

5. Let's remove it soon before it is actually widely used and implemented.

Reasons for keeping it plus mitigations

1. A (very low) number are using it.
   Let's upgrade them.

2. It has been in the spec for many years.
   That doesn't make it a good idea, and only a low number of apps are using it anyway (see 1).

Conditions

1. Globbing should NOT be removed until there is a replacement for the functionality it provides.

People who need to have a say in this

@timbl @melvincarvalho

[michielbdejong]

+1

[elf-pavlik]

👍 I questioned need for it a long time ago in solid/solid#116

[dmitrizagidulin]

+1

[gobengo]

Eventually it is probably ideal for developer trust and adoption to adopt a Linux-like "never break userspace" policy, and to never make backward-incompatible changes like this. What is in the way of adopting such a policy? (i.e. what other spec features should be considered 'at risk for removal' before a 'v1'?)

Edit: I don't think it's necessary to adopt such a policy today. But maybe by a year from now? I am +1 on this proposal in the interest of applying Occam's Razor to the core of the spec. Just think it's good to trim and apply learnings all at once here instead of continuing a piecemeal feature-removal strategy ad-infinitum. AS2 made backward-incompatible changes far too long into the spec, IMO, and it actively undercut my ability to get it adopted inside my organization at the time.

[RubenVerborgh]

Eventually it is probably ideal for developer trust and adoption to adopt a Linux-like "never break userspace" policy

Agreed: eventually.

What is in the way of adopting such a policy?

W3C standardization would be a good way of going through the spec with scrutiny, and identifying and fixing issues (such as globbing).

[melvincarvalho]

-1

This is in use

Consider this a formal objection.

Please by all means work on a HTTP2 library that could possibly act as a replacement. That would go some way to me withdrawing this objection.

[melvincarvalho]

More specifically, I think we did discuss this in the past, and globbing is not quite the same as grabbing all the content in a directory. Is that useful? I think there can be an argument for yes and one for no. For example in an LDPC you get all the file content in a directory, and that's useful. Not suggesting those are equivalents, but is more a meta point.

Also some stats on how well deployed http2 is would be handy. And a proposal on how http2 could replace globbing. I've had lots of interest from my social network app, in private discussions, including from former facebook people. And darcy etc. That uses globbing extensively. Could it be replaced. Possibly, but might take some design and time. Not time I currently have in the next quarter or two.

Im increasingly deploying solid servers on all my devices now including yesterday my android phone. So eventually I could see solid deployed widely including IoT. So http2 usage would be an interesting data point here. We cant just be a chrome specific project, we should think about solid as web servers running everywhere, in your home, your fridge, your watch, your phone etc.

There's a few things id like to see fixed and working before tackling this. So my more in depth answer is, not for now, we could mark it as "revisit", like many of our issues are.

However, speccing out a possible http2 solution seems to be a good idea, and I'd support that proposal.

[RubenVerborgh]

@melvincarvalho I'm in full agreement. Nothing will or should be removed until there is a replacement. I planned to explicitly state this in the issue, but forgot; will update now.

[RubenVerborgh]

Tracking implementation of such a client-side feature in solid/solid#253. Just a small note that we will likely not need anything HTTP/2-specific: HTTP/2 will automatically optimize the sequence of requests.

[gobengo]

On pros cons (from solid/solid#253 (comment))

So let's just be honest about what are the pros/cons. From #145 the OP starts with

1. No one really wants globbing. People want cross-file data access, and there are better ways of achieving that. Globbing has always been a hack for accessing data in multiple files. It was not thought through well (see below).

I don't pretend to know about this.

2. Globbing is expensive on the server-side.

Only if you implement it naively. Alternate datastores (or if the datastore is a filesystem, shell out to bash or a C lib instead of globbing in node) can make this an O(1) lookup. This assertion needs more justification.

3. Globbing can lead to denial of service (on server and client).

See 2. DDOS are a risk no matter what, e.g. by repeatedly getting full directory listings that are huge and taking up all available OS connections. It's not that unique to globbing. Practically, pros would deploy behind a DDOS-protecting middleware that makes this a non-issue.

4. Whatever can be achieved through globbing, can be achieved as efficiently without. With HTTP/2, there is zero overhead in just going through the files on the client side.

My argument here is meant to analyze if this is true.

At this point I think everyone agrees the 'HTTP/2' mention isn't what's important. Even over HTTP there is no'zero overhead', but the overhead is probably negligible in the vast majority of near-term scenarios.

5. Let's remove it soon before it is actually widely used and implemented.

No argument here.

---

So 1 and 5 are likely good reasons. 3 is a bit of an overstatement ('zero overhead'), but can be rephrased to be just as convincing.

---

And the scalable solution here will be querying. Globbing is just a poor man's query; let's replace that with a client-side solution, and built proper query interfaces instead.

Totally agree! +1